2024-09-16T04:30:15.437 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T04:30:15.453 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T04:30:15.531 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906037 branch: squid description: rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906037' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) sha1: 8921c8ef947807b717db041d85f0761b7914d5fc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 700 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 5616 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 15927/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 targets: smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJtpr4W0XE9G5N7sApdfuavl8bPfa9QcKIetLpZaCAVyMA+r39WFdgAXnkxkvI+nHgJsLR6oEXgantmBIw5PQ7s= tasks: - install: null - ceph: create_rbd_pool: false log-ignorelist: - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(SLOW_OPS\) - slow request - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - ec_lost_unfound: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T04:30:15.531 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T04:30:15.532 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T04:30:15.532 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T04:30:15.534 INFO:teuthology.task.internal:Checking packages... 2024-09-16T04:30:15.555 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-16T04:30:15.556 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T04:30:15.556 INFO:teuthology.packaging:ref: None 2024-09-16T04:30:15.556 INFO:teuthology.packaging:tag: None 2024-09-16T04:30:15.556 INFO:teuthology.packaging:branch: squid 2024-09-16T04:30:15.556 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:30:15.556 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-16T04:30:15.763 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-328.gad35db06 2024-09-16T04:30:15.765 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T04:30:15.772 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T04:30:15.772 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T04:30:15.793 INFO:teuthology.task.internal:Saving configuration 2024-09-16T04:30:15.805 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T04:30:15.812 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T04:30:15.836 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906037', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 04:25:15.996209', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJtpr4W0XE9G5N7sApdfuavl8bPfa9QcKIetLpZaCAVyMA+r39WFdgAXnkxkvI+nHgJsLR6oEXgantmBIw5PQ7s='} 2024-09-16T04:30:15.836 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T04:30:15.842 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2024-09-16T04:30:15.842 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T04:30:15.893 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3e4ef6bac0>, signals=[15]) 2024-09-16T04:30:15.894 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T04:30:15.905 INFO:teuthology.task.internal:Opening connections... 2024-09-16T04:30:15.905 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T04:30:15.906 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:30:15.997 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T04:30:16.028 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2024-09-16T04:30:16.047 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2024-09-16T04:30:16.047 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2024-09-16T04:30:16.111 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2024-09-16T04:30:16.111 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2024-09-16T04:30:16.112 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2024-09-16T04:30:16.112 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2024-09-16T04:30:16.112 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2024-09-16T04:30:16.112 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2024-09-16T04:30:16.112 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T04:30:16.112 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2024-09-16T04:30:16.112 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2024-09-16T04:30:16.112 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T04:30:16.113 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2024-09-16T04:30:16.113 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T04:30:16.113 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T04:30:16.113 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T04:30:16.114 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2024-09-16T04:30:16.134 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T04:30:16.310 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T04:30:16.497 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T04:30:16.497 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2024-09-16T04:30:16.515 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T04:30:16.580 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T04:30:16.581 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2024-09-16T04:30:16.682 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T04:30:16.714 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T04:30:16.728 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T04:30:16.728 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T04:30:16.728 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T04:30:16.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:30:16.746 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-09-16T04:30:16.804 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-503.el9.x86_64 2024-09-16T04:30:16.804 INFO:teuthology.task.kernel:Running kernel on smithi152: 5.14.0-503.el9.x86_64 2024-09-16T04:30:16.804 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum install -y kernel 2024-09-16T04:30:19.260 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-16T04:30:24.985 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 20 MB 00:04 2024-09-16T04:30:29.419 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-16T04:30:30.862 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 19 kB 00:00 2024-09-16T04:30:31.875 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-16T04:30:37.355 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-16T04:30:39.370 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T04:30:39.370 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T04:30:39.470 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:30:39.477 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:30:39.477 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-09-16T04:30:39.477 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:30:39.478 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:30:39.478 INFO:teuthology.orchestra.run.smithi152.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T04:30:39.478 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T04:30:39.478 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T04:30:39.478 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T04:30:39.478 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T04:30:39.478 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:30:39.478 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:30:39.478 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:30:39.478 INFO:teuthology.orchestra.run.smithi152.stdout:Install 4 Packages 2024-09-16T04:30:39.478 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:30:39.478 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 79 M 2024-09-16T04:30:39.478 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 126 M 2024-09-16T04:30:39.479 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:30:39.590 INFO:teuthology.orchestra.run.smithi152.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 247 kB/s | 23 kB 00:00 2024-09-16T04:30:39.915 INFO:teuthology.orchestra.run.smithi152.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-09-16T04:30:40.266 INFO:teuthology.orchestra.run.smithi152.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 45 MB/s | 35 MB 00:00 2024-09-16T04:30:40.633 INFO:teuthology.orchestra.run.smithi152.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 27 MB/s | 29 MB 00:01 2024-09-16T04:30:40.633 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:30:40.633 INFO:teuthology.orchestra.run.smithi152.stdout:Total 68 MB/s | 79 MB 00:01 2024-09-16T04:30:41.324 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:30:41.694 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:30:41.694 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:30:42.657 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:30:42.657 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:30:45.109 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:30:45.779 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T04:30:45.868 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:30:48.671 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:30:48.813 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:30:53.660 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:30:53.706 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:30:58.590 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:31:37.256 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:31:37.302 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:31:37.642 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:31:37.642 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T04:31:37.642 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:31:37.642 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:31:38.453 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:31:38.453 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:31:38.453 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:31:38.453 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T04:31:38.453 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T04:31:38.453 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T04:31:38.454 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T04:31:38.454 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:31:38.454 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:31:38.820 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T04:31:38.899 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T04:31:38.899 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T04:31:38.899 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T04:31:38.899 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T04:31:38.899 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T04:31:38.900 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T04:31:38.900 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi152.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T04:31:38.900 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T04:31:38.959 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T04:31:38.959 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T04:31:38.959 DEBUG:teuthology.orchestra.run.smithi152:> sudo rpm -qi grub2-tools 2024-09-16T04:31:39.001 INFO:teuthology.orchestra.run.smithi152.stdout:Name : grub2-tools 2024-09-16T04:31:39.002 INFO:teuthology.orchestra.run.smithi152.stdout:Epoch : 1 2024-09-16T04:31:39.002 INFO:teuthology.orchestra.run.smithi152.stdout:Version : 2.06 2024-09-16T04:31:39.002 INFO:teuthology.orchestra.run.smithi152.stdout:Release : 82.el9 2024-09-16T04:31:39.002 INFO:teuthology.orchestra.run.smithi152.stdout:Architecture: x86_64 2024-09-16T04:31:39.002 INFO:teuthology.orchestra.run.smithi152.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T04:31:39.002 INFO:teuthology.orchestra.run.smithi152.stdout:Group : Unspecified 2024-09-16T04:31:39.002 INFO:teuthology.orchestra.run.smithi152.stdout:Size : 8274670 2024-09-16T04:31:39.002 INFO:teuthology.orchestra.run.smithi152.stdout:License : GPLv3+ 2024-09-16T04:31:39.002 INFO:teuthology.orchestra.run.smithi152.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T04:31:39.002 INFO:teuthology.orchestra.run.smithi152.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T04:31:39.002 INFO:teuthology.orchestra.run.smithi152.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T04:31:39.002 INFO:teuthology.orchestra.run.smithi152.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T04:31:39.002 INFO:teuthology.orchestra.run.smithi152.stdout:Packager : builder@centos.org 2024-09-16T04:31:39.003 INFO:teuthology.orchestra.run.smithi152.stdout:Vendor : CentOS 2024-09-16T04:31:39.003 INFO:teuthology.orchestra.run.smithi152.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T04:31:39.003 INFO:teuthology.orchestra.run.smithi152.stdout:Summary : Support tools for GRUB. 2024-09-16T04:31:39.003 INFO:teuthology.orchestra.run.smithi152.stdout:Description : 2024-09-16T04:31:39.003 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:31:39.003 INFO:teuthology.orchestra.run.smithi152.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T04:31:39.003 INFO:teuthology.orchestra.run.smithi152.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T04:31:39.003 INFO:teuthology.orchestra.run.smithi152.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T04:31:39.003 INFO:teuthology.orchestra.run.smithi152.stdout:hardware devices. 2024-09-16T04:31:39.003 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:31:39.003 INFO:teuthology.orchestra.run.smithi152.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T04:31:39.004 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T04:31:39.004 INFO:teuthology.task.kernel:Updating grub on smithi152 to boot 5.14.0-505.el9.x86_64 2024-09-16T04:31:39.004 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T04:31:39.782 INFO:teuthology.orchestra.run.smithi152.stderr:Generating grub configuration file ... 2024-09-16T04:31:41.196 INFO:teuthology.orchestra.run.smithi152.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T04:31:41.222 INFO:teuthology.orchestra.run.smithi152.stderr:done 2024-09-16T04:31:41.225 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-09-16T04:31:41.241 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.lwEzIzA4Z7 2024-09-16T04:31:41.241 DEBUG:teuthology.orchestra.run.smithi152:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lwEzIzA4Z7 2024-09-16T04:31:41.307 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.lwEzIzA4Z7 2024-09-16T04:31:41.462 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.lwEzIzA4Z7 is 6KB 2024-09-16T04:31:41.511 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.lwEzIzA4Z7 2024-09-16T04:31:41.526 DEBUG:teuthology.orchestra.run.smithi152:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T04:31:41.589 INFO:teuthology.orchestra.run.smithi152.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T04:31:41.589 INFO:teuthology.orchestra.run.smithi152.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T04:31:41.589 INFO:teuthology.orchestra.run.smithi152.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T04:31:41.589 INFO:teuthology.orchestra.run.smithi152.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T04:31:41.591 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T04:31:41.734 DEBUG:teuthology.orchestra.run.smithi152:> sudo shutdown -r now 2024-09-16T04:32:11.753 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T04:32:11.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-16T04:32:11.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:32:46.397 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-09-16T04:32:55.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-16T04:32:55.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:33:13.789 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-09-16T04:33:25.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-16T04:33:25.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:33:26.262 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-09-16T04:33:26.693 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-16T04:33:26.693 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T04:33:26.693 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-09-16T04:33:26.709 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-505.el9.x86_64 2024-09-16T04:33:26.709 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T04:33:26.709 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T04:33:26.709 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T04:33:27.710 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T04:33:27.710 DEBUG:teuthology.orchestra.run.smithi152:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T04:33:27.784 INFO:teuthology.orchestra.run.smithi152.stdout:ttyS1 2024-09-16T04:33:27.812 DEBUG:teuthology.parallel:result is None 2024-09-16T04:33:27.812 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T04:33:27.819 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T04:33:27.819 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T04:33:27.834 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T04:33:27.841 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T04:33:27.851 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T04:33:27.851 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T04:33:27.895 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T04:33:27.903 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T04:33:27.904 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T04:33:27.960 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:33:27.970 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:33:27.972 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T04:33:27.979 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T04:33:27.979 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T04:33:28.035 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T04:33:28.045 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T04:33:28.045 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T04:33:28.088 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:33:28.163 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:33:28.208 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:33:28.289 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:33:28.325 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:33:28.326 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T04:33:28.388 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2024-09-16T04:33:28.471 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:33:28.585 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T04:33:28.593 INFO:teuthology.task.internal:Starting timer... 2024-09-16T04:33:28.594 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T04:33:28.608 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T04:33:28.616 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-16T04:33:28.616 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2024-09-16T04:33:28.669 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2024-09-16T04:33:28.670 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T04:33:28.670 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2024-09-16T04:33:28.698 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2024-09-16T04:33:28.698 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi152.front.sepia.ceph.com': 'permissive'} 2024-09-16T04:33:28.698 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:33:28.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:33:28.761 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T04:33:28.761 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2024-09-16T04:33:28.859 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T04:33:28.868 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T04:33:28.881 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T04:33:28.882 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi152.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T04:37:13.995 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi152.front.sepia.ceph.com')] 2024-09-16T04:37:13.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-16T04:37:13.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:37:14.075 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-09-16T04:37:14.156 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-16T04:37:14.156 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T04:37:14.171 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T04:37:14.171 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T04:37:14.172 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:37:14.225 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T04:37:14.239 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T04:37:14.263 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2024-09-16T04:37:14.274 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2024-09-16T04:37:14.289 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T04:37:14.302 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T04:37:14.357 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:37:14.361 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:37:14.361 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-09-16T04:37:14.361 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:37:14.361 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:37:14.361 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:37:14.361 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:37:14.362 INFO:teuthology.run_tasks:Running task install... 2024-09-16T04:37:14.370 DEBUG:teuthology.task.install:project ceph 2024-09-16T04:37:14.370 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T04:37:14.370 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T04:37:14.370 INFO:teuthology.task.install:Using flavor: default 2024-09-16T04:37:14.375 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T04:37:14.376 INFO:teuthology.task.install:extra packages: [] 2024-09-16T04:37:14.376 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T04:37:14.376 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:37:14.529 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T04:37:14.529 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T04:37:14.639 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T04:37:14.640 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:37:14.640 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T04:37:14.667 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T04:37:14.667 DEBUG:teuthology.orchestra.run.smithi152:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T04:37:14.734 DEBUG:teuthology.orchestra.run.smithi152:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T04:37:14.811 DEBUG:teuthology.orchestra.run.smithi152:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T04:37:14.875 INFO:teuthology.orchestra.run.smithi152.stdout:check_obsoletes = 1 2024-09-16T04:37:14.876 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum clean all 2024-09-16T04:37:15.100 INFO:teuthology.orchestra.run.smithi152.stdout:45 files removed 2024-09-16T04:37:15.127 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-radosgw 2024-09-16T04:37:15.690 INFO:teuthology.orchestra.run.smithi152.stdout:ceph packages for x86_64 298 kB/s | 83 kB 00:00 2024-09-16T04:37:15.973 INFO:teuthology.orchestra.run.smithi152.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-16T04:37:16.249 INFO:teuthology.orchestra.run.smithi152.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-16T04:37:16.857 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-16T04:37:18.826 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-16T04:37:23.068 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-16T04:37:25.352 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 19 kB 00:01 2024-09-16T04:37:26.272 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-16T04:37:31.820 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-16T04:37:34.171 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:37:34.177 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:37:34.177 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T04:37:34.177 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:37:34.177 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:37:34.177 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T04:37:34.177 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrading: 2024-09-16T04:37:34.177 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T04:37:34.177 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T04:37:34.177 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T04:37:34.178 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T04:37:34.178 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T04:37:34.178 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T04:37:34.178 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T04:37:34.178 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T04:37:34.178 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T04:37:34.178 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T04:37:34.178 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T04:37:34.178 INFO:teuthology.orchestra.run.smithi152.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T04:37:34.178 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T04:37:34.178 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T04:37:34.178 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T04:37:34.179 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T04:37:34.179 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T04:37:34.179 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T04:37:34.179 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T04:37:34.179 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T04:37:34.179 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T04:37:34.179 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T04:37:34.179 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T04:37:34.179 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T04:37:34.179 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T04:37:34.179 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T04:37:34.179 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T04:37:34.179 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T04:37:34.180 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T04:37:34.180 INFO:teuthology.orchestra.run.smithi152.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T04:37:34.180 INFO:teuthology.orchestra.run.smithi152.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T04:37:34.180 INFO:teuthology.orchestra.run.smithi152.stdout:Installing weak dependencies: 2024-09-16T04:37:34.180 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T04:37:34.180 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:37:34.181 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:37:34.181 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:37:34.181 INFO:teuthology.orchestra.run.smithi152.stdout:Install 30 Packages 2024-09-16T04:37:34.181 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrade 2 Packages 2024-09-16T04:37:34.181 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:37:34.181 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 60 M 2024-09-16T04:37:34.181 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:37:35.079 INFO:teuthology.orchestra.run.smithi152.stdout:(1/32): ceph-base-19.1.1-326.g8921c8ef.el9.x86_ 7.4 MB/s | 5.4 MB 00:00 2024-09-16T04:37:35.188 INFO:teuthology.orchestra.run.smithi152.stdout:(2/32): ceph-radosgw-19.1.1-326.g8921c8ef.el9.x 13 MB/s | 11 MB 00:00 2024-09-16T04:37:35.213 INFO:teuthology.orchestra.run.smithi152.stdout:(3/32): ceph-selinux-19.1.1-326.g8921c8ef.el9.x 189 kB/s | 25 kB 00:00 2024-09-16T04:37:35.255 INFO:teuthology.orchestra.run.smithi152.stdout:(4/32): libcephfs2-19.1.1-326.g8921c8ef.el9.x86 11 MB/s | 751 kB 00:00 2024-09-16T04:37:35.289 INFO:teuthology.orchestra.run.smithi152.stdout:(5/32): libradosstriper1-19.1.1-326.g8921c8ef.e 6.6 MB/s | 506 kB 00:00 2024-09-16T04:37:35.339 INFO:teuthology.orchestra.run.smithi152.stdout:(6/32): python3-ceph-argparse-19.1.1-326.g8921c 905 kB/s | 45 kB 00:00 2024-09-16T04:37:35.573 INFO:teuthology.orchestra.run.smithi152.stdout:(7/32): ceph-common-19.1.1-326.g8921c8ef.el9.x8 17 MB/s | 21 MB 00:01 2024-09-16T04:37:35.606 INFO:teuthology.orchestra.run.smithi152.stdout:(8/32): python3-ceph-common-19.1.1-326.g8921c8e 510 kB/s | 136 kB 00:00 2024-09-16T04:37:35.632 INFO:teuthology.orchestra.run.smithi152.stdout:(9/32): python3-cephfs-19.1.1-326.g8921c8ef.el9 2.7 MB/s | 161 kB 00:00 2024-09-16T04:37:35.715 INFO:teuthology.orchestra.run.smithi152.stdout:(10/32): librgw2-19.1.1-326.g8921c8ef.el9.x86_6 12 MB/s | 5.3 MB 00:00 2024-09-16T04:37:35.740 INFO:teuthology.orchestra.run.smithi152.stdout:(11/32): python3-rados-19.1.1-326.g8921c8ef.el9 2.3 MB/s | 320 kB 00:00 2024-09-16T04:37:35.766 INFO:teuthology.orchestra.run.smithi152.stdout:(12/32): python3-rgw-19.1.1-326.g8921c8ef.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-16T04:37:35.792 INFO:teuthology.orchestra.run.smithi152.stdout:(13/32): python3-rbd-19.1.1-326.g8921c8ef.el9.x 1.8 MB/s | 300 kB 00:00 2024-09-16T04:37:35.874 INFO:teuthology.orchestra.run.smithi152.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 799 kB/s | 107 kB 00:00 2024-09-16T04:37:35.891 INFO:teuthology.orchestra.run.smithi152.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 452 kB/s | 45 kB 00:00 2024-09-16T04:37:35.942 INFO:teuthology.orchestra.run.smithi152.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 912 kB/s | 160 kB 00:00 2024-09-16T04:37:35.983 INFO:teuthology.orchestra.run.smithi152.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.9 MB/s | 662 kB 00:00 2024-09-16T04:37:36.001 INFO:teuthology.orchestra.run.smithi152.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 724 kB/s | 42 kB 00:00 2024-09-16T04:37:36.026 INFO:teuthology.orchestra.run.smithi152.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-16T04:37:36.059 INFO:teuthology.orchestra.run.smithi152.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-09-16T04:37:36.076 INFO:teuthology.orchestra.run.smithi152.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 476 kB/s | 36 kB 00:00 2024-09-16T04:37:36.093 INFO:teuthology.orchestra.run.smithi152.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 708 kB/s | 47 kB 00:00 2024-09-16T04:37:36.152 INFO:teuthology.orchestra.run.smithi152.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 600 kB/s | 26 kB 00:00 2024-09-16T04:37:36.177 INFO:teuthology.orchestra.run.smithi152.stdout:(24/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-16T04:37:36.194 INFO:teuthology.orchestra.run.smithi152.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-09-16T04:37:36.219 INFO:teuthology.orchestra.run.smithi152.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-16T04:37:36.261 INFO:teuthology.orchestra.run.smithi152.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-16T04:37:36.320 INFO:teuthology.orchestra.run.smithi152.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-09-16T04:37:36.420 INFO:teuthology.orchestra.run.smithi152.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-09-16T04:37:36.562 INFO:teuthology.orchestra.run.smithi152.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-09-16T04:37:36.620 INFO:teuthology.orchestra.run.smithi152.stdout:(31/32): librados2-19.1.1-326.g8921c8ef.el9.x86 11 MB/s | 3.4 MB 00:00 2024-09-16T04:37:36.679 INFO:teuthology.orchestra.run.smithi152.stdout:(32/32): librbd1-19.1.1-326.g8921c8ef.el9.x86_6 12 MB/s | 3.2 MB 00:00 2024-09-16T04:37:36.684 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:36.684 INFO:teuthology.orchestra.run.smithi152.stdout:Total 24 MB/s | 60 MB 00:02 2024-09-16T04:37:36.787 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:37:36.824 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:37:36.825 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:37:37.484 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:37:37.485 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:37:38.973 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:37:39.054 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-16T04:37:39.084 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-16T04:37:39.109 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-16T04:37:39.132 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:37:39.157 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:37:39.357 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:37:39.434 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:37:39.782 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-16T04:37:39.894 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:37:39.986 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:37:40.075 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 8/34 2024-09-16T04:37:40.112 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:37:40.175 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:37:40.212 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-16T04:37:40.271 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 11/34 2024-09-16T04:37:40.342 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:37:40.364 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:37:40.428 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:37:40.471 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-16T04:37:40.578 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-16T04:37:40.639 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-16T04:37:40.679 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-16T04:37:40.728 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-16T04:37:40.757 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-16T04:37:40.795 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-16T04:37:41.099 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-16T04:37:41.136 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:37:41.226 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:37:41.266 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 23/34 2024-09-16T04:37:41.310 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-16T04:37:41.789 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-16T04:37:41.822 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:37:41.892 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:37:41.962 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 27/34 2024-09-16T04:37:41.991 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 28/34 2024-09-16T04:37:44.185 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:37:44.237 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:37:44.791 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:37:44.832 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:37:44.908 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:37:44.908 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T04:37:44.908 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T04:37:44.908 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:37:44.941 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:37:55.171 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:37:55.172 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /sys 2024-09-16T04:37:55.172 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /proc 2024-09-16T04:37:55.172 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /mnt 2024-09-16T04:37:55.172 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /var/tmp 2024-09-16T04:37:55.172 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /home 2024-09-16T04:37:55.172 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /root 2024-09-16T04:37:55.172 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /tmp 2024-09-16T04:37:55.172 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:37:56.138 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:37:56.164 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:37:56.164 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:37:56.164 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T04:37:56.164 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:37:56.164 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:37:56.164 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:37:56.166 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:37:56.248 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:37:56.249 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:37:57.483 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:37:57.483 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 1/34 2024-09-16T04:37:57.483 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2/34 2024-09-16T04:37:57.483 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 3/34 2024-09-16T04:37:57.484 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 4/34 2024-09-16T04:37:57.484 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 5/34 2024-09-16T04:37:57.484 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 6/34 2024-09-16T04:37:57.484 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:37:57.484 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 8/34 2024-09-16T04:37:57.485 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 9/34 2024-09-16T04:37:57.485 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 10/34 2024-09-16T04:37:57.485 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 11/34 2024-09-16T04:37:57.485 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:37:57.485 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:37:57.485 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-16T04:37:57.486 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-16T04:37:57.486 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-16T04:37:57.486 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-16T04:37:57.486 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-16T04:37:57.486 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-16T04:37:57.486 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-16T04:37:57.486 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-16T04:37:57.486 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-16T04:37:57.486 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-16T04:37:57.486 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-16T04:37:57.487 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-16T04:37:57.487 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-16T04:37:57.487 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-16T04:37:57.487 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-16T04:37:57.487 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-16T04:37:57.487 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-16T04:37:57.487 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:37:57.487 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-16T04:37:57.487 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 33/34 2024-09-16T04:37:58.645 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:37:58.645 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:37:58.645 INFO:teuthology.orchestra.run.smithi152.stdout:Upgraded: 2024-09-16T04:37:58.645 INFO:teuthology.orchestra.run.smithi152.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.646 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.646 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:37:58.646 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T04:37:58.646 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.646 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.646 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.646 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.646 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T04:37:58.646 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T04:37:58.647 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T04:37:58.647 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.647 INFO:teuthology.orchestra.run.smithi152.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T04:37:58.647 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T04:37:58.647 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T04:37:58.647 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.647 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T04:37:58.647 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.647 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T04:37:58.647 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T04:37:58.647 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T04:37:58.648 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.648 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.648 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.648 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T04:37:58.648 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.648 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.648 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.648 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T04:37:58.648 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T04:37:58.648 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T04:37:58.648 INFO:teuthology.orchestra.run.smithi152.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T04:37:58.648 INFO:teuthology.orchestra.run.smithi152.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T04:37:58.648 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:37:58.649 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:37:58.988 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-test 2024-09-16T04:37:59.559 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:28 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:37:59.691 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:37:59.692 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:37:59.692 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T04:37:59.692 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:37:59.692 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:37:59.692 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T04:37:59.692 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T04:37:59.693 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T04:37:59.693 INFO:teuthology.orchestra.run.smithi152.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T04:37:59.693 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T04:37:59.693 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:37:59.693 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:37:59.693 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:37:59.693 INFO:teuthology.orchestra.run.smithi152.stdout:Install 4 Packages 2024-09-16T04:37:59.693 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:37:59.694 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 49 M 2024-09-16T04:37:59.694 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 209 M 2024-09-16T04:37:59.694 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:37:59.893 INFO:teuthology.orchestra.run.smithi152.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-09-16T04:37:59.935 INFO:teuthology.orchestra.run.smithi152.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-16T04:38:00.127 INFO:teuthology.orchestra.run.smithi152.stdout:(3/4): libcephsqlite-19.1.1-326.g8921c8ef.el9.x 423 kB/s | 163 kB 00:00 2024-09-16T04:38:01.861 INFO:teuthology.orchestra.run.smithi152.stdout:(4/4): ceph-test-19.1.1-326.g8921c8ef.el9.x86_6 23 MB/s | 49 MB 00:02 2024-09-16T04:38:01.863 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:38:01.863 INFO:teuthology.orchestra.run.smithi152.stdout:Total 23 MB/s | 49 MB 00:02 2024-09-16T04:38:01.890 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:38:01.910 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:38:01.910 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:38:02.298 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:38:02.299 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:38:02.824 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:38:02.915 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T04:38:02.967 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T04:38:03.004 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:38:07.606 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:38:07.693 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:38:10.124 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:38:10.125 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 1/4 2024-09-16T04:38:10.125 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2/4 2024-09-16T04:38:10.125 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T04:38:10.448 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T04:38:10.448 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:10.448 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:38:10.448 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:38:10.448 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:38:10.448 INFO:teuthology.orchestra.run.smithi152.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T04:38:10.448 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T04:38:10.448 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:10.449 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:38:10.719 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph 2024-09-16T04:38:11.305 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:40 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:38:11.453 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:38:11.453 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:11.453 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T04:38:11.453 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:11.454 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:38:11.454 INFO:teuthology.orchestra.run.smithi152.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T04:38:11.454 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T04:38:11.454 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T04:38:11.454 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T04:38:11.454 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T04:38:11.454 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T04:38:11.454 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T04:38:11.454 INFO:teuthology.orchestra.run.smithi152.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T04:38:11.454 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T04:38:11.454 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T04:38:11.454 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T04:38:11.455 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T04:38:11.455 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T04:38:11.455 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T04:38:11.455 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T04:38:11.455 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T04:38:11.455 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T04:38:11.455 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T04:38:11.455 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T04:38:11.455 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T04:38:11.455 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T04:38:11.455 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T04:38:11.455 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T04:38:11.456 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T04:38:11.456 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T04:38:11.456 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T04:38:11.456 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T04:38:11.456 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T04:38:11.456 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T04:38:11.456 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T04:38:11.456 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T04:38:11.456 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T04:38:11.456 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T04:38:11.456 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T04:38:11.456 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T04:38:11.457 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T04:38:11.457 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T04:38:11.457 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T04:38:11.457 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:11.457 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:38:11.457 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:11.457 INFO:teuthology.orchestra.run.smithi152.stdout:Install 37 Packages 2024-09-16T04:38:11.457 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:11.458 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 30 M 2024-09-16T04:38:11.458 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 108 M 2024-09-16T04:38:11.458 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:38:11.918 INFO:teuthology.orchestra.run.smithi152.stdout:(1/37): ceph-19.1.1-326.g8921c8ef.el9.x86_64.rp 50 kB/s | 6.5 kB 00:00 2024-09-16T04:38:12.077 INFO:teuthology.orchestra.run.smithi152.stdout:(2/37): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_6 5.2 MB/s | 1.5 MB 00:00 2024-09-16T04:38:12.127 INFO:teuthology.orchestra.run.smithi152.stdout:(3/37): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_6 7.0 MB/s | 2.4 MB 00:00 2024-09-16T04:38:12.236 INFO:teuthology.orchestra.run.smithi152.stdout:(4/37): ceph-mgr-modules-core-19.1.1-326.g8921c 2.3 MB/s | 249 kB 00:00 2024-09-16T04:38:12.344 INFO:teuthology.orchestra.run.smithi152.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-09-16T04:38:12.436 INFO:teuthology.orchestra.run.smithi152.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 13 MB/s | 1.2 MB 00:00 2024-09-16T04:38:12.503 INFO:teuthology.orchestra.run.smithi152.stdout:(7/37): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_6 8.0 MB/s | 4.7 MB 00:00 2024-09-16T04:38:12.528 INFO:teuthology.orchestra.run.smithi152.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-09-16T04:38:12.712 INFO:teuthology.orchestra.run.smithi152.stdout:(9/37): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_6 27 MB/s | 17 MB 00:00 2024-09-16T04:38:12.737 INFO:teuthology.orchestra.run.smithi152.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 598 kB/s | 125 kB 00:00 2024-09-16T04:38:12.763 INFO:teuthology.orchestra.run.smithi152.stdout:(11/37): python3-pycparser-2.20-6.el9.noarch.rp 521 kB/s | 135 kB 00:00 2024-09-16T04:38:12.805 INFO:teuthology.orchestra.run.smithi152.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 188 kB 00:00 2024-09-16T04:38:12.863 INFO:teuthology.orchestra.run.smithi152.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 596 kB/s | 35 kB 00:00 2024-09-16T04:38:12.889 INFO:teuthology.orchestra.run.smithi152.stdout:(14/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.2 MB/s | 215 kB 00:00 2024-09-16T04:38:12.922 INFO:teuthology.orchestra.run.smithi152.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-16T04:38:12.947 INFO:teuthology.orchestra.run.smithi152.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 920 kB/s | 77 kB 00:00 2024-09-16T04:38:12.973 INFO:teuthology.orchestra.run.smithi152.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 607 kB/s | 51 kB 00:00 2024-09-16T04:38:12.998 INFO:teuthology.orchestra.run.smithi152.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-16T04:38:13.057 INFO:teuthology.orchestra.run.smithi152.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 853 kB/s | 43 kB 00:00 2024-09-16T04:38:13.082 INFO:teuthology.orchestra.run.smithi152.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 166 kB/s | 22 kB 00:00 2024-09-16T04:38:13.107 INFO:teuthology.orchestra.run.smithi152.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-09-16T04:38:13.141 INFO:teuthology.orchestra.run.smithi152.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-16T04:38:13.178 INFO:teuthology.orchestra.run.smithi152.stdout:(23/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.7 MB/s | 358 kB 00:00 2024-09-16T04:38:13.194 INFO:teuthology.orchestra.run.smithi152.stdout:(24/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 123 kB/s | 11 kB 00:00 2024-09-16T04:38:13.211 INFO:teuthology.orchestra.run.smithi152.stdout:(25/37): python3-jaraco-classes-3.2.1-5.el9.noa 258 kB/s | 18 kB 00:00 2024-09-16T04:38:13.228 INFO:teuthology.orchestra.run.smithi152.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-16T04:38:13.245 INFO:teuthology.orchestra.run.smithi152.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 579 kB/s | 19 kB 00:00 2024-09-16T04:38:13.262 INFO:teuthology.orchestra.run.smithi152.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-16T04:38:13.287 INFO:teuthology.orchestra.run.smithi152.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 788 kB/s | 46 kB 00:00 2024-09-16T04:38:13.312 INFO:teuthology.orchestra.run.smithi152.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T04:38:13.338 INFO:teuthology.orchestra.run.smithi152.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-16T04:38:13.363 INFO:teuthology.orchestra.run.smithi152.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-16T04:38:13.388 INFO:teuthology.orchestra.run.smithi152.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 709 kB/s | 36 kB 00:00 2024-09-16T04:38:13.422 INFO:teuthology.orchestra.run.smithi152.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-16T04:38:13.447 INFO:teuthology.orchestra.run.smithi152.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 802 kB/s | 20 kB 00:00 2024-09-16T04:38:13.473 INFO:teuthology.orchestra.run.smithi152.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-16T04:38:13.506 INFO:teuthology.orchestra.run.smithi152.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-16T04:38:13.510 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:38:13.510 INFO:teuthology.orchestra.run.smithi152.stdout:Total 15 MB/s | 30 MB 00:02 2024-09-16T04:38:13.665 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:38:13.704 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:38:13.704 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:38:14.188 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:38:14.189 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:38:15.559 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:38:15.617 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T04:38:15.659 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T04:38:15.695 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T04:38:15.767 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T04:38:15.818 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T04:38:15.851 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T04:38:15.952 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T04:38:16.030 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T04:38:16.070 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T04:38:16.105 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T04:38:16.135 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T04:38:16.178 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T04:38:16.228 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T04:38:16.277 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T04:38:16.320 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T04:38:16.354 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T04:38:16.403 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T04:38:16.435 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T04:38:16.509 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T04:38:16.664 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T04:38:16.751 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T04:38:16.804 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T04:38:16.843 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T04:38:16.899 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T04:38:16.965 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T04:38:17.159 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T04:38:17.213 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T04:38:17.291 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T04:38:17.389 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T04:38:17.440 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T04:38:17.562 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T04:38:17.717 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 32/37 2024-09-16T04:38:17.747 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:38:17.773 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:38:17.773 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:38:17.773 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T04:38:17.774 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:38:17.774 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:38:17.774 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:19.206 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:38:19.233 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:38:19.234 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:38:19.234 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T04:38:19.234 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:38:19.234 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:38:19.234 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:19.673 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:38:19.697 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:38:19.697 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:38:19.698 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T04:38:19.698 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:38:19.698 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:38:19.698 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:19.924 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:38:19.947 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:38:19.948 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:38:19.948 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T04:38:19.948 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:38:19.948 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:38:19.948 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:20.012 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:38:21.819 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:38:21.819 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/37 2024-09-16T04:38:21.819 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2/37 2024-09-16T04:38:21.819 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 3/37 2024-09-16T04:38:21.820 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 4/37 2024-09-16T04:38:21.820 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 5/37 2024-09-16T04:38:21.820 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 6/37 2024-09-16T04:38:21.820 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T04:38:21.820 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T04:38:21.820 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T04:38:21.820 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T04:38:21.820 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T04:38:21.820 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T04:38:21.820 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T04:38:21.820 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T04:38:21.820 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T04:38:21.820 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T04:38:21.820 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T04:38:21.821 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T04:38:21.821 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T04:38:21.821 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T04:38:21.821 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T04:38:21.821 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T04:38:21.821 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T04:38:21.821 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T04:38:21.821 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T04:38:21.821 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T04:38:21.821 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T04:38:21.821 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T04:38:21.822 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T04:38:21.822 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T04:38:21.823 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T04:38:21.823 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T04:38:21.823 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T04:38:21.823 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T04:38:21.823 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T04:38:21.823 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T04:38:22.816 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T04:38:22.816 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:22.816 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:38:22.816 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:38:22.816 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:38:22.816 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:38:22.816 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:38:22.816 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:38:22.817 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:38:22.817 INFO:teuthology.orchestra.run.smithi152.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T04:38:22.817 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T04:38:22.817 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T04:38:22.817 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T04:38:22.817 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T04:38:22.817 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T04:38:22.817 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T04:38:22.817 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T04:38:22.817 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T04:38:22.817 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T04:38:22.817 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T04:38:22.817 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T04:38:22.818 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T04:38:22.818 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T04:38:22.818 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T04:38:22.818 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T04:38:22.818 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T04:38:22.818 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T04:38:22.818 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T04:38:22.818 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T04:38:22.818 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T04:38:22.818 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T04:38:22.818 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T04:38:22.818 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T04:38:22.818 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T04:38:22.818 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T04:38:22.819 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T04:38:22.819 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T04:38:22.819 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T04:38:22.819 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T04:38:22.819 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T04:38:22.819 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:22.819 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:38:23.210 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-base 2024-09-16T04:38:23.779 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:52 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:38:23.868 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:38:23.910 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:38:23.911 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T04:38:23.911 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:38:23.984 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install cephadm 2024-09-16T04:38:24.550 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:38:24.681 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:38:24.682 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:24.682 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T04:38:24.682 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:24.682 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:38:24.682 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T04:38:24.682 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:24.682 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:38:24.682 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:24.682 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-16T04:38:24.682 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:24.682 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 765 k 2024-09-16T04:38:24.682 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 771 k 2024-09-16T04:38:24.683 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:38:24.972 INFO:teuthology.orchestra.run.smithi152.stdout:cephadm-19.1.1-326.g8921c8ef.el9.noarch.rpm 2.6 MB/s | 765 kB 00:00 2024-09-16T04:38:24.972 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:38:24.972 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.6 MB/s | 765 kB 00:00 2024-09-16T04:38:24.973 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:38:24.978 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:38:24.978 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:38:24.984 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:38:24.984 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:38:25.063 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:38:25.540 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:38:25.626 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:38:25.977 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:38:26.235 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:38:26.235 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:26.236 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:38:26.236 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:38:26.236 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:26.236 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:38:26.441 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T04:38:27.012 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:38:27.145 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:38:27.146 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:27.147 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-09-16T04:38:27.147 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:27.147 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:38:27.147 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T04:38:27.147 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:27.147 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:38:27.147 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:27.147 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-16T04:38:27.147 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:27.147 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 145 k 2024-09-16T04:38:27.147 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 435 k 2024-09-16T04:38:27.147 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:38:27.352 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-immutable-object-cache-19.1.1-326.g8921c8e 710 kB/s | 145 kB 00:00 2024-09-16T04:38:27.352 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:38:27.353 INFO:teuthology.orchestra.run.smithi152.stdout:Total 705 kB/s | 145 kB 00:00 2024-09-16T04:38:27.353 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:38:27.360 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:38:27.360 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:38:27.417 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:38:27.417 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:38:27.579 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:38:27.661 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:38:27.687 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:38:27.687 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:38:27.687 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T04:38:27.687 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:28.601 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:38:28.602 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:28.602 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:38:28.602 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:38:28.602 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:28.602 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:38:28.837 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr 2024-09-16T04:38:29.412 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:38:29.503 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:38:29.544 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:38:29.545 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T04:38:29.546 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:38:29.625 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T04:38:30.185 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:38:30.317 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:38:30.318 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T04:38:30.318 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:30.318 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:38:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T04:38:30.318 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T04:38:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T04:38:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T04:38:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T04:38:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T04:38:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T04:38:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T04:38:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T04:38:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T04:38:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T04:38:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T04:38:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T04:38:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:30.319 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:38:30.319 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:Install 12 Packages 2024-09-16T04:38:30.320 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 8.7 M 2024-09-16T04:38:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 104 M 2024-09-16T04:38:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:38:30.672 INFO:teuthology.orchestra.run.smithi152.stdout:(1/12): ceph-prometheus-alerts-19.1.1-326.g8921 113 kB/s | 16 kB 00:00 2024-09-16T04:38:30.697 INFO:teuthology.orchestra.run.smithi152.stdout:(2/12): ceph-grafana-dashboards-19.1.1-326.g892 186 kB/s | 32 kB 00:00 2024-09-16T04:38:30.872 INFO:teuthology.orchestra.run.smithi152.stdout:(3/12): ceph-mgr-dashboard-19.1.1-326.g8921c8ef 10 MB/s | 3.6 MB 00:00 2024-09-16T04:38:31.089 INFO:teuthology.orchestra.run.smithi152.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.9 MB/s | 863 kB 00:00 2024-09-16T04:38:31.156 INFO:teuthology.orchestra.run.smithi152.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.0 MB/s | 548 kB 00:00 2024-09-16T04:38:31.173 INFO:teuthology.orchestra.run.smithi152.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 1.2 MB/s | 19 kB 00:00 2024-09-16T04:38:31.265 INFO:teuthology.orchestra.run.smithi152.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 22 MB/s | 2.0 MB 00:00 2024-09-16T04:38:31.315 INFO:teuthology.orchestra.run.smithi152.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-09-16T04:38:31.340 INFO:teuthology.orchestra.run.smithi152.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-16T04:38:31.374 INFO:teuthology.orchestra.run.smithi152.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.5 MB/s | 188 kB 00:00 2024-09-16T04:38:31.507 INFO:teuthology.orchestra.run.smithi152.stdout:(11/12): python3-protobuf-3.14.0-13.el9.noarch. 332 kB/s | 269 kB 00:00 2024-09-16T04:38:31.699 INFO:teuthology.orchestra.run.smithi152.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:01 2024-09-16T04:38:31.703 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:38:31.703 INFO:teuthology.orchestra.run.smithi152.stdout:Total 6.3 MB/s | 8.7 MB 00:01 2024-09-16T04:38:31.751 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:38:31.768 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:38:31.768 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:38:31.974 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:38:31.975 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:38:32.520 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:38:32.608 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T04:38:32.643 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T04:38:32.690 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T04:38:32.737 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T04:38:32.834 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T04:38:32.970 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T04:38:33.063 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T04:38:33.108 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T04:38:33.145 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:38:33.176 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 10/12 2024-09-16T04:38:34.230 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 11/12 2024-09-16T04:38:34.277 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:38:35.458 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:38:35.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 1/12 2024-09-16T04:38:35.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 2/12 2024-09-16T04:38:35.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 3/12 2024-09-16T04:38:35.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T04:38:35.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T04:38:35.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T04:38:35.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T04:38:35.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T04:38:35.458 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:38:35.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T04:38:35.459 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T04:38:35.777 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T04:38:35.777 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:35.777 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:38:35.777 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T04:38:35.777 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:38:35.778 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:38:35.778 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:38:35.778 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T04:38:35.778 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T04:38:35.778 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T04:38:35.778 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T04:38:35.778 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T04:38:35.778 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T04:38:35.778 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T04:38:35.778 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T04:38:35.778 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:35.778 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:38:36.053 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T04:38:36.610 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:05 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:38:36.743 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:38:36.744 INFO:teuthology.orchestra.run.smithi152.stdout:====================================================================================== 2024-09-16T04:38:36.745 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T04:38:36.745 INFO:teuthology.orchestra.run.smithi152.stdout:====================================================================================== 2024-09-16T04:38:36.745 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:38:36.745 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T04:38:36.745 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T04:38:36.745 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T04:38:36.745 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T04:38:36.745 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T04:38:36.745 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T04:38:36.745 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T04:38:36.745 INFO:teuthology.orchestra.run.smithi152.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T04:38:36.746 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T04:38:36.746 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T04:38:36.746 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T04:38:36.746 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T04:38:36.746 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T04:38:36.746 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:36.746 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:38:36.746 INFO:teuthology.orchestra.run.smithi152.stdout:====================================================================================== 2024-09-16T04:38:36.746 INFO:teuthology.orchestra.run.smithi152.stdout:Install 12 Packages 2024-09-16T04:38:36.746 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:36.747 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 38 M 2024-09-16T04:38:36.747 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 212 M 2024-09-16T04:38:36.747 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:38:37.498 INFO:teuthology.orchestra.run.smithi152.stdout:(1/12): ceph-mgr-diskprediction-local-19.1.1-32 16 MB/s | 7.4 MB 00:00 2024-09-16T04:38:37.590 INFO:teuthology.orchestra.run.smithi152.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 358 kB/s | 33 kB 00:00 2024-09-16T04:38:37.748 INFO:teuthology.orchestra.run.smithi152.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 19 MB/s | 3.0 MB 00:00 2024-09-16T04:38:37.824 INFO:teuthology.orchestra.run.smithi152.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 242 kB/s | 18 kB 00:00 2024-09-16T04:38:37.874 INFO:teuthology.orchestra.run.smithi152.stdout:(5/12): libquadmath-11.5.0-2.el9.x86_64.rpm 230 kB/s | 191 kB 00:00 2024-09-16T04:38:37.891 INFO:teuthology.orchestra.run.smithi152.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 576 kB/s | 38 kB 00:00 2024-09-16T04:38:37.966 INFO:teuthology.orchestra.run.smithi152.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-09-16T04:38:38.025 INFO:teuthology.orchestra.run.smithi152.stdout:(8/12): libgfortran-11.5.0-2.el9.x86_64.rpm 816 kB/s | 801 kB 00:00 2024-09-16T04:38:38.175 INFO:teuthology.orchestra.run.smithi152.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 5.1 MB 00:00 2024-09-16T04:38:38.267 INFO:teuthology.orchestra.run.smithi152.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.0 MB/s | 250 kB 00:00 2024-09-16T04:38:38.350 INFO:teuthology.orchestra.run.smithi152.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 10 MB/s | 4.9 MB 00:00 2024-09-16T04:38:38.584 INFO:teuthology.orchestra.run.smithi152.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-09-16T04:38:38.585 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:38:38.585 INFO:teuthology.orchestra.run.smithi152.stdout:Total 21 MB/s | 38 MB 00:01 2024-09-16T04:38:38.944 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:38:38.970 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:38:38.970 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:38:39.417 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:38:39.418 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:38:40.221 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:38:40.293 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T04:38:40.342 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:38:40.441 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:38:40.483 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T04:38:40.669 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T04:38:40.698 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T04:38:40.786 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T04:38:41.448 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T04:38:41.562 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T04:38:42.821 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:38:43.857 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T04:38:43.888 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:38:46.642 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:38:46.642 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 1/12 2024-09-16T04:38:46.642 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:38:46.642 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:38:46.643 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T04:38:46.643 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T04:38:46.643 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T04:38:46.643 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T04:38:46.643 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T04:38:46.643 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T04:38:46.643 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:38:46.643 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T04:38:47.067 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T04:38:47.068 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:47.068 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:38:47.068 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:38:47.068 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T04:38:47.068 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T04:38:47.068 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T04:38:47.068 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T04:38:47.068 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T04:38:47.068 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T04:38:47.068 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T04:38:47.068 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T04:38:47.069 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T04:38:47.069 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T04:38:47.069 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T04:38:47.069 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:47.069 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:38:47.441 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-rook 2024-09-16T04:38:48.022 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:17 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:38:48.156 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:38:48.157 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:48.157 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T04:38:48.157 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:48.157 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:38:48.157 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T04:38:48.158 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T04:38:48.158 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T04:38:48.158 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T04:38:48.158 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T04:38:48.158 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T04:38:48.158 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T04:38:48.158 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T04:38:48.158 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T04:38:48.158 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T04:38:48.158 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T04:38:48.158 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T04:38:48.158 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T04:38:48.158 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T04:38:48.158 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:48.159 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:38:48.159 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:48.159 INFO:teuthology.orchestra.run.smithi152.stdout:Install 13 Packages 2024-09-16T04:38:48.159 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:48.159 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 2.2 M 2024-09-16T04:38:48.159 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 26 M 2024-09-16T04:38:48.160 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:38:48.563 INFO:teuthology.orchestra.run.smithi152.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 218 kB/s | 26 kB 00:00 2024-09-16T04:38:48.587 INFO:teuthology.orchestra.run.smithi152.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 134 kB/s | 19 kB 00:00 2024-09-16T04:38:48.613 INFO:teuthology.orchestra.run.smithi152.stdout:(3/13): ceph-mgr-rook-19.1.1-326.g8921c8ef.el9. 293 kB/s | 49 kB 00:00 2024-09-16T04:38:48.730 INFO:teuthology.orchestra.run.smithi152.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-16T04:38:48.755 INFO:teuthology.orchestra.run.smithi152.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 222 kB 00:00 2024-09-16T04:38:48.789 INFO:teuthology.orchestra.run.smithi152.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 916 kB/s | 54 kB 00:00 2024-09-16T04:38:48.814 INFO:teuthology.orchestra.run.smithi152.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-09-16T04:38:48.847 INFO:teuthology.orchestra.run.smithi152.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 349 kB/s | 32 kB 00:00 2024-09-16T04:38:48.881 INFO:teuthology.orchestra.run.smithi152.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 156 kB/s | 14 kB 00:00 2024-09-16T04:38:48.915 INFO:teuthology.orchestra.run.smithi152.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-16T04:38:48.949 INFO:teuthology.orchestra.run.smithi152.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-16T04:38:48.974 INFO:teuthology.orchestra.run.smithi152.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.4 MB/s | 220 kB 00:00 2024-09-16T04:38:49.007 INFO:teuthology.orchestra.run.smithi152.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-16T04:38:49.010 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:38:49.010 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.6 MB/s | 2.2 MB 00:00 2024-09-16T04:38:49.075 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:38:49.090 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:38:49.091 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:38:49.180 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:38:49.180 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:38:49.681 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:38:49.838 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T04:38:49.891 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T04:38:49.955 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T04:38:50.001 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T04:38:50.051 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T04:38:50.161 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T04:38:50.263 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T04:38:50.318 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T04:38:51.033 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T04:38:51.083 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T04:38:51.116 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T04:38:51.165 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T04:38:51.186 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:38:52.013 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:38:52.013 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 1/13 2024-09-16T04:38:52.013 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T04:38:52.013 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T04:38:52.013 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T04:38:52.013 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T04:38:52.014 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T04:38:52.014 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T04:38:52.014 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T04:38:52.014 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T04:38:52.014 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T04:38:52.014 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T04:38:52.014 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T04:38:52.359 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T04:38:52.359 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:52.359 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:38:52.360 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:38:52.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T04:38:52.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T04:38:52.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T04:38:52.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T04:38:52.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T04:38:52.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T04:38:52.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T04:38:52.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T04:38:52.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T04:38:52.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T04:38:52.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T04:38:52.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T04:38:52.361 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:52.361 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:38:52.659 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T04:38:53.225 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:22 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:38:53.358 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:38:53.359 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:53.359 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T04:38:53.359 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:53.359 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:38:53.359 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T04:38:53.360 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T04:38:53.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T04:38:53.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T04:38:53.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T04:38:53.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T04:38:53.360 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T04:38:53.360 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:53.360 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:38:53.360 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:53.360 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2024-09-16T04:38:53.360 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:53.361 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 7.0 M 2024-09-16T04:38:53.361 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 33 M 2024-09-16T04:38:53.361 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:38:54.041 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-16T04:38:54.066 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): ceph-mgr-cephadm-19.1.1-326.g8921c8ef.el 619 kB/s | 146 kB 00:00 2024-09-16T04:38:54.125 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 988 kB/s | 58 kB 00:00 2024-09-16T04:38:54.167 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 1.7 MB/s | 74 kB 00:00 2024-09-16T04:38:54.201 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.4 MB/s | 548 kB 00:00 2024-09-16T04:38:54.459 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 9.5 MB/s | 6.0 MB 00:00 2024-09-16T04:38:54.462 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:38:54.462 INFO:teuthology.orchestra.run.smithi152.stdout:Total 6.4 MB/s | 7.0 MB 00:01 2024-09-16T04:38:54.534 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:38:54.546 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:38:54.546 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:38:54.645 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:38:54.645 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:38:54.957 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:38:55.133 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T04:38:55.186 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T04:38:55.704 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T04:38:55.789 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T04:38:55.851 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T04:38:55.881 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:38:56.734 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:38:56.734 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/6 2024-09-16T04:38:56.734 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T04:38:56.734 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T04:38:56.734 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T04:38:56.734 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T04:38:57.014 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T04:38:57.014 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:57.014 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:38:57.014 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:38:57.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T04:38:57.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T04:38:57.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T04:38:57.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T04:38:57.015 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T04:38:57.015 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:57.015 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:38:57.290 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-fuse 2024-09-16T04:38:57.856 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:26 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:38:57.992 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:38:57.993 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:57.993 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T04:38:57.993 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:57.993 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:38:57.993 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T04:38:57.993 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-16T04:38:57.993 INFO:teuthology.orchestra.run.smithi152.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T04:38:57.993 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:57.993 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:38:57.993 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:38:57.994 INFO:teuthology.orchestra.run.smithi152.stdout:Install 2 Packages 2024-09-16T04:38:57.994 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:57.994 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 976 k 2024-09-16T04:38:57.994 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 2.9 M 2024-09-16T04:38:57.994 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:38:58.169 INFO:teuthology.orchestra.run.smithi152.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 535 kB/s | 80 kB 00:00 2024-09-16T04:38:58.294 INFO:teuthology.orchestra.run.smithi152.stdout:(2/2): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86_6 3.2 MB/s | 896 kB 00:00 2024-09-16T04:38:58.294 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:38:58.295 INFO:teuthology.orchestra.run.smithi152.stdout:Total 3.2 MB/s | 976 kB 00:00 2024-09-16T04:38:58.306 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:38:58.329 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:38:58.329 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:38:58.411 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:38:58.411 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:38:58.605 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:38:58.737 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T04:38:58.802 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:38:59.474 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:38:59.474 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/2 2024-09-16T04:38:59.758 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T04:38:59.758 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:59.758 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:38:59.758 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T04:38:59.758 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:38:59.758 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:39:00.033 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-volume 2024-09-16T04:39:00.612 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:29 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:39:00.748 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:39:00.749 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:00.749 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-16T04:39:00.749 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:00.749 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:39:00.750 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T04:39:00.750 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:00.750 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:39:00.750 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:00.750 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-16T04:39:00.750 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:00.750 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 285 k 2024-09-16T04:39:00.750 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 1.3 M 2024-09-16T04:39:00.750 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:39:01.189 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-volume-19.1.1-326.g8921c8ef.el9.noarch.rpm 650 kB/s | 285 kB 00:00 2024-09-16T04:39:01.190 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:39:01.190 INFO:teuthology.orchestra.run.smithi152.stdout:Total 647 kB/s | 285 kB 00:00 2024-09-16T04:39:01.190 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:39:01.199 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:39:01.199 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:39:01.225 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:39:01.225 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:39:01.452 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:39:01.524 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:39:01.541 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:39:01.541 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:39:01.541 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T04:39:01.541 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:02.556 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:39:02.556 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:02.556 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:39:02.556 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:39:02.557 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:02.557 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:39:02.886 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados-devel 2024-09-16T04:39:03.465 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:39:03.599 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:39:03.600 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:03.600 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-09-16T04:39:03.600 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:03.600 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:39:03.600 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T04:39:03.600 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:03.600 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:39:03.600 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:03.600 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-16T04:39:03.600 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:03.601 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 127 k 2024-09-16T04:39:03.601 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 456 k 2024-09-16T04:39:03.601 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:39:03.810 INFO:teuthology.orchestra.run.smithi152.stdout:librados-devel-19.1.1-326.g8921c8ef.el9.x86_64. 606 kB/s | 127 kB 00:00 2024-09-16T04:39:03.810 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:39:03.810 INFO:teuthology.orchestra.run.smithi152.stdout:Total 603 kB/s | 127 kB 00:00 2024-09-16T04:39:03.811 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:39:03.816 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:39:03.816 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:39:03.885 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:39:03.886 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:39:04.061 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:39:04.157 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:39:04.585 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:39:04.849 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:39:04.849 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:04.849 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:39:04.849 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:39:04.849 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:04.849 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:39:05.095 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs2 2024-09-16T04:39:05.681 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:39:05.774 INFO:teuthology.orchestra.run.smithi152.stdout:Package libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:39:05.820 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:39:05.821 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T04:39:05.821 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:39:05.898 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs-devel 2024-09-16T04:39:06.467 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:39:06.601 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:39:06.601 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:06.601 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-09-16T04:39:06.601 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:06.601 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:39:06.602 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T04:39:06.602 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:06.602 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:39:06.602 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:06.602 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-16T04:39:06.602 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:06.602 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 32 k 2024-09-16T04:39:06.602 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 142 k 2024-09-16T04:39:06.602 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:39:06.745 INFO:teuthology.orchestra.run.smithi152.stdout:libcephfs-devel-19.1.1-326.g8921c8ef.el9.x86_64 222 kB/s | 32 kB 00:00 2024-09-16T04:39:06.746 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:39:06.746 INFO:teuthology.orchestra.run.smithi152.stdout:Total 220 kB/s | 32 kB 00:00 2024-09-16T04:39:06.746 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:39:06.750 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:39:06.750 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:39:06.772 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:39:06.773 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:39:06.865 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:39:06.967 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:39:07.340 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:39:07.586 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:39:07.586 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:07.586 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:39:07.586 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:39:07.586 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:07.586 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:39:07.775 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados2 2024-09-16T04:39:08.341 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:39:08.431 INFO:teuthology.orchestra.run.smithi152.stdout:Package librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:39:08.475 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:39:08.476 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T04:39:08.476 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:39:08.554 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librbd1 2024-09-16T04:39:09.124 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:39:09.217 INFO:teuthology.orchestra.run.smithi152.stdout:Package librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:39:09.262 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:39:09.263 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T04:39:09.263 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:39:09.325 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rados 2024-09-16T04:39:09.892 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:39:09.982 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:39:10.027 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:39:10.028 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T04:39:10.028 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:39:10.103 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rgw 2024-09-16T04:39:10.674 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:39:10.765 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:39:10.809 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:39:10.810 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T04:39:10.810 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:39:10.896 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-cephfs 2024-09-16T04:39:11.462 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:39:11.553 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:39:11.596 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:39:11.597 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T04:39:11.598 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:39:11.670 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rbd 2024-09-16T04:39:12.241 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:39:12.332 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:39:12.376 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:39:12.377 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-16T04:39:12.377 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:39:12.441 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-fuse 2024-09-16T04:39:13.025 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:39:13.158 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:39:13.159 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:13.159 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-09-16T04:39:13.159 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:13.159 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:39:13.159 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T04:39:13.159 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:13.159 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:39:13.160 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:13.160 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-16T04:39:13.160 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:13.160 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 85 k 2024-09-16T04:39:13.160 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 231 k 2024-09-16T04:39:13.160 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:39:13.398 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-fuse-19.1.1-326.g8921c8ef.el9.x86_64.rpm 358 kB/s | 85 kB 00:00 2024-09-16T04:39:13.399 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:39:13.399 INFO:teuthology.orchestra.run.smithi152.stdout:Total 355 kB/s | 85 kB 00:00 2024-09-16T04:39:13.399 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:39:13.406 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:39:13.406 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:39:13.469 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:39:13.470 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:39:13.627 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:39:13.721 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:39:14.117 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:39:14.381 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:39:14.381 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:14.381 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:39:14.381 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:39:14.381 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:14.381 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:39:14.708 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-mirror 2024-09-16T04:39:15.303 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:39:15.445 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:39:15.445 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:15.445 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-09-16T04:39:15.445 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:15.446 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:39:15.446 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T04:39:15.446 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:15.446 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:39:15.446 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:15.446 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-16T04:39:15.446 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:15.446 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 3.1 M 2024-09-16T04:39:15.446 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 13 M 2024-09-16T04:39:15.446 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:39:16.318 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-mirror-19.1.1-326.g8921c8ef.el9.x86_64.rpm 3.6 MB/s | 3.1 MB 00:00 2024-09-16T04:39:16.318 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:39:16.319 INFO:teuthology.orchestra.run.smithi152.stdout:Total 3.6 MB/s | 3.1 MB 00:00 2024-09-16T04:39:16.319 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:39:16.329 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:39:16.329 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:39:16.410 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:39:16.410 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:39:16.860 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:39:16.927 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:39:16.953 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:39:16.953 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:39:16.953 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T04:39:16.953 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:39:16.953 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:39:16.954 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:18.136 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:39:18.136 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:18.136 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:39:18.136 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:39:18.136 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:18.137 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:39:18.378 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-nbd 2024-09-16T04:39:18.963 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 04:37:31 AM UTC. 2024-09-16T04:39:19.098 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-16T04:39:19.099 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:19.099 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-09-16T04:39:19.099 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:19.099 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-16T04:39:19.099 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T04:39:19.099 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:19.099 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-16T04:39:19.099 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-16T04:39:19.100 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-16T04:39:19.100 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:19.100 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 170 k 2024-09-16T04:39:19.100 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 494 k 2024-09-16T04:39:19.100 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-16T04:39:19.312 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_64.rpm 803 kB/s | 170 kB 00:00 2024-09-16T04:39:19.313 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:39:19.313 INFO:teuthology.orchestra.run.smithi152.stdout:Total 798 kB/s | 170 kB 00:00 2024-09-16T04:39:19.313 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-16T04:39:19.321 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-16T04:39:19.321 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-16T04:39:19.385 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-16T04:39:19.385 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-16T04:39:19.536 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-16T04:39:19.635 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:39:20.079 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:39:20.336 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:39:20.336 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:20.336 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-16T04:39:20.336 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:39:20.336 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:20.336 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-16T04:39:20.612 DEBUG:teuthology.parallel:result is None 2024-09-16T04:39:20.612 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:39:20.761 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T04:39:20.783 INFO:teuthology.orchestra.run.smithi152.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T04:39:20.784 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T04:39:20.784 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T04:39:20.786 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T04:39:20.786 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:20.786 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T04:39:20.849 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T04:39:20.849 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:20.849 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T04:39:20.914 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T04:39:20.978 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T04:39:20.978 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:20.979 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T04:39:21.043 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T04:39:21.108 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T04:39:21.108 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:21.108 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T04:39:21.172 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T04:39:21.238 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T04:39:21.333 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T04:39:21.333 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /var/log/ceph 2024-09-16T04:39:21.358 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T04:39:21.359 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T04:39:21.424 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T04:39:21.424 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T04:39:21.498 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T04:39:21.498 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T04:39:21.498 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906037', 'branch': 'squid', 'description': 'rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7906037', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 700, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 5616, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'sleep_before_teardown': 0, 'subset': '15927/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4', 'targets': {'smithi152.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJtpr4W0XE9G5N7sApdfuavl8bPfa9QcKIetLpZaCAVyMA+r39WFdgAXnkxkvI+nHgJsLR6oEXgantmBIw5PQ7s='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'create_rbd_pool': False, 'log-ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'cluster': 'ceph'}}, {'ec_lost_unfound': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_21:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T04:39:21.499 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T04:39:21.546 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T04:39:21.612 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:21.612 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T04:39:21.668 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T04:39:21.668 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2024-09-16T04:39:21.724 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T04:39:21.725 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:21.725 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-16T04:39:21.725 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:21.725 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:39:21.725 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-16 04:39:17.197231481 +0000 2024-09-16T04:39:21.725 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-16 04:36:53.958085379 +0000 2024-09-16T04:39:21.725 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-16 04:36:53.958085379 +0000 2024-09-16T04:39:21.725 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-09-16 04:36:53.958085379 +0000 2024-09-16T04:39:21.726 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T04:39:21.787 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-16T04:39:21.788 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-16T04:39:21.788 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000209911 s, 2.4 MB/s 2024-09-16T04:39:21.789 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T04:39:21.846 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2024-09-16T04:39:21.901 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T04:39:21.902 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:21.902 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-16T04:39:21.902 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:21.902 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:39:21.902 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-16 04:39:17.197231481 +0000 2024-09-16T04:39:21.902 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-16 04:36:54.279082009 +0000 2024-09-16T04:39:21.902 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-16 04:36:54.279082009 +0000 2024-09-16T04:39:21.902 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-09-16 04:36:54.279082009 +0000 2024-09-16T04:39:21.902 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000187007 s, 2.7 MB/s 2024-09-16T04:39:21.966 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T04:39:22.023 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2024-09-16T04:39:22.078 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T04:39:22.079 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:22.079 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-16T04:39:22.079 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:22.079 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:39:22.079 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-16 04:39:17.198231469 +0000 2024-09-16T04:39:22.079 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-16 04:36:54.599078650 +0000 2024-09-16T04:39:22.079 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-16 04:36:54.599078650 +0000 2024-09-16T04:39:22.079 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-09-16 04:36:54.599078650 +0000 2024-09-16T04:39:22.079 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T04:39:22.142 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-16T04:39:22.142 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-16T04:39:22.142 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000172248 s, 3.0 MB/s 2024-09-16T04:39:22.144 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T04:39:22.201 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2024-09-16T04:39:22.256 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T04:39:22.256 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:22.257 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-16T04:39:22.257 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:22.257 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:39:22.257 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-16 04:39:17.198231469 +0000 2024-09-16T04:39:22.257 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-16 04:36:54.925075228 +0000 2024-09-16T04:39:22.257 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-16 04:36:54.925075228 +0000 2024-09-16T04:39:22.258 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-09-16 04:36:54.925075228 +0000 2024-09-16T04:39:22.258 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T04:39:22.320 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-16T04:39:22.320 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-16T04:39:22.320 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000186752 s, 2.7 MB/s 2024-09-16T04:39:22.321 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T04:39:22.378 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T04:39:22.378 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-16T04:39:22.378 INFO:tasks.ceph:Generating config... 2024-09-16T04:39:22.379 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T04:39:22.379 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-16T04:39:22.379 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-16T04:39:22.379 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-16T04:39:22.379 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T04:39:22.379 INFO:tasks.ceph:[global] ms type = async 2024-09-16T04:39:22.380 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T04:39:22.380 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T04:39:22.380 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T04:39:22.380 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T04:39:22.380 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T04:39:22.380 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T04:39:22.380 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T04:39:22.380 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-09-16T04:39:22.380 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-16T04:39:22.380 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T04:39:22.380 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T04:39:22.380 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T04:39:22.380 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T04:39:22.380 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T04:39:22.381 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T04:39:22.381 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T04:39:22.381 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T04:39:22.381 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T04:39:22.381 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T04:39:22.381 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T04:39:22.381 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T04:39:22.381 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T04:39:22.382 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T04:39:22.382 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T04:39:22.382 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T04:39:22.382 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T04:39:22.382 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T04:39:22.382 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T04:39:22.382 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T04:39:22.476 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T04:39:22.479 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T04:39:22.522 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:39:22.587 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.152'), ('mon.b', '[v2:172.21.15.152:3301,v1:172.21.15.152:6790]'), ('mon.c', '[v2:172.21.15.152:3302,v1:172.21.15.152:6791]')] 2024-09-16T04:39:22.587 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.152,[v2:172.21.15.152:3301,v1:172.21.15.152:6790],[v2:172.21.15.152:3302,v1:172.21.15.152:6791]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-16T04:39:22.588 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:22.588 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T04:39:22.644 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.152 --addv b '[v2:172.21.15.152:3301,v1:172.21.15.152:6790]' --addv c '[v2:172.21.15.152:3302,v1:172.21.15.152:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:39:22.733 INFO:teuthology.orchestra.run.smithi152.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:39:22.733 INFO:teuthology.orchestra.run.smithi152.stdout:monmaptool: generated fsid 47829f7b-b43c-44e0-b4e2-77e6a22c542c 2024-09-16T04:39:22.733 INFO:teuthology.orchestra.run.smithi152.stdout:setting min_mon_release = quincy 2024-09-16T04:39:22.733 INFO:teuthology.orchestra.run.smithi152.stdout:epoch 0 2024-09-16T04:39:22.734 INFO:teuthology.orchestra.run.smithi152.stdout:fsid 47829f7b-b43c-44e0-b4e2-77e6a22c542c 2024-09-16T04:39:22.734 INFO:teuthology.orchestra.run.smithi152.stdout:last_changed 2024-09-16T04:39:22.732706+0000 2024-09-16T04:39:22.734 INFO:teuthology.orchestra.run.smithi152.stdout:created 2024-09-16T04:39:22.732706+0000 2024-09-16T04:39:22.734 INFO:teuthology.orchestra.run.smithi152.stdout:min_mon_release 17 (quincy) 2024-09-16T04:39:22.734 INFO:teuthology.orchestra.run.smithi152.stdout:election_strategy: 3 2024-09-16T04:39:22.734 INFO:teuthology.orchestra.run.smithi152.stdout:0: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.a 2024-09-16T04:39:22.734 INFO:teuthology.orchestra.run.smithi152.stdout:1: [v2:172.21.15.152:3301/0,v1:172.21.15.152:6790/0] mon.b 2024-09-16T04:39:22.734 INFO:teuthology.orchestra.run.smithi152.stdout:2: [v2:172.21.15.152:3302/0,v1:172.21.15.152:6791/0] mon.c 2024-09-16T04:39:22.734 INFO:teuthology.orchestra.run.smithi152.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T04:39:22.735 DEBUG:teuthology.orchestra.run.smithi152:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T04:39:22.753 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 47829f7b-b43c-44e0-b4e2-77e6a22c542c... 2024-09-16T04:39:22.754 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T04:39:22.836 INFO:teuthology.orchestra.run.smithi152.stdout:[global] 2024-09-16T04:39:22.836 INFO:teuthology.orchestra.run.smithi152.stdout: chdir = "" 2024-09-16T04:39:22.836 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T04:39:22.837 INFO:teuthology.orchestra.run.smithi152.stdout: auth supported = cephx 2024-09-16T04:39:22.837 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.837 INFO:teuthology.orchestra.run.smithi152.stdout: filestore xattr use omap = true 2024-09-16T04:39:22.837 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.837 INFO:teuthology.orchestra.run.smithi152.stdout: mon clock drift allowed = 1.000 2024-09-16T04:39:22.837 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.837 INFO:teuthology.orchestra.run.smithi152.stdout: osd crush chooseleaf type = 0 2024-09-16T04:39:22.837 INFO:teuthology.orchestra.run.smithi152.stdout: auth debug = true 2024-09-16T04:39:22.837 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.837 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on old message = true 2024-09-16T04:39:22.837 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on bug = true 2024-09-16T04:39:22.837 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.837 INFO:teuthology.orchestra.run.smithi152.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T04:39:22.837 INFO:teuthology.orchestra.run.smithi152.stdout: mon pg warn max object skew = 0 2024-09-16T04:39:22.838 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.838 INFO:teuthology.orchestra.run.smithi152.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T04:39:22.838 INFO:teuthology.orchestra.run.smithi152.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T04:39:22.838 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.838 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default size = 2 2024-09-16T04:39:22.838 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.838 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow primary affinity = true 2024-09-16T04:39:22.838 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow pg remap = true 2024-09-16T04:39:22.838 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on legacy crush tunables = false 2024-09-16T04:39:22.839 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on crush straw calc version zero = false 2024-09-16T04:39:22.839 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on no sortbitwise = false 2024-09-16T04:39:22.839 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on osd down out interval zero = false 2024-09-16T04:39:22.839 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on too few osds = false 2024-09-16T04:39:22.839 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T04:39:22.839 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T04:39:22.839 INFO:teuthology.orchestra.run.smithi152.stdout: mon_allow_pool_size_one = true 2024-09-16T04:39:22.839 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.839 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T04:39:22.839 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.839 INFO:teuthology.orchestra.run.smithi152.stdout: osd default data pool replay window = 5 2024-09-16T04:39:22.839 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.839 INFO:teuthology.orchestra.run.smithi152.stdout: mon allow pool delete = true 2024-09-16T04:39:22.840 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.840 INFO:teuthology.orchestra.run.smithi152.stdout: mon cluster log file level = debug 2024-09-16T04:39:22.840 INFO:teuthology.orchestra.run.smithi152.stdout: debug asserts on shutdown = true 2024-09-16T04:39:22.840 INFO:teuthology.orchestra.run.smithi152.stdout: mon health detail to clog = false 2024-09-16T04:39:22.840 INFO:teuthology.orchestra.run.smithi152.stdout: mon host = "172.21.15.152,[v2:172.21.15.152:3301,v1:172.21.15.152:6790],[v2:172.21.15.152:3302,v1:172.21.15.152:6791]" 2024-09-16T04:39:22.840 INFO:teuthology.orchestra.run.smithi152.stdout: mon client directed command retry = 5 2024-09-16T04:39:22.840 INFO:teuthology.orchestra.run.smithi152.stdout: mon election default strategy = 3 2024-09-16T04:39:22.840 INFO:teuthology.orchestra.run.smithi152.stdout: ms bind msgr1 = False 2024-09-16T04:39:22.840 INFO:teuthology.orchestra.run.smithi152.stdout: ms bind msgr2 = True 2024-09-16T04:39:22.840 INFO:teuthology.orchestra.run.smithi152.stdout: ms inject socket failures = 5000 2024-09-16T04:39:22.840 INFO:teuthology.orchestra.run.smithi152.stdout: ms type = async 2024-09-16T04:39:22.841 INFO:teuthology.orchestra.run.smithi152.stdout: fsid = 47829f7b-b43c-44e0-b4e2-77e6a22c542c 2024-09-16T04:39:22.841 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.841 INFO:teuthology.orchestra.run.smithi152.stdout:[osd] 2024-09-16T04:39:22.841 INFO:teuthology.orchestra.run.smithi152.stdout: osd journal size = 100 2024-09-16T04:39:22.841 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.841 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub load threshold = 5.0 2024-09-16T04:39:22.843 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub max interval = 600 2024-09-16T04:39:22.843 INFO:teuthology.orchestra.run.smithi152.stdout: osd mclock profile = high_recovery_ops 2024-09-16T04:39:22.843 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.843 INFO:teuthology.orchestra.run.smithi152.stdout: osd recover clone overlap = true 2024-09-16T04:39:22.843 INFO:teuthology.orchestra.run.smithi152.stdout: osd recovery max chunk = 1048576 2024-09-16T04:39:22.843 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.843 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug shutdown = true 2024-09-16T04:39:22.844 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug op order = true 2024-09-16T04:39:22.844 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify stray on activate = true 2024-09-16T04:39:22.844 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug trim objects = true 2024-09-16T04:39:22.844 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.844 INFO:teuthology.orchestra.run.smithi152.stdout: osd open classes on start = true 2024-09-16T04:39:22.844 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug pg log writeout = true 2024-09-16T04:39:22.844 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.844 INFO:teuthology.orchestra.run.smithi152.stdout: osd deep scrub update digest min age = 30 2024-09-16T04:39:22.844 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.844 INFO:teuthology.orchestra.run.smithi152.stdout: osd map max advance = 10 2024-09-16T04:39:22.844 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.844 INFO:teuthology.orchestra.run.smithi152.stdout: journal zero on create = true 2024-09-16T04:39:22.845 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.845 INFO:teuthology.orchestra.run.smithi152.stdout: filestore ondisk finisher threads = 3 2024-09-16T04:39:22.845 INFO:teuthology.orchestra.run.smithi152.stdout: filestore apply finisher threads = 3 2024-09-16T04:39:22.845 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.845 INFO:teuthology.orchestra.run.smithi152.stdout: bdev debug aio = true 2024-09-16T04:39:22.845 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug misdirected ops = true 2024-09-16T04:39:22.845 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore block size = 96636764160 2024-09-16T04:39:22.845 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore compression algorithm = zstd 2024-09-16T04:39:22.846 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore compression mode = aggressive 2024-09-16T04:39:22.846 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore fsck on mount = True 2024-09-16T04:39:22.846 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore zero block detection = True 2024-09-16T04:39:22.846 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluefs = 20 2024-09-16T04:39:22.846 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluestore = 20 2024-09-16T04:39:22.846 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-16T04:39:22.846 INFO:teuthology.orchestra.run.smithi152.stdout: debug osd = 20 2024-09-16T04:39:22.846 INFO:teuthology.orchestra.run.smithi152.stdout: debug rocksdb = 10 2024-09-16T04:39:22.846 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T04:39:22.847 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd full ratio = 0.9 2024-09-16T04:39:22.847 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd nearfull ratio = 0.8 2024-09-16T04:39:22.847 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify cached snaps = True 2024-09-16T04:39:22.847 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify missing on start = True 2024-09-16T04:39:22.847 INFO:teuthology.orchestra.run.smithi152.stdout: osd failsafe full ratio = 0.95 2024-09-16T04:39:22.847 INFO:teuthology.orchestra.run.smithi152.stdout: osd mclock override recovery settings = True 2024-09-16T04:39:22.847 INFO:teuthology.orchestra.run.smithi152.stdout: osd objectstore = bluestore 2024-09-16T04:39:22.847 INFO:teuthology.orchestra.run.smithi152.stdout: osd op queue = debug_random 2024-09-16T04:39:22.847 INFO:teuthology.orchestra.run.smithi152.stdout: osd op queue cut off = debug_random 2024-09-16T04:39:22.848 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.848 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr] 2024-09-16T04:39:22.848 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-16T04:39:22.848 INFO:teuthology.orchestra.run.smithi152.stdout: debug mgr = 20 2024-09-16T04:39:22.848 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2024-09-16T04:39:22.848 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-09-16T04:39:22.848 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:39:22.848 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:39:22.848 INFO:teuthology.orchestra.run.smithi152.stdout: mgr/telemetry/nag = false 2024-09-16T04:39:22.848 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.848 INFO:teuthology.orchestra.run.smithi152.stdout:[mon] 2024-09-16T04:39:22.849 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-16T04:39:22.849 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2024-09-16T04:39:22.849 INFO:teuthology.orchestra.run.smithi152.stdout: debug paxos = 20 2024-09-16T04:39:22.849 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-09-16T04:39:22.849 INFO:teuthology.orchestra.run.smithi152.stdout: mon data avail warn = 5 2024-09-16T04:39:22.849 INFO:teuthology.orchestra.run.smithi152.stdout: mon mgr mkfs grace = 240 2024-09-16T04:39:22.849 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:39:22.849 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd reporter subtree level = osd 2024-09-16T04:39:22.849 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd prime pg temp = true 2024-09-16T04:39:22.849 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:39:22.849 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.849 INFO:teuthology.orchestra.run.smithi152.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T04:39:22.849 INFO:teuthology.orchestra.run.smithi152.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T04:39:22.849 INFO:teuthology.orchestra.run.smithi152.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T04:39:22.850 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.850 INFO:teuthology.orchestra.run.smithi152.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T04:39:22.850 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T04:39:22.850 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T04:39:22.850 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.850 INFO:teuthology.orchestra.run.smithi152.stdout: # 1m isn't quite enough 2024-09-16T04:39:22.850 INFO:teuthology.orchestra.run.smithi152.stdout: mon_down_mkfs_grace = 2m 2024-09-16T04:39:22.850 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.850 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_filestore_osds = false 2024-09-16T04:39:22.850 INFO:teuthology.orchestra.run.smithi152.stdout: mon scrub interval = 300 2024-09-16T04:39:22.850 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:22.850 INFO:teuthology.orchestra.run.smithi152.stdout:[client] 2024-09-16T04:39:22.851 INFO:teuthology.orchestra.run.smithi152.stdout: rgw cache enabled = true 2024-09-16T04:39:22.851 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable ops log = true 2024-09-16T04:39:22.851 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable usage log = true 2024-09-16T04:39:22.851 INFO:teuthology.orchestra.run.smithi152.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T04:39:22.851 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T04:39:22.851 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.a] 2024-09-16T04:39:22.851 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.b] 2024-09-16T04:39:22.851 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.c] 2024-09-16T04:39:22.852 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T04:39:22.852 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T04:39:22.948 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T04:39:22.948 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:22.948 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T04:39:22.963 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:22.963 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T04:39:23.018 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T04:39:23.018 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:23.018 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T04:39:23.018 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:39:23.093 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:23.094 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:39:23.149 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T04:39:23.149 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T04:39:23.149 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T04:39:23.241 INFO:teuthology.orchestra.run.smithi152.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T04:39:23.243 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T04:39:23.243 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T04:39:23.243 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T04:39:23.244 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-16T04:39:23.244 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T04:39:23.269 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T04:39:23.269 INFO:tasks.ceph:role: osd.0 2024-09-16T04:39:23.269 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T04:39:23.269 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T04:39:23.341 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:39:23.342 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:39:23.342 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:39:23.342 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:39:23.342 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:39:23.342 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:39:23.342 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:39:23.342 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:39:23.342 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:39:23.342 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:39:23.854 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-09-16T04:39:23.856 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-09-16T04:39:23.856 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T04:39:24.008 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T04:39:24.039 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T04:39:24.105 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T04:39:24.105 INFO:tasks.ceph:role: osd.1 2024-09-16T04:39:24.105 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T04:39:24.105 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T04:39:24.177 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:39:24.178 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:39:24.178 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:39:24.178 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:39:24.178 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:39:24.178 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:39:24.178 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:39:24.178 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:39:24.178 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:39:24.178 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:39:24.632 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-09-16T04:39:24.636 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-09-16T04:39:24.636 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T04:39:24.728 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T04:39:24.756 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T04:39:24.821 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T04:39:24.821 INFO:tasks.ceph:role: osd.2 2024-09-16T04:39:24.821 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T04:39:24.821 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T04:39:24.893 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:39:24.893 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:39:24.893 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:39:24.893 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:39:24.893 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:39:24.893 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:39:24.893 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:39:24.893 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:39:24.893 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:39:24.894 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:39:25.383 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-09-16T04:39:25.385 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-09-16T04:39:25.385 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T04:39:25.547 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T04:39:25.578 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T04:39:25.643 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T04:39:25.643 INFO:tasks.ceph:role: osd.3 2024-09-16T04:39:25.644 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T04:39:25.644 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T04:39:25.715 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:39:25.716 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:39:25.716 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:39:25.716 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:39:25.716 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:39:25.716 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:39:25.716 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:39:25.716 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:39:25.716 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:39:25.716 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:39:26.229 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-09-16T04:39:26.232 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-09-16T04:39:26.232 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-16T04:39:26.379 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T04:39:26.412 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:39:26.505 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:26.503+0000 7f71b891b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T04:39:26.505 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:26.504+0000 7f71b891b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T04:39:26.505 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:26.504+0000 7f71b891b640 -1 bdev(0x56382b367800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T04:39:26.506 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:26.504+0000 7f71b891b640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T04:39:30.153 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T04:39:30.184 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:39:30.275 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:30.274+0000 7faa08a0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T04:39:30.276 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:30.274+0000 7faa08a0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T04:39:30.276 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:30.274+0000 7faa08a0f640 -1 bdev(0x55a2880a3800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T04:39:30.276 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:30.274+0000 7faa08a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T04:39:33.200 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T04:39:33.230 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:39:33.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:33.318+0000 7fcae282e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T04:39:33.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:33.319+0000 7fcae282e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T04:39:33.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:33.319+0000 7fcae282e640 -1 bdev(0x55cab854d800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T04:39:33.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:33.319+0000 7fcae282e640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T04:39:36.744 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T04:39:36.773 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:39:36.864 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:36.862+0000 7f14f921c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T04:39:36.864 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:36.863+0000 7f14f921c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T04:39:36.864 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:36.863+0000 7f14f921c640 -1 bdev(0x55dc155e5800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T04:39:36.864 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:36.863+0000 7f14f921c640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T04:39:40.029 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T04:39:40.057 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T04:39:40.058 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:40.058 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T04:39:40.126 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:40.126 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T04:39:40.194 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:40.194 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T04:39:40.261 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:40.261 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T04:39:40.327 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:40.327 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T04:39:40.393 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T04:39:40.393 DEBUG:teuthology.orchestra.run.smithi152:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T04:39:40.457 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr.x] 2024-09-16T04:39:40.458 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQB7tudmmzhVDhAAYRLwqJ5eTcnziQMD52tHDw== 2024-09-16T04:39:40.458 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.0] 2024-09-16T04:39:40.458 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQB+tudmAmYRHhAAVOZotka2gUjK3dNl+8pghg== 2024-09-16T04:39:40.458 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.1] 2024-09-16T04:39:40.458 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQCCtudmIn9kEBAA+4xoEDAfIvDHwKhHiGmkUg== 2024-09-16T04:39:40.458 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.2] 2024-09-16T04:39:40.458 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQCFtudmnmkMExAA2b1Lc7P2D12fswvgpsn8WQ== 2024-09-16T04:39:40.458 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.3] 2024-09-16T04:39:40.458 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQCItudmznJ3MxAAsi82WYdbTMwJ0pRmdeWqgg== 2024-09-16T04:39:40.459 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:39:40.552 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:39:40.603 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:39:40.654 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:39:40.705 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:39:40.755 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T04:39:40.755 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T04:39:40.780 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:39:40.876 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T04:39:40.900 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T04:39:40.965 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:39:41.061 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T04:39:41.086 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T04:39:41.151 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:39:41.246 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T04:39:41.271 DEBUG:teuthology.orchestra.run.smithi152:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:39:41.326 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T04:39:41.327 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:39:41.327 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:39:41.369 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:39:41.369 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T04:39:41.370 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T04:39:41.373 INFO:tasks.ceph.mon.b:Started 2024-09-16T04:39:41.373 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:39:41.373 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:39:41.375 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:39:41.376 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph config set mgr mgr_pool false --force 2024-09-16T04:39:48.176 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T04:39:48.177 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T04:39:48.177 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T04:39:48.179 INFO:tasks.ceph.mgr.x:Started 2024-09-16T04:39:48.179 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T04:39:48.180 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T04:39:48.352 INFO:teuthology.orchestra.run.smithi152.stderr:adjusted tunables profile to default 2024-09-16T04:39:48.363 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T04:39:48.363 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T04:39:48.363 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:48.364 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T04:39:48.388 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:48.388 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T04:39:48.453 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:48.453 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T04:39:48.509 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:48.506+0000 7f4c3fe0e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T04:39:48.518 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-16T04:39:48.518 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T04:39:48.552 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:48.549+0000 7f4c3fe0e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T04:39:48.576 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new d8b076bc-f4ae-4100-8cdf-78560225fea9 0 2024-09-16T04:39:48.707 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:48.705+0000 7f4c3fe0e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T04:39:48.760 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:48.757+0000 7f4c3fe0e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T04:39:48.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:39:48.919+0000 7f03f7fff640 0 --2- 172.21.15.152:0/3818458016 >> [v2:172.21.15.152:3301/0,v1:172.21.15.152:6790/0] conn(0x7f03f80726a0 0x7f03f80970a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T04:39:48.922 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:48.919+0000 7f4c3fe0e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:39:49.003 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-09-16T04:39:49.014 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 023d4748-7485-4d8b-8594-54a238b67b71 1 2024-09-16T04:39:49.040 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:49.037+0000 7f4c3fe0e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T04:39:49.194 INFO:teuthology.orchestra.run.smithi152.stdout:1 2024-09-16T04:39:49.205 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new c7bfb282-86b4-4bdc-902d-0a0cfbadf5ed 2 2024-09-16T04:39:49.389 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:49.387+0000 7f4c3fe0e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T04:39:49.432 INFO:teuthology.orchestra.run.smithi152.stdout:2 2024-09-16T04:39:49.444 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new f82045bc-6fe9-4755-8c25-ea7738b90d6f 3 2024-09-16T04:39:49.444 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:49.441+0000 7f4c3fe0e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T04:39:49.603 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:49.601+0000 7f4c3fe0e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T04:39:49.628 INFO:teuthology.orchestra.run.smithi152.stdout:3 2024-09-16T04:39:49.639 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T04:39:49.639 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T04:39:49.642 INFO:tasks.ceph.osd.0:Started 2024-09-16T04:39:49.643 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T04:39:49.643 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T04:39:49.646 INFO:tasks.ceph.osd.1:Started 2024-09-16T04:39:49.646 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T04:39:49.646 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T04:39:49.650 INFO:tasks.ceph.osd.2:Started 2024-09-16T04:39:49.651 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T04:39:49.651 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T04:39:49.653 INFO:tasks.ceph.osd.3:Started 2024-09-16T04:39:49.653 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:39:49.851 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:39:49.851 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":6,"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","created":"2024-09-16T04:39:47.674616+0000","modified":"2024-09-16T04:39:49.620271+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T04:39:49.620271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8b076bc-f4ae-4100-8cdf-78560225fea9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"023d4748-7485-4d8b-8594-54a238b67b71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c7bfb282-86b4-4bdc-902d-0a0cfbadf5ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f82045bc-6fe9-4755-8c25-ea7738b90d6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:39:49.862 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T04:39:49.863 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T04:39:50.265 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:39:50.422 INFO:teuthology.misc.health.smithi152.stdout: 2024-09-16T04:39:50.422 INFO:teuthology.misc.health.smithi152.stdout:{"epoch":6,"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","created":"2024-09-16T04:39:47.674616+0000","modified":"2024-09-16T04:39:49.620271+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T04:39:49.620271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8b076bc-f4ae-4100-8cdf-78560225fea9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"023d4748-7485-4d8b-8594-54a238b67b71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c7bfb282-86b4-4bdc-902d-0a0cfbadf5ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f82045bc-6fe9-4755-8c25-ea7738b90d6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:39:50.432 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-16T04:39:50.625 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:50.622+0000 7f4c3fe0e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T04:39:50.677 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:50.675+0000 7f4c3fe0e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T04:39:51.050 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:51.048+0000 7f4c3fe0e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T04:39:51.138 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:51.136+0000 7f4c3fe0e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T04:39:51.245 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:51.242+0000 7f4c3fe0e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:39:51.291 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:51.288+0000 7f4c3fe0e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T04:39:51.350 INFO:tasks.ceph.osd.0.smithi152.stderr:2024-09-16T04:39:51.347+0000 7f6126745640 -1 Falling back to public interface 2024-09-16T04:39:51.369 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-16T04:39:51.367+0000 7f3c6ca0b640 -1 Falling back to public interface 2024-09-16T04:39:51.373 INFO:tasks.ceph.osd.1.smithi152.stderr:2024-09-16T04:39:51.371+0000 7fc68ce18640 -1 Falling back to public interface 2024-09-16T04:39:51.408 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:51.405+0000 7f4c3fe0e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T04:39:51.461 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:51.458+0000 7f4c3fe0e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T04:39:51.506 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:51.504+0000 7f4c3fe0e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T04:39:51.618 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:51.616+0000 7f4c3fe0e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T04:39:51.621 INFO:tasks.ceph.osd.3.smithi152.stderr:2024-09-16T04:39:51.619+0000 7f267062a640 -1 Falling back to public interface 2024-09-16T04:39:51.671 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:51.669+0000 7f4c3fe0e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T04:39:51.722 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:51.720+0000 7f4c3fe0e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T04:39:51.790 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:51.787+0000 7f4c3fe0e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T04:39:51.833 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:51.831+0000 7f4c3fe0e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T04:39:51.877 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:51.875+0000 7f4c3fe0e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T04:39:52.534 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:52.532+0000 7f4c3fe0e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T04:39:53.548 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:53.546+0000 7f4bdfaf6640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T04:39:53.549 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-09-16T04:39:53.547+0000 7f4bdfaf6640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T04:39:55.195 INFO:tasks.ceph.osd.1.smithi152.stderr:2024-09-16T04:39:55.193+0000 7fc68ce18640 -1 osd.1 0 log_to_monitors true 2024-09-16T04:39:55.212 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-16T04:39:55.210+0000 7f3c6ca0b640 -1 osd.2 0 log_to_monitors true 2024-09-16T04:39:55.413 INFO:tasks.ceph.osd.3.smithi152.stderr:2024-09-16T04:39:55.411+0000 7f267062a640 -1 osd.3 0 log_to_monitors true 2024-09-16T04:39:55.437 INFO:tasks.ceph.osd.0.smithi152.stderr:2024-09-16T04:39:55.435+0000 7f6126745640 -1 osd.0 0 log_to_monitors true 2024-09-16T04:39:56.560 INFO:tasks.ceph.osd.0.smithi152.stderr:2024-09-16T04:39:56.557+0000 7f61226d4640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T04:39:56.560 INFO:tasks.ceph.osd.3.smithi152.stderr:2024-09-16T04:39:56.558+0000 7f266c5b9640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T04:39:56.569 INFO:tasks.ceph.osd.3.smithi152.stderr:2024-09-16T04:39:56.566+0000 7f26683e1640 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:39:56.569 INFO:tasks.ceph.osd.0.smithi152.stderr:2024-09-16T04:39:56.566+0000 7f611e4fc640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:39:56.835 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:39:57.221 INFO:teuthology.misc.health.smithi152.stdout: 2024-09-16T04:39:57.221 INFO:teuthology.misc.health.smithi152.stdout:{"epoch":8,"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","created":"2024-09-16T04:39:47.674616+0000","modified":"2024-09-16T04:39:56.546327+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T04:39:49.620271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8b076bc-f4ae-4100-8cdf-78560225fea9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"023d4748-7485-4d8b-8594-54a238b67b71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c7bfb282-86b4-4bdc-902d-0a0cfbadf5ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f82045bc-6fe9-4755-8c25-ea7738b90d6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:39:57.236 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-16T04:39:57.332 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-16T04:39:57.329+0000 7f3c683f9640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T04:39:57.332 INFO:tasks.ceph.osd.1.smithi152.stderr:2024-09-16T04:39:57.329+0000 7fc68877a640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T04:39:57.335 INFO:tasks.ceph.osd.1.smithi152.stderr:2024-09-16T04:39:57.333+0000 7fc68059a640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:39:57.335 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-16T04:39:57.333+0000 7f3c60219640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:40:03.640 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:40:03.918 INFO:teuthology.misc.health.smithi152.stdout: 2024-09-16T04:40:03.918 INFO:teuthology.misc.health.smithi152.stdout:{"epoch":10,"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","created":"2024-09-16T04:39:47.674616+0000","modified":"2024-09-16T04:39:58.570569+0000","last_up_change":"2024-09-16T04:39:57.560897+0000","last_in_change":"2024-09-16T04:39:49.620271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8b076bc-f4ae-4100-8cdf-78560225fea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":36071542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6801","nonce":36071542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":36071542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":36071542}]},"public_addr":"172.21.15.152:6800/36071542","cluster_addr":"172.21.15.152:6801/36071542","heartbeat_back_addr":"172.21.15.152:6803/36071542","heartbeat_front_addr":"172.21.15.152:6802/36071542","state":["exists","up"]},{"osd":1,"uuid":"023d4748-7485-4d8b-8594-54a238b67b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1844087228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":1844087228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6811","nonce":1844087228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1844087228}]},"public_addr":"172.21.15.152:6808/1844087228","cluster_addr":"172.21.15.152:6809/1844087228","heartbeat_back_addr":"172.21.15.152:6811/1844087228","heartbeat_front_addr":"172.21.15.152:6810/1844087228","state":["exists","up"]},{"osd":2,"uuid":"c7bfb282-86b4-4bdc-902d-0a0cfbadf5ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2854825795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2854825795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":2854825795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2854825795}]},"public_addr":"172.21.15.152:6804/2854825795","cluster_addr":"172.21.15.152:6805/2854825795","heartbeat_back_addr":"172.21.15.152:6807/2854825795","heartbeat_front_addr":"172.21.15.152:6806/2854825795","state":["exists","up"]},{"osd":3,"uuid":"f82045bc-6fe9-4755-8c25-ea7738b90d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3902150580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6813","nonce":3902150580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":3902150580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3902150580}]},"public_addr":"172.21.15.152:6812/3902150580","cluster_addr":"172.21.15.152:6813/3902150580","heartbeat_back_addr":"172.21.15.152:6815/3902150580","heartbeat_front_addr":"172.21.15.152:6814/3902150580","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.429729+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.175556+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.243970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.373375+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:40:03.929 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-16T04:40:03.929 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T04:40:03.929 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T04:40:03.930 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T04:40:03.930 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T04:40:03.930 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T04:40:04.252 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:04.264 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":4,"flags":0,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":4283621395}]},"active_addr":"172.21.15.152:6816/4283621395","active_change":"2024-09-16T04:39:52.535487+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":3081909234}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":318342109}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":2554308321}]}]} 2024-09-16T04:40:04.266 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T04:40:04.266 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T04:40:04.266 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:40:04.540 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:04.541 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":10,"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","created":"2024-09-16T04:39:47.674616+0000","modified":"2024-09-16T04:39:58.570569+0000","last_up_change":"2024-09-16T04:39:57.560897+0000","last_in_change":"2024-09-16T04:39:49.620271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8b076bc-f4ae-4100-8cdf-78560225fea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":36071542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6801","nonce":36071542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":36071542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":36071542}]},"public_addr":"172.21.15.152:6800/36071542","cluster_addr":"172.21.15.152:6801/36071542","heartbeat_back_addr":"172.21.15.152:6803/36071542","heartbeat_front_addr":"172.21.15.152:6802/36071542","state":["exists","up"]},{"osd":1,"uuid":"023d4748-7485-4d8b-8594-54a238b67b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1844087228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":1844087228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6811","nonce":1844087228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1844087228}]},"public_addr":"172.21.15.152:6808/1844087228","cluster_addr":"172.21.15.152:6809/1844087228","heartbeat_back_addr":"172.21.15.152:6811/1844087228","heartbeat_front_addr":"172.21.15.152:6810/1844087228","state":["exists","up"]},{"osd":2,"uuid":"c7bfb282-86b4-4bdc-902d-0a0cfbadf5ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2854825795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2854825795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":2854825795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2854825795}]},"public_addr":"172.21.15.152:6804/2854825795","cluster_addr":"172.21.15.152:6805/2854825795","heartbeat_back_addr":"172.21.15.152:6807/2854825795","heartbeat_front_addr":"172.21.15.152:6806/2854825795","state":["exists","up"]},{"osd":3,"uuid":"f82045bc-6fe9-4755-8c25-ea7738b90d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3902150580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6813","nonce":3902150580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":3902150580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3902150580}]},"public_addr":"172.21.15.152:6812/3902150580","cluster_addr":"172.21.15.152:6813/3902150580","heartbeat_back_addr":"172.21.15.152:6815/3902150580","heartbeat_front_addr":"172.21.15.152:6814/3902150580","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.429729+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.175556+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.243970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.373375+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:40:04.553 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T04:40:04.553 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:40:04.834 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:04.834 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":10,"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","created":"2024-09-16T04:39:47.674616+0000","modified":"2024-09-16T04:39:58.570569+0000","last_up_change":"2024-09-16T04:39:57.560897+0000","last_in_change":"2024-09-16T04:39:49.620271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8b076bc-f4ae-4100-8cdf-78560225fea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":36071542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6801","nonce":36071542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":36071542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":36071542}]},"public_addr":"172.21.15.152:6800/36071542","cluster_addr":"172.21.15.152:6801/36071542","heartbeat_back_addr":"172.21.15.152:6803/36071542","heartbeat_front_addr":"172.21.15.152:6802/36071542","state":["exists","up"]},{"osd":1,"uuid":"023d4748-7485-4d8b-8594-54a238b67b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1844087228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":1844087228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6811","nonce":1844087228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1844087228}]},"public_addr":"172.21.15.152:6808/1844087228","cluster_addr":"172.21.15.152:6809/1844087228","heartbeat_back_addr":"172.21.15.152:6811/1844087228","heartbeat_front_addr":"172.21.15.152:6810/1844087228","state":["exists","up"]},{"osd":2,"uuid":"c7bfb282-86b4-4bdc-902d-0a0cfbadf5ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2854825795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2854825795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":2854825795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2854825795}]},"public_addr":"172.21.15.152:6804/2854825795","cluster_addr":"172.21.15.152:6805/2854825795","heartbeat_back_addr":"172.21.15.152:6807/2854825795","heartbeat_front_addr":"172.21.15.152:6806/2854825795","state":["exists","up"]},{"osd":3,"uuid":"f82045bc-6fe9-4755-8c25-ea7738b90d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3902150580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6813","nonce":3902150580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":3902150580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3902150580}]},"public_addr":"172.21.15.152:6812/3902150580","cluster_addr":"172.21.15.152:6813/3902150580","heartbeat_back_addr":"172.21.15.152:6815/3902150580","heartbeat_front_addr":"172.21.15.152:6814/3902150580","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.429729+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.175556+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.243970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.373375+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:40:04.847 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T04:40:04.848 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T04:40:04.848 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T04:40:04.848 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T04:40:04.989 INFO:teuthology.orchestra.run.smithi152.stdout:38654705667 2024-09-16T04:40:04.990 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:40:04.991 INFO:teuthology.orchestra.run.smithi152.stdout:38654705668 2024-09-16T04:40:04.991 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:40:05.008 INFO:teuthology.orchestra.run.smithi152.stdout:38654705667 2024-09-16T04:40:05.009 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:40:05.009 INFO:teuthology.orchestra.run.smithi152.stdout:38654705669 2024-09-16T04:40:05.010 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:40:05.315 INFO:teuthology.orchestra.run.smithi152.stdout:38654705666 2024-09-16T04:40:05.317 INFO:teuthology.orchestra.run.smithi152.stdout:38654705667 2024-09-16T04:40:05.329 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.3 2024-09-16T04:40:05.335 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.1 2024-09-16T04:40:05.368 INFO:teuthology.orchestra.run.smithi152.stdout:38654705668 2024-09-16T04:40:05.377 INFO:teuthology.orchestra.run.smithi152.stdout:38654705666 2024-09-16T04:40:05.381 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705668 for osd.2 2024-09-16T04:40:05.389 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.0 2024-09-16T04:40:06.330 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:40:06.336 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:40:06.381 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:40:06.391 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:40:06.676 INFO:teuthology.orchestra.run.smithi152.stdout:38654705667 2024-09-16T04:40:06.684 INFO:teuthology.orchestra.run.smithi152.stdout:38654705668 2024-09-16T04:40:06.693 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.3 2024-09-16T04:40:06.694 DEBUG:teuthology.parallel:result is None 2024-09-16T04:40:06.702 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.1 2024-09-16T04:40:06.702 DEBUG:teuthology.parallel:result is None 2024-09-16T04:40:06.739 INFO:teuthology.orchestra.run.smithi152.stdout:38654705669 2024-09-16T04:40:06.745 INFO:teuthology.orchestra.run.smithi152.stdout:38654705667 2024-09-16T04:40:06.752 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.2 2024-09-16T04:40:06.752 DEBUG:teuthology.parallel:result is None 2024-09-16T04:40:06.759 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.0 2024-09-16T04:40:06.759 DEBUG:teuthology.parallel:result is None 2024-09-16T04:40:06.759 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:40:06.759 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:40:07.074 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:07.075 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:40:07.087 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-16T04:40:06.541969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108472,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378888,"statfs":{"total":386547056640,"available":386435981312,"internally_reserved":0,"allocated":360448,"data_stored":74580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-16T04:40:07.087 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:40:07.358 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:07.358 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:40:07.370 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-16T04:40:06.541969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108472,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378888,"statfs":{"total":386547056640,"available":386435981312,"internally_reserved":0,"allocated":360448,"data_stored":74580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-16T04:40:07.371 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:40:07.371 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T04:40:07.371 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T04:40:07.371 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T04:40:07.681 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:07.682 INFO:teuthology.orchestra.run.smithi152.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T04:40:07.694 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T04:40:07.694 INFO:teuthology.run_tasks:Running task ec_lost_unfound... 2024-09-16T04:40:07.718 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:40:07.994 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:07.994 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":10,"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","created":"2024-09-16T04:39:47.674616+0000","modified":"2024-09-16T04:39:58.570569+0000","last_up_change":"2024-09-16T04:39:57.560897+0000","last_in_change":"2024-09-16T04:39:49.620271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8b076bc-f4ae-4100-8cdf-78560225fea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":36071542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6801","nonce":36071542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":36071542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":36071542}]},"public_addr":"172.21.15.152:6800/36071542","cluster_addr":"172.21.15.152:6801/36071542","heartbeat_back_addr":"172.21.15.152:6803/36071542","heartbeat_front_addr":"172.21.15.152:6802/36071542","state":["exists","up"]},{"osd":1,"uuid":"023d4748-7485-4d8b-8594-54a238b67b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1844087228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":1844087228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6811","nonce":1844087228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1844087228}]},"public_addr":"172.21.15.152:6808/1844087228","cluster_addr":"172.21.15.152:6809/1844087228","heartbeat_back_addr":"172.21.15.152:6811/1844087228","heartbeat_front_addr":"172.21.15.152:6810/1844087228","state":["exists","up"]},{"osd":2,"uuid":"c7bfb282-86b4-4bdc-902d-0a0cfbadf5ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2854825795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2854825795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":2854825795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2854825795}]},"public_addr":"172.21.15.152:6804/2854825795","cluster_addr":"172.21.15.152:6805/2854825795","heartbeat_back_addr":"172.21.15.152:6807/2854825795","heartbeat_front_addr":"172.21.15.152:6806/2854825795","state":["exists","up"]},{"osd":3,"uuid":"f82045bc-6fe9-4755-8c25-ea7738b90d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3902150580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6813","nonce":3902150580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":3902150580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3902150580}]},"public_addr":"172.21.15.152:6812/3902150580","cluster_addr":"172.21.15.152:6813/3902150580","heartbeat_back_addr":"172.21.15.152:6815/3902150580","heartbeat_front_addr":"172.21.15.152:6814/3902150580","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.429729+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.175556+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.243970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.373375+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:40:08.007 INFO:tasks.ec_lost_unfound.ceph_manager:[] 2024-09-16T04:40:08.007 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-16T04:40:08.008 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:40:08.280 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:08.281 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:40:08.293 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-16T04:40:06.541969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108472,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378888,"statfs":{"total":386547056640,"available":386435981312,"internally_reserved":0,"allocated":360448,"data_stored":74580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-16T04:40:08.293 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:40:08.567 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:08.568 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:40:08.580 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-16T04:40:08.542188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108472,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378888,"statfs":{"total":386547056640,"available":386435981312,"internally_reserved":0,"allocated":360448,"data_stored":74580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":18645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-16T04:40:08.581 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-16T04:40:08.581 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set lost_unfound k=2 m=2 crush-failure-domain=osd 2024-09-16T04:40:09.626 INFO:tasks.ec_lost_unfound.ceph_manager:creating pool_name unique_pool_0 2024-09-16T04:40:09.626 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure lost_unfound 2024-09-16T04:40:11.617 INFO:teuthology.orchestra.run.smithi152.stderr:pool 'unique_pool_0' created 2024-09-16T04:40:11.637 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 min_size 2 2024-09-16T04:40:12.616 INFO:teuthology.orchestra.run.smithi152.stderr:set pool 1 min_size to 2 2024-09-16T04:40:12.640 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-16T04:40:13.617 INFO:teuthology.orchestra.run.smithi152.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-16T04:40:14.650 INFO:tasks.util.rados:rados -p unique_pool_0 put dummy /etc/group 2024-09-16T04:40:14.650 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put dummy /etc/group 2024-09-16T04:40:14.709 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T04:40:14.709 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T04:40:14.844 INFO:teuthology.orchestra.run.smithi152.stdout:38654705670 2024-09-16T04:40:14.845 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:40:14.846 INFO:teuthology.orchestra.run.smithi152.stdout:38654705671 2024-09-16T04:40:14.846 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:40:15.171 INFO:teuthology.orchestra.run.smithi152.stdout:38654705670 2024-09-16T04:40:15.176 INFO:teuthology.orchestra.run.smithi152.stdout:38654705669 2024-09-16T04:40:15.186 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705671 got 38654705670 for osd.1 2024-09-16T04:40:15.189 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705670 got 38654705669 for osd.0 2024-09-16T04:40:16.188 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:40:16.190 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:40:16.485 INFO:teuthology.orchestra.run.smithi152.stdout:38654705669 2024-09-16T04:40:16.489 INFO:teuthology.orchestra.run.smithi152.stdout:38654705670 2024-09-16T04:40:16.498 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705670 got 38654705669 for osd.0 2024-09-16T04:40:16.503 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705671 got 38654705670 for osd.1 2024-09-16T04:40:17.500 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:40:17.503 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:40:17.803 INFO:teuthology.orchestra.run.smithi152.stdout:38654705670 2024-09-16T04:40:17.816 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705670 got 38654705670 for osd.0 2024-09-16T04:40:17.816 DEBUG:teuthology.parallel:result is None 2024-09-16T04:40:17.836 INFO:teuthology.orchestra.run.smithi152.stdout:38654705671 2024-09-16T04:40:17.849 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705671 got 38654705671 for osd.1 2024-09-16T04:40:17.849 DEBUG:teuthology.parallel:result is None 2024-09-16T04:40:17.849 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-09-16T04:40:17.850 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:40:18.160 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:18.161 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:40:18.173 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T04:40:16.543473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108620,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378740,"statfs":{"total":386547056640,"available":386435829760,"internally_reserved":0,"allocated":507904,"data_stored":133886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.617176+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.617176+0000","last_undegraded":"2024-09-16T04:40:12.617176+0000","last_fullsized":"2024-09-16T04:40:12.617176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617887+0000","last_change":"2024-09-16T04:40:13.637120+0000","last_active":"2024-09-16T04:40:14.617887+0000","last_peered":"2024-09-16T04:40:14.617887+0000","last_clean":"2024-09-16T04:40:14.617887+0000","last_became_active":"2024-09-16T04:40:13.636399+0000","last_became_peered":"2024-09-16T04:40:13.636399+0000","last_unstale":"2024-09-16T04:40:14.617887+0000","last_undegraded":"2024-09-16T04:40:14.617887+0000","last_fullsized":"2024-09-16T04:40:14.617887+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:42.568657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.618209+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.618209+0000","last_undegraded":"2024-09-16T04:40:12.618209+0000","last_fullsized":"2024-09-16T04:40:12.618209+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617159+0000","last_change":"2024-09-16T04:40:13.632054+0000","last_active":"2024-09-16T04:40:14.617159+0000","last_peered":"2024-09-16T04:40:14.617159+0000","last_clean":"2024-09-16T04:40:14.617159+0000","last_became_active":"2024-09-16T04:40:13.629885+0000","last_became_peered":"2024-09-16T04:40:13.629885+0000","last_unstale":"2024-09-16T04:40:14.617159+0000","last_undegraded":"2024-09-16T04:40:14.617159+0000","last_fullsized":"2024-09-16T04:40:14.617159+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:21.446474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.616836+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.616836+0000","last_undegraded":"2024-09-16T04:40:12.616836+0000","last_fullsized":"2024-09-16T04:40:12.616836+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.616883+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.616883+0000","last_undegraded":"2024-09-16T04:40:12.616883+0000","last_fullsized":"2024-09-16T04:40:12.616883+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.705352+0000","last_change":"2024-09-16T04:40:13.626272+0000","last_active":"2024-09-16T04:40:14.705352+0000","last_peered":"2024-09-16T04:40:14.705352+0000","last_clean":"2024-09-16T04:40:14.705352+0000","last_became_active":"2024-09-16T04:40:13.625866+0000","last_became_peered":"2024-09-16T04:40:13.625866+0000","last_unstale":"2024-09-16T04:40:14.705352+0000","last_undegraded":"2024-09-16T04:40:14.705352+0000","last_fullsized":"2024-09-16T04:40:14.705352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:41.631481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617676+0000","last_change":"2024-09-16T04:40:13.630616+0000","last_active":"2024-09-16T04:40:14.617676+0000","last_peered":"2024-09-16T04:40:14.617676+0000","last_clean":"2024-09-16T04:40:14.617676+0000","last_became_active":"2024-09-16T04:40:13.630240+0000","last_became_peered":"2024-09-16T04:40:13.630240+0000","last_unstale":"2024-09-16T04:40:14.617676+0000","last_undegraded":"2024-09-16T04:40:14.617676+0000","last_fullsized":"2024-09-16T04:40:14.617676+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:25.350686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.618315+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.618315+0000","last_undegraded":"2024-09-16T04:40:12.618315+0000","last_fullsized":"2024-09-16T04:40:12.618315+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.616925+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.616925+0000","last_undegraded":"2024-09-16T04:40:12.616925+0000","last_fullsized":"2024-09-16T04:40:12.616925+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.617213+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.617213+0000","last_undegraded":"2024-09-16T04:40:12.617213+0000","last_fullsized":"2024-09-16T04:40:12.617213+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617689+0000","last_change":"2024-09-16T04:40:13.638219+0000","last_active":"2024-09-16T04:40:14.617689+0000","last_peered":"2024-09-16T04:40:14.617689+0000","last_clean":"2024-09-16T04:40:14.617689+0000","last_became_active":"2024-09-16T04:40:13.637832+0000","last_became_peered":"2024-09-16T04:40:13.637832+0000","last_unstale":"2024-09-16T04:40:14.617689+0000","last_undegraded":"2024-09-16T04:40:14.617689+0000","last_fullsized":"2024-09-16T04:40:14.617689+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:04:38.388133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617696+0000","last_change":"2024-09-16T04:40:13.632248+0000","last_active":"2024-09-16T04:40:14.617696+0000","last_peered":"2024-09-16T04:40:14.617696+0000","last_clean":"2024-09-16T04:40:14.617696+0000","last_became_active":"2024-09-16T04:40:13.630436+0000","last_became_peered":"2024-09-16T04:40:13.630436+0000","last_unstale":"2024-09-16T04:40:14.617696+0000","last_undegraded":"2024-09-16T04:40:14.617696+0000","last_fullsized":"2024-09-16T04:40:14.617696+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:21:17.640198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617912+0000","last_change":"2024-09-16T04:40:13.629954+0000","last_active":"2024-09-16T04:40:14.617912+0000","last_peered":"2024-09-16T04:40:14.617912+0000","last_clean":"2024-09-16T04:40:14.617912+0000","last_became_active":"2024-09-16T04:40:13.629576+0000","last_became_peered":"2024-09-16T04:40:13.629576+0000","last_unstale":"2024-09-16T04:40:14.617912+0000","last_undegraded":"2024-09-16T04:40:14.617912+0000","last_fullsized":"2024-09-16T04:40:14.617912+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:02.490377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617336+0000","last_change":"2024-09-16T04:40:13.631818+0000","last_active":"2024-09-16T04:40:14.617336+0000","last_peered":"2024-09-16T04:40:14.617336+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:40:13.631383+0000","last_became_peered":"2024-09-16T04:40:13.631383+0000","last_unstale":"2024-09-16T04:40:14.617336+0000","last_undegraded":"2024-09-16T04:40:14.617336+0000","last_fullsized":"2024-09-16T04:40:14.617336+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:44:30.068689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617518+0000","last_change":"2024-09-16T04:40:13.630037+0000","last_active":"2024-09-16T04:40:14.617518+0000","last_peered":"2024-09-16T04:40:14.617518+0000","last_clean":"2024-09-16T04:40:14.617518+0000","last_became_active":"2024-09-16T04:40:13.629643+0000","last_became_peered":"2024-09-16T04:40:13.629643+0000","last_unstale":"2024-09-16T04:40:14.617518+0000","last_undegraded":"2024-09-16T04:40:14.617518+0000","last_fullsized":"2024-09-16T04:40:14.617518+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:27.740812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:40:18.174 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:40:18.446 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:18.446 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:40:18.459 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T04:40:16.543473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108620,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378740,"statfs":{"total":386547056640,"available":386435829760,"internally_reserved":0,"allocated":507904,"data_stored":133886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.617176+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.617176+0000","last_undegraded":"2024-09-16T04:40:12.617176+0000","last_fullsized":"2024-09-16T04:40:12.617176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617887+0000","last_change":"2024-09-16T04:40:13.637120+0000","last_active":"2024-09-16T04:40:14.617887+0000","last_peered":"2024-09-16T04:40:14.617887+0000","last_clean":"2024-09-16T04:40:14.617887+0000","last_became_active":"2024-09-16T04:40:13.636399+0000","last_became_peered":"2024-09-16T04:40:13.636399+0000","last_unstale":"2024-09-16T04:40:14.617887+0000","last_undegraded":"2024-09-16T04:40:14.617887+0000","last_fullsized":"2024-09-16T04:40:14.617887+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:42.568657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.618209+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.618209+0000","last_undegraded":"2024-09-16T04:40:12.618209+0000","last_fullsized":"2024-09-16T04:40:12.618209+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617159+0000","last_change":"2024-09-16T04:40:13.632054+0000","last_active":"2024-09-16T04:40:14.617159+0000","last_peered":"2024-09-16T04:40:14.617159+0000","last_clean":"2024-09-16T04:40:14.617159+0000","last_became_active":"2024-09-16T04:40:13.629885+0000","last_became_peered":"2024-09-16T04:40:13.629885+0000","last_unstale":"2024-09-16T04:40:14.617159+0000","last_undegraded":"2024-09-16T04:40:14.617159+0000","last_fullsized":"2024-09-16T04:40:14.617159+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:21.446474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.616836+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.616836+0000","last_undegraded":"2024-09-16T04:40:12.616836+0000","last_fullsized":"2024-09-16T04:40:12.616836+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.616883+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.616883+0000","last_undegraded":"2024-09-16T04:40:12.616883+0000","last_fullsized":"2024-09-16T04:40:12.616883+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.705352+0000","last_change":"2024-09-16T04:40:13.626272+0000","last_active":"2024-09-16T04:40:14.705352+0000","last_peered":"2024-09-16T04:40:14.705352+0000","last_clean":"2024-09-16T04:40:14.705352+0000","last_became_active":"2024-09-16T04:40:13.625866+0000","last_became_peered":"2024-09-16T04:40:13.625866+0000","last_unstale":"2024-09-16T04:40:14.705352+0000","last_undegraded":"2024-09-16T04:40:14.705352+0000","last_fullsized":"2024-09-16T04:40:14.705352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:41.631481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617676+0000","last_change":"2024-09-16T04:40:13.630616+0000","last_active":"2024-09-16T04:40:14.617676+0000","last_peered":"2024-09-16T04:40:14.617676+0000","last_clean":"2024-09-16T04:40:14.617676+0000","last_became_active":"2024-09-16T04:40:13.630240+0000","last_became_peered":"2024-09-16T04:40:13.630240+0000","last_unstale":"2024-09-16T04:40:14.617676+0000","last_undegraded":"2024-09-16T04:40:14.617676+0000","last_fullsized":"2024-09-16T04:40:14.617676+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:25.350686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.618315+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.618315+0000","last_undegraded":"2024-09-16T04:40:12.618315+0000","last_fullsized":"2024-09-16T04:40:12.618315+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.616925+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.616925+0000","last_undegraded":"2024-09-16T04:40:12.616925+0000","last_fullsized":"2024-09-16T04:40:12.616925+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.617213+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.617213+0000","last_undegraded":"2024-09-16T04:40:12.617213+0000","last_fullsized":"2024-09-16T04:40:12.617213+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617689+0000","last_change":"2024-09-16T04:40:13.638219+0000","last_active":"2024-09-16T04:40:14.617689+0000","last_peered":"2024-09-16T04:40:14.617689+0000","last_clean":"2024-09-16T04:40:14.617689+0000","last_became_active":"2024-09-16T04:40:13.637832+0000","last_became_peered":"2024-09-16T04:40:13.637832+0000","last_unstale":"2024-09-16T04:40:14.617689+0000","last_undegraded":"2024-09-16T04:40:14.617689+0000","last_fullsized":"2024-09-16T04:40:14.617689+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:04:38.388133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617696+0000","last_change":"2024-09-16T04:40:13.632248+0000","last_active":"2024-09-16T04:40:14.617696+0000","last_peered":"2024-09-16T04:40:14.617696+0000","last_clean":"2024-09-16T04:40:14.617696+0000","last_became_active":"2024-09-16T04:40:13.630436+0000","last_became_peered":"2024-09-16T04:40:13.630436+0000","last_unstale":"2024-09-16T04:40:14.617696+0000","last_undegraded":"2024-09-16T04:40:14.617696+0000","last_fullsized":"2024-09-16T04:40:14.617696+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:21:17.640198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617912+0000","last_change":"2024-09-16T04:40:13.629954+0000","last_active":"2024-09-16T04:40:14.617912+0000","last_peered":"2024-09-16T04:40:14.617912+0000","last_clean":"2024-09-16T04:40:14.617912+0000","last_became_active":"2024-09-16T04:40:13.629576+0000","last_became_peered":"2024-09-16T04:40:13.629576+0000","last_unstale":"2024-09-16T04:40:14.617912+0000","last_undegraded":"2024-09-16T04:40:14.617912+0000","last_fullsized":"2024-09-16T04:40:14.617912+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:02.490377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617336+0000","last_change":"2024-09-16T04:40:13.631818+0000","last_active":"2024-09-16T04:40:14.617336+0000","last_peered":"2024-09-16T04:40:14.617336+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:40:13.631383+0000","last_became_peered":"2024-09-16T04:40:13.631383+0000","last_unstale":"2024-09-16T04:40:14.617336+0000","last_undegraded":"2024-09-16T04:40:14.617336+0000","last_fullsized":"2024-09-16T04:40:14.617336+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:44:30.068689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617518+0000","last_change":"2024-09-16T04:40:13.630037+0000","last_active":"2024-09-16T04:40:14.617518+0000","last_peered":"2024-09-16T04:40:14.617518+0000","last_clean":"2024-09-16T04:40:14.617518+0000","last_became_active":"2024-09-16T04:40:13.629643+0000","last_became_peered":"2024-09-16T04:40:13.629643+0000","last_unstale":"2024-09-16T04:40:14.617518+0000","last_undegraded":"2024-09-16T04:40:14.617518+0000","last_fullsized":"2024-09-16T04:40:14.617518+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:27.740812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:40:18.460 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:40:18.735 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:18.736 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:40:18.747 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-16T04:40:18.543765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108620,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378740,"statfs":{"total":386547056640,"available":386435829760,"internally_reserved":0,"allocated":507904,"data_stored":133886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.617176+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.617176+0000","last_undegraded":"2024-09-16T04:40:12.617176+0000","last_fullsized":"2024-09-16T04:40:12.617176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617887+0000","last_change":"2024-09-16T04:40:13.637120+0000","last_active":"2024-09-16T04:40:14.617887+0000","last_peered":"2024-09-16T04:40:14.617887+0000","last_clean":"2024-09-16T04:40:14.617887+0000","last_became_active":"2024-09-16T04:40:13.636399+0000","last_became_peered":"2024-09-16T04:40:13.636399+0000","last_unstale":"2024-09-16T04:40:14.617887+0000","last_undegraded":"2024-09-16T04:40:14.617887+0000","last_fullsized":"2024-09-16T04:40:14.617887+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:42.568657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.618209+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.618209+0000","last_undegraded":"2024-09-16T04:40:12.618209+0000","last_fullsized":"2024-09-16T04:40:12.618209+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617159+0000","last_change":"2024-09-16T04:40:13.632054+0000","last_active":"2024-09-16T04:40:14.617159+0000","last_peered":"2024-09-16T04:40:14.617159+0000","last_clean":"2024-09-16T04:40:14.617159+0000","last_became_active":"2024-09-16T04:40:13.629885+0000","last_became_peered":"2024-09-16T04:40:13.629885+0000","last_unstale":"2024-09-16T04:40:14.617159+0000","last_undegraded":"2024-09-16T04:40:14.617159+0000","last_fullsized":"2024-09-16T04:40:14.617159+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:21.446474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.616836+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.616836+0000","last_undegraded":"2024-09-16T04:40:12.616836+0000","last_fullsized":"2024-09-16T04:40:12.616836+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.616883+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.616883+0000","last_undegraded":"2024-09-16T04:40:12.616883+0000","last_fullsized":"2024-09-16T04:40:12.616883+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.705352+0000","last_change":"2024-09-16T04:40:13.626272+0000","last_active":"2024-09-16T04:40:14.705352+0000","last_peered":"2024-09-16T04:40:14.705352+0000","last_clean":"2024-09-16T04:40:14.705352+0000","last_became_active":"2024-09-16T04:40:13.625866+0000","last_became_peered":"2024-09-16T04:40:13.625866+0000","last_unstale":"2024-09-16T04:40:14.705352+0000","last_undegraded":"2024-09-16T04:40:14.705352+0000","last_fullsized":"2024-09-16T04:40:14.705352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:41.631481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617676+0000","last_change":"2024-09-16T04:40:13.630616+0000","last_active":"2024-09-16T04:40:14.617676+0000","last_peered":"2024-09-16T04:40:14.617676+0000","last_clean":"2024-09-16T04:40:14.617676+0000","last_became_active":"2024-09-16T04:40:13.630240+0000","last_became_peered":"2024-09-16T04:40:13.630240+0000","last_unstale":"2024-09-16T04:40:14.617676+0000","last_undegraded":"2024-09-16T04:40:14.617676+0000","last_fullsized":"2024-09-16T04:40:14.617676+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:25.350686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.618315+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.618315+0000","last_undegraded":"2024-09-16T04:40:12.618315+0000","last_fullsized":"2024-09-16T04:40:12.618315+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.616925+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.616925+0000","last_undegraded":"2024-09-16T04:40:12.616925+0000","last_fullsized":"2024-09-16T04:40:12.616925+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-16T04:40:12.617213+0000","last_change":"2024-09-16T04:40:11.606968+0000","last_active":"2024-09-16T04:40:11.606968+0000","last_peered":"2024-09-16T04:40:11.606968+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:40:12.617213+0000","last_undegraded":"2024-09-16T04:40:12.617213+0000","last_fullsized":"2024-09-16T04:40:12.617213+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617689+0000","last_change":"2024-09-16T04:40:13.638219+0000","last_active":"2024-09-16T04:40:14.617689+0000","last_peered":"2024-09-16T04:40:14.617689+0000","last_clean":"2024-09-16T04:40:14.617689+0000","last_became_active":"2024-09-16T04:40:13.637832+0000","last_became_peered":"2024-09-16T04:40:13.637832+0000","last_unstale":"2024-09-16T04:40:14.617689+0000","last_undegraded":"2024-09-16T04:40:14.617689+0000","last_fullsized":"2024-09-16T04:40:14.617689+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:04:38.388133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617696+0000","last_change":"2024-09-16T04:40:13.632248+0000","last_active":"2024-09-16T04:40:14.617696+0000","last_peered":"2024-09-16T04:40:14.617696+0000","last_clean":"2024-09-16T04:40:14.617696+0000","last_became_active":"2024-09-16T04:40:13.630436+0000","last_became_peered":"2024-09-16T04:40:13.630436+0000","last_unstale":"2024-09-16T04:40:14.617696+0000","last_undegraded":"2024-09-16T04:40:14.617696+0000","last_fullsized":"2024-09-16T04:40:14.617696+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:21:17.640198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617912+0000","last_change":"2024-09-16T04:40:13.629954+0000","last_active":"2024-09-16T04:40:14.617912+0000","last_peered":"2024-09-16T04:40:14.617912+0000","last_clean":"2024-09-16T04:40:14.617912+0000","last_became_active":"2024-09-16T04:40:13.629576+0000","last_became_peered":"2024-09-16T04:40:13.629576+0000","last_unstale":"2024-09-16T04:40:14.617912+0000","last_undegraded":"2024-09-16T04:40:14.617912+0000","last_fullsized":"2024-09-16T04:40:14.617912+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:02.490377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617336+0000","last_change":"2024-09-16T04:40:13.631818+0000","last_active":"2024-09-16T04:40:14.617336+0000","last_peered":"2024-09-16T04:40:14.617336+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:40:13.631383+0000","last_became_peered":"2024-09-16T04:40:13.631383+0000","last_unstale":"2024-09-16T04:40:14.617336+0000","last_undegraded":"2024-09-16T04:40:14.617336+0000","last_fullsized":"2024-09-16T04:40:14.617336+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:44:30.068689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617518+0000","last_change":"2024-09-16T04:40:13.630037+0000","last_active":"2024-09-16T04:40:14.617518+0000","last_peered":"2024-09-16T04:40:14.617518+0000","last_clean":"2024-09-16T04:40:14.617518+0000","last_became_active":"2024-09-16T04:40:13.629643+0000","last_became_peered":"2024-09-16T04:40:13.629643+0000","last_unstale":"2024-09-16T04:40:14.617518+0000","last_undegraded":"2024-09-16T04:40:14.617518+0000","last_fullsized":"2024-09-16T04:40:14.617518+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:27.740812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":31760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":35183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:40:21.749 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:40:22.033 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:22.033 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:40:22.045 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-16T04:40:20.544012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108696,"kb_used_data":560,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378664,"statfs":{"total":386547056640,"available":386435751936,"internally_reserved":0,"allocated":573440,"data_stored":170680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617230+0000","last_change":"2024-09-16T04:40:13.631781+0000","last_active":"2024-09-16T04:40:14.617230+0000","last_peered":"2024-09-16T04:40:14.617230+0000","last_clean":"2024-09-16T04:40:14.617230+0000","last_became_active":"2024-09-16T04:40:13.631321+0000","last_became_peered":"2024-09-16T04:40:13.631321+0000","last_unstale":"2024-09-16T04:40:14.617230+0000","last_undegraded":"2024-09-16T04:40:14.617230+0000","last_fullsized":"2024-09-16T04:40:14.617230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:50:37.890096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617887+0000","last_change":"2024-09-16T04:40:13.637120+0000","last_active":"2024-09-16T04:40:14.617887+0000","last_peered":"2024-09-16T04:40:14.617887+0000","last_clean":"2024-09-16T04:40:14.617887+0000","last_became_active":"2024-09-16T04:40:13.636399+0000","last_became_peered":"2024-09-16T04:40:13.636399+0000","last_unstale":"2024-09-16T04:40:14.617887+0000","last_undegraded":"2024-09-16T04:40:14.617887+0000","last_fullsized":"2024-09-16T04:40:14.617887+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:42.568657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617151+0000","last_change":"2024-09-16T04:40:13.633473+0000","last_active":"2024-09-16T04:40:14.617151+0000","last_peered":"2024-09-16T04:40:14.617151+0000","last_clean":"2024-09-16T04:40:14.617151+0000","last_became_active":"2024-09-16T04:40:13.628514+0000","last_became_peered":"2024-09-16T04:40:13.628514+0000","last_unstale":"2024-09-16T04:40:14.617151+0000","last_undegraded":"2024-09-16T04:40:14.617151+0000","last_fullsized":"2024-09-16T04:40:14.617151+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:43:39.380916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617159+0000","last_change":"2024-09-16T04:40:13.632054+0000","last_active":"2024-09-16T04:40:14.617159+0000","last_peered":"2024-09-16T04:40:14.617159+0000","last_clean":"2024-09-16T04:40:14.617159+0000","last_became_active":"2024-09-16T04:40:13.629885+0000","last_became_peered":"2024-09-16T04:40:13.629885+0000","last_unstale":"2024-09-16T04:40:14.617159+0000","last_undegraded":"2024-09-16T04:40:14.617159+0000","last_fullsized":"2024-09-16T04:40:14.617159+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:21.446474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617067+0000","last_change":"2024-09-16T04:40:13.628449+0000","last_active":"2024-09-16T04:40:14.617067+0000","last_peered":"2024-09-16T04:40:14.617067+0000","last_clean":"2024-09-16T04:40:14.617067+0000","last_became_active":"2024-09-16T04:40:13.628014+0000","last_became_peered":"2024-09-16T04:40:13.628014+0000","last_unstale":"2024-09-16T04:40:14.617067+0000","last_undegraded":"2024-09-16T04:40:14.617067+0000","last_fullsized":"2024-09-16T04:40:14.617067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:02.156867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.616606+0000","last_change":"2024-09-16T04:40:13.630582+0000","last_active":"2024-09-16T04:40:14.616606+0000","last_peered":"2024-09-16T04:40:14.616606+0000","last_clean":"2024-09-16T04:40:14.616606+0000","last_became_active":"2024-09-16T04:40:13.629641+0000","last_became_peered":"2024-09-16T04:40:13.629641+0000","last_unstale":"2024-09-16T04:40:14.616606+0000","last_undegraded":"2024-09-16T04:40:14.616606+0000","last_fullsized":"2024-09-16T04:40:14.616606+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:04.041415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.705352+0000","last_change":"2024-09-16T04:40:13.626272+0000","last_active":"2024-09-16T04:40:14.705352+0000","last_peered":"2024-09-16T04:40:14.705352+0000","last_clean":"2024-09-16T04:40:14.705352+0000","last_became_active":"2024-09-16T04:40:13.625866+0000","last_became_peered":"2024-09-16T04:40:13.625866+0000","last_unstale":"2024-09-16T04:40:14.705352+0000","last_undegraded":"2024-09-16T04:40:14.705352+0000","last_fullsized":"2024-09-16T04:40:14.705352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:41.631481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617676+0000","last_change":"2024-09-16T04:40:13.630616+0000","last_active":"2024-09-16T04:40:14.617676+0000","last_peered":"2024-09-16T04:40:14.617676+0000","last_clean":"2024-09-16T04:40:14.617676+0000","last_became_active":"2024-09-16T04:40:13.630240+0000","last_became_peered":"2024-09-16T04:40:13.630240+0000","last_unstale":"2024-09-16T04:40:14.617676+0000","last_undegraded":"2024-09-16T04:40:14.617676+0000","last_fullsized":"2024-09-16T04:40:14.617676+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:25.350686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"16'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.616849+0000","last_change":"2024-09-16T04:40:13.630775+0000","last_active":"2024-09-16T04:40:14.616849+0000","last_peered":"2024-09-16T04:40:14.616849+0000","last_clean":"2024-09-16T04:40:14.616849+0000","last_became_active":"2024-09-16T04:40:13.630179+0000","last_became_peered":"2024-09-16T04:40:13.630179+0000","last_unstale":"2024-09-16T04:40:14.616849+0000","last_undegraded":"2024-09-16T04:40:14.616849+0000","last_fullsized":"2024-09-16T04:40:14.616849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:14:57.037845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617299+0000","last_change":"2024-09-16T04:40:13.628691+0000","last_active":"2024-09-16T04:40:14.617299+0000","last_peered":"2024-09-16T04:40:14.617299+0000","last_clean":"2024-09-16T04:40:14.617299+0000","last_became_active":"2024-09-16T04:40:13.628306+0000","last_became_peered":"2024-09-16T04:40:13.628306+0000","last_unstale":"2024-09-16T04:40:14.617299+0000","last_undegraded":"2024-09-16T04:40:14.617299+0000","last_fullsized":"2024-09-16T04:40:14.617299+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:23:18.162483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.616961+0000","last_change":"2024-09-16T04:40:13.633606+0000","last_active":"2024-09-16T04:40:14.616961+0000","last_peered":"2024-09-16T04:40:14.616961+0000","last_clean":"2024-09-16T04:40:14.616961+0000","last_became_active":"2024-09-16T04:40:13.633197+0000","last_became_peered":"2024-09-16T04:40:13.633197+0000","last_unstale":"2024-09-16T04:40:14.616961+0000","last_undegraded":"2024-09-16T04:40:14.616961+0000","last_fullsized":"2024-09-16T04:40:14.616961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:58:57.647169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617689+0000","last_change":"2024-09-16T04:40:13.638219+0000","last_active":"2024-09-16T04:40:14.617689+0000","last_peered":"2024-09-16T04:40:14.617689+0000","last_clean":"2024-09-16T04:40:14.617689+0000","last_became_active":"2024-09-16T04:40:13.637832+0000","last_became_peered":"2024-09-16T04:40:13.637832+0000","last_unstale":"2024-09-16T04:40:14.617689+0000","last_undegraded":"2024-09-16T04:40:14.617689+0000","last_fullsized":"2024-09-16T04:40:14.617689+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:04:38.388133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617696+0000","last_change":"2024-09-16T04:40:13.632248+0000","last_active":"2024-09-16T04:40:14.617696+0000","last_peered":"2024-09-16T04:40:14.617696+0000","last_clean":"2024-09-16T04:40:14.617696+0000","last_became_active":"2024-09-16T04:40:13.630436+0000","last_became_peered":"2024-09-16T04:40:13.630436+0000","last_unstale":"2024-09-16T04:40:14.617696+0000","last_undegraded":"2024-09-16T04:40:14.617696+0000","last_fullsized":"2024-09-16T04:40:14.617696+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:21:17.640198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617912+0000","last_change":"2024-09-16T04:40:13.629954+0000","last_active":"2024-09-16T04:40:14.617912+0000","last_peered":"2024-09-16T04:40:14.617912+0000","last_clean":"2024-09-16T04:40:14.617912+0000","last_became_active":"2024-09-16T04:40:13.629576+0000","last_became_peered":"2024-09-16T04:40:13.629576+0000","last_unstale":"2024-09-16T04:40:14.617912+0000","last_undegraded":"2024-09-16T04:40:14.617912+0000","last_fullsized":"2024-09-16T04:40:14.617912+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:02.490377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617336+0000","last_change":"2024-09-16T04:40:13.631818+0000","last_active":"2024-09-16T04:40:14.617336+0000","last_peered":"2024-09-16T04:40:14.617336+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:40:13.631383+0000","last_became_peered":"2024-09-16T04:40:13.631383+0000","last_unstale":"2024-09-16T04:40:14.617336+0000","last_undegraded":"2024-09-16T04:40:14.617336+0000","last_fullsized":"2024-09-16T04:40:14.617336+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:44:30.068689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T04:40:14.617518+0000","last_change":"2024-09-16T04:40:13.630037+0000","last_active":"2024-09-16T04:40:14.617518+0000","last_peered":"2024-09-16T04:40:14.617518+0000","last_clean":"2024-09-16T04:40:14.617518+0000","last_became_active":"2024-09-16T04:40:13.629643+0000","last_became_peered":"2024-09-16T04:40:13.629643+0000","last_unstale":"2024-09-16T04:40:14.617518+0000","last_undegraded":"2024-09-16T04:40:14.617518+0000","last_fullsized":"2024-09-16T04:40:14.617518+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:27.740812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":42670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":143360,"data_stored":42670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":143360,"data_stored":42670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":42670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:40:22.046 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-09-16T04:40:22.046 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-09-16T04:40:22.047 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-09-16T04:40:22.098 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-09-16T04:40:22.098 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-09-16T04:40:22.149 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_1 2024-09-16T04:40:22.150 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_1 2024-09-16T04:40:22.200 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-09-16T04:40:22.201 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-09-16T04:40:22.251 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-09-16T04:40:22.252 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-09-16T04:40:22.303 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_2 2024-09-16T04:40:22.304 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_2 2024-09-16T04:40:22.353 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-09-16T04:40:22.354 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-09-16T04:40:22.406 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-09-16T04:40:22.406 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-09-16T04:40:22.460 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_3 2024-09-16T04:40:22.460 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_3 2024-09-16T04:40:22.510 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-09-16T04:40:22.510 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-09-16T04:40:22.561 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-09-16T04:40:22.561 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-09-16T04:40:22.613 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_4 2024-09-16T04:40:22.614 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_4 2024-09-16T04:40:22.664 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-09-16T04:40:22.664 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-09-16T04:40:22.715 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-09-16T04:40:22.715 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-09-16T04:40:22.769 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_5 2024-09-16T04:40:22.769 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_5 2024-09-16T04:40:22.819 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-09-16T04:40:22.819 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-09-16T04:40:22.870 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-09-16T04:40:22.870 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-09-16T04:40:22.921 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_6 2024-09-16T04:40:22.921 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_6 2024-09-16T04:40:22.971 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-09-16T04:40:22.971 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-09-16T04:40:23.022 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-09-16T04:40:23.022 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-09-16T04:40:23.073 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_7 2024-09-16T04:40:23.073 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_7 2024-09-16T04:40:23.123 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-09-16T04:40:23.123 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-09-16T04:40:23.177 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-09-16T04:40:23.177 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-09-16T04:40:23.228 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_8 2024-09-16T04:40:23.229 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_8 2024-09-16T04:40:23.278 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-09-16T04:40:23.278 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-09-16T04:40:23.330 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-09-16T04:40:23.330 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-09-16T04:40:23.381 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_9 2024-09-16T04:40:23.381 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_9 2024-09-16T04:40:23.431 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 injectargs '--osd-recovery-delay-start 1000 --osd-min-pg-log-entries 100000000' 2024-09-16T04:40:23.552 INFO:teuthology.orchestra.run.smithi152.stdout:{} 2024-09-16T04:40:23.553 INFO:teuthology.orchestra.run.smithi152.stderr:osd_min_pg_log_entries = '' (not observed, change may require restart) osd_recovery_delay_start = '' 2024-09-16T04:40:23.563 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T04:40:23.564 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:40:23.599 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T04:40:23.599 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2024-09-16T04:40:23.929 INFO:tasks.ceph.mon.a.smithi152.stderr:2024-09-16T04:40:23.928+0000 7f6a81e5e640 -1 mon.a@0(leader).osd e17 definitely_dead 0 2024-09-16T04:40:23.929 INFO:teuthology.orchestra.run.smithi152.stderr:osd.0 is already down. 2024-09-16T04:40:23.942 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T04:40:23.942 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:40:23.969 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T04:40:23.970 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 3 2024-09-16T04:40:24.299 INFO:tasks.ceph.mon.a.smithi152.stderr:2024-09-16T04:40:24.297+0000 7f6a81e5e640 -1 mon.a@0(leader).osd e17 definitely_dead 0 2024-09-16T04:40:24.644 INFO:teuthology.orchestra.run.smithi152.stderr:marked down osd.3. 2024-09-16T04:40:24.660 INFO:tasks.util.rados:rados -p unique_pool_0 put new_1 /etc/group 2024-09-16T04:40:24.661 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_1 /etc/group 2024-09-16T04:40:25.656 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-09-16T04:40:25.657 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-09-16T04:40:25.712 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-09-16T04:40:25.713 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-09-16T04:40:25.765 INFO:tasks.util.rados:rados -p unique_pool_0 put new_2 /etc/group 2024-09-16T04:40:25.765 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_2 /etc/group 2024-09-16T04:40:25.816 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-09-16T04:40:25.816 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-09-16T04:40:25.866 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-09-16T04:40:25.867 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-09-16T04:40:25.918 INFO:tasks.util.rados:rados -p unique_pool_0 put new_3 /etc/group 2024-09-16T04:40:25.918 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_3 /etc/group 2024-09-16T04:40:25.969 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-09-16T04:40:25.969 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-09-16T04:40:26.020 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-09-16T04:40:26.020 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-09-16T04:40:26.070 INFO:tasks.util.rados:rados -p unique_pool_0 put new_4 /etc/group 2024-09-16T04:40:26.071 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_4 /etc/group 2024-09-16T04:40:26.121 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-09-16T04:40:26.121 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-09-16T04:40:26.172 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-09-16T04:40:26.172 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-09-16T04:40:26.223 INFO:tasks.util.rados:rados -p unique_pool_0 put new_5 /etc/group 2024-09-16T04:40:26.223 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_5 /etc/group 2024-09-16T04:40:26.274 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-09-16T04:40:26.274 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-09-16T04:40:26.325 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-09-16T04:40:26.325 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-09-16T04:40:26.375 INFO:tasks.util.rados:rados -p unique_pool_0 put new_6 /etc/group 2024-09-16T04:40:26.376 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_6 /etc/group 2024-09-16T04:40:26.426 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-09-16T04:40:26.427 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-09-16T04:40:26.477 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-09-16T04:40:26.477 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-09-16T04:40:26.529 INFO:tasks.util.rados:rados -p unique_pool_0 put new_7 /etc/group 2024-09-16T04:40:26.529 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_7 /etc/group 2024-09-16T04:40:26.581 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-09-16T04:40:26.581 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-09-16T04:40:26.632 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-09-16T04:40:26.632 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-09-16T04:40:26.683 INFO:tasks.util.rados:rados -p unique_pool_0 put new_8 /etc/group 2024-09-16T04:40:26.683 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_8 /etc/group 2024-09-16T04:40:26.734 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-09-16T04:40:26.734 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-09-16T04:40:26.785 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-09-16T04:40:26.785 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-09-16T04:40:26.836 INFO:tasks.util.rados:rados -p unique_pool_0 put new_9 /etc/group 2024-09-16T04:40:26.836 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_9 /etc/group 2024-09-16T04:40:26.887 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-09-16T04:40:26.888 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-09-16T04:40:26.938 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-09-16T04:40:26.938 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-09-16T04:40:26.990 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T04:40:26.990 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:40:27.023 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T04:40:27.023 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2024-09-16T04:40:27.346 INFO:tasks.ceph.mon.a.smithi152.stderr:2024-09-16T04:40:27.345+0000 7f6a81e5e640 -1 mon.a@0(leader).osd e19 definitely_dead 0 2024-09-16T04:40:27.663 INFO:teuthology.orchestra.run.smithi152.stderr:marked down osd.1. 2024-09-16T04:40:27.677 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd lost 1 --yes-i-really-mean-it 2024-09-16T04:40:28.666 INFO:teuthology.orchestra.run.smithi152.stderr:marked osd lost in epoch 20 2024-09-16T04:40:28.681 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T04:40:28.681 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T04:40:28.684 INFO:tasks.ceph.osd.0:Started 2024-09-16T04:40:28.684 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:40:28.785 INFO:teuthology.orchestra.run.smithi152.stderr:no valid command found; 10 closest matches: 2024-09-16T04:40:28.785 INFO:teuthology.orchestra.run.smithi152.stderr:0 2024-09-16T04:40:28.785 INFO:teuthology.orchestra.run.smithi152.stderr:1 2024-09-16T04:40:28.785 INFO:teuthology.orchestra.run.smithi152.stderr:2 2024-09-16T04:40:28.785 INFO:teuthology.orchestra.run.smithi152.stderr:abort 2024-09-16T04:40:28.785 INFO:teuthology.orchestra.run.smithi152.stderr:assert 2024-09-16T04:40:28.785 INFO:teuthology.orchestra.run.smithi152.stderr:config diff 2024-09-16T04:40:28.785 INFO:teuthology.orchestra.run.smithi152.stderr:config diff get 2024-09-16T04:40:28.785 INFO:teuthology.orchestra.run.smithi152.stderr:config get 2024-09-16T04:40:28.786 INFO:teuthology.orchestra.run.smithi152.stderr:config help [] 2024-09-16T04:40:28.786 INFO:teuthology.orchestra.run.smithi152.stderr:config set ... 2024-09-16T04:40:28.786 INFO:teuthology.orchestra.run.smithi152.stderr:admin_socket: invalid command 2024-09-16T04:40:28.787 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:40:28.788 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-16T04:40:29.853 INFO:tasks.ceph.osd.0.smithi152.stderr:2024-09-16T04:40:29.852+0000 7fb1ac631640 -1 Falling back to public interface 2024-09-16T04:40:33.364 INFO:tasks.ceph.osd.0.smithi152.stderr:2024-09-16T04:40:33.362+0000 7fb1ac631640 -1 osd.0 16 log_to_monitors true 2024-09-16T04:40:33.738 INFO:tasks.ceph.osd.0.smithi152.stderr:2024-09-16T04:40:33.736+0000 7fb1a43e8640 -1 osd.0 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:40:33.788 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:40:33.902 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:40:33.903 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-16T04:40:33.903 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-16T04:40:33.903 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:40:33.911 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.0 to be up 2024-09-16T04:40:33.912 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:40:34.187 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:34.187 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":22,"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","created":"2024-09-16T04:39:47.674616+0000","modified":"2024-09-16T04:40:33.722971+0000","last_up_change":"2024-09-16T04:40:27.655473+0000","last_in_change":"2024-09-16T04:39:49.620271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-16T04:40:10.631384+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"d8b076bc-f4ae-4100-8cdf-78560225fea9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":36071542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6801","nonce":36071542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":36071542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":36071542}]},"public_addr":"172.21.15.152:6800/36071542","cluster_addr":"172.21.15.152:6801/36071542","heartbeat_back_addr":"172.21.15.152:6803/36071542","heartbeat_front_addr":"172.21.15.152:6802/36071542","state":["exists"]},{"osd":1,"uuid":"023d4748-7485-4d8b-8594-54a238b67b71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1844087228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":1844087228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6811","nonce":1844087228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1844087228}]},"public_addr":"172.21.15.152:6808/1844087228","cluster_addr":"172.21.15.152:6809/1844087228","heartbeat_back_addr":"172.21.15.152:6811/1844087228","heartbeat_front_addr":"172.21.15.152:6810/1844087228","state":["exists"]},{"osd":2,"uuid":"c7bfb282-86b4-4bdc-902d-0a0cfbadf5ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2854825795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2854825795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":2854825795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2854825795}]},"public_addr":"172.21.15.152:6804/2854825795","cluster_addr":"172.21.15.152:6805/2854825795","heartbeat_back_addr":"172.21.15.152:6807/2854825795","heartbeat_front_addr":"172.21.15.152:6806/2854825795","state":["exists","up"]},{"osd":3,"uuid":"f82045bc-6fe9-4755-8c25-ea7738b90d6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3902150580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6813","nonce":3902150580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":3902150580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3902150580}]},"public_addr":"172.21.15.152:6812/3902150580","cluster_addr":"172.21.15.152:6813/3902150580","heartbeat_back_addr":"172.21.15.152:6815/3902150580","heartbeat_front_addr":"172.21.15.152:6814/3902150580","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.429729+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.175556+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.243970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.373375+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:40:37.201 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:40:37.492 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:37.492 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":24,"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","created":"2024-09-16T04:39:47.674616+0000","modified":"2024-09-16T04:40:35.734720+0000","last_up_change":"2024-09-16T04:40:34.731345+0000","last_in_change":"2024-09-16T04:39:49.620271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-16T04:40:10.631384+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"d8b076bc-f4ae-4100-8cdf-78560225fea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":23,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4015439555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6801","nonce":4015439555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":4015439555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4015439555}]},"public_addr":"172.21.15.152:6800/4015439555","cluster_addr":"172.21.15.152:6801/4015439555","heartbeat_back_addr":"172.21.15.152:6803/4015439555","heartbeat_front_addr":"172.21.15.152:6802/4015439555","state":["exists","up"]},{"osd":1,"uuid":"023d4748-7485-4d8b-8594-54a238b67b71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1844087228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":1844087228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6811","nonce":1844087228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1844087228}]},"public_addr":"172.21.15.152:6808/1844087228","cluster_addr":"172.21.15.152:6809/1844087228","heartbeat_back_addr":"172.21.15.152:6811/1844087228","heartbeat_front_addr":"172.21.15.152:6810/1844087228","state":["exists"]},{"osd":2,"uuid":"c7bfb282-86b4-4bdc-902d-0a0cfbadf5ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2854825795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2854825795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":2854825795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2854825795}]},"public_addr":"172.21.15.152:6804/2854825795","cluster_addr":"172.21.15.152:6805/2854825795","heartbeat_back_addr":"172.21.15.152:6807/2854825795","heartbeat_front_addr":"172.21.15.152:6806/2854825795","state":["exists","up"]},{"osd":3,"uuid":"f82045bc-6fe9-4755-8c25-ea7738b90d6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3902150580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6813","nonce":3902150580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":3902150580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3902150580}]},"public_addr":"172.21.15.152:6812/3902150580","cluster_addr":"172.21.15.152:6813/3902150580","heartbeat_back_addr":"172.21.15.152:6815/3902150580","heartbeat_front_addr":"172.21.15.152:6814/3902150580","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.429729+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.175556+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.243970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.373375+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:40:37.505 INFO:tasks.ec_lost_unfound.ceph_manager:osd.0 is up 2024-09-16T04:40:37.506 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T04:40:37.506 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T04:40:37.509 INFO:tasks.ceph.osd.3:Started 2024-09-16T04:40:37.509 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:40:37.608 INFO:teuthology.orchestra.run.smithi152.stderr:no valid command found; 10 closest matches: 2024-09-16T04:40:37.608 INFO:teuthology.orchestra.run.smithi152.stderr:0 2024-09-16T04:40:37.608 INFO:teuthology.orchestra.run.smithi152.stderr:1 2024-09-16T04:40:37.608 INFO:teuthology.orchestra.run.smithi152.stderr:2 2024-09-16T04:40:37.608 INFO:teuthology.orchestra.run.smithi152.stderr:abort 2024-09-16T04:40:37.608 INFO:teuthology.orchestra.run.smithi152.stderr:assert 2024-09-16T04:40:37.608 INFO:teuthology.orchestra.run.smithi152.stderr:config diff 2024-09-16T04:40:37.608 INFO:teuthology.orchestra.run.smithi152.stderr:config diff get 2024-09-16T04:40:37.609 INFO:teuthology.orchestra.run.smithi152.stderr:config get 2024-09-16T04:40:37.609 INFO:teuthology.orchestra.run.smithi152.stderr:config help [] 2024-09-16T04:40:37.609 INFO:teuthology.orchestra.run.smithi152.stderr:config set ... 2024-09-16T04:40:37.609 INFO:teuthology.orchestra.run.smithi152.stderr:admin_socket: invalid command 2024-09-16T04:40:37.610 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:40:37.610 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-16T04:40:38.675 INFO:tasks.ceph.osd.3.smithi152.stderr:2024-09-16T04:40:38.673+0000 7fcc0efa6640 -1 Falling back to public interface 2024-09-16T04:40:40.751 INFO:tasks.ceph.mon.a.smithi152.stderr:2024-09-16T04:40:40.749+0000 7f6a84663640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 14 pgs recovery_unfound (PG_DAMAGED) 2024-09-16T04:40:42.612 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:40:42.696 INFO:tasks.ceph.osd.3.smithi152.stderr:2024-09-16T04:40:42.695+0000 7fcc0efa6640 -1 osd.3 17 log_to_monitors true 2024-09-16T04:40:42.722 INFO:teuthology.orchestra.run.smithi152.stderr:no valid command found; 10 closest matches: 2024-09-16T04:40:42.722 INFO:teuthology.orchestra.run.smithi152.stderr:0 2024-09-16T04:40:42.722 INFO:teuthology.orchestra.run.smithi152.stderr:1 2024-09-16T04:40:42.722 INFO:teuthology.orchestra.run.smithi152.stderr:2 2024-09-16T04:40:42.722 INFO:teuthology.orchestra.run.smithi152.stderr:abort 2024-09-16T04:40:42.722 INFO:teuthology.orchestra.run.smithi152.stderr:assert 2024-09-16T04:40:42.722 INFO:teuthology.orchestra.run.smithi152.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:40:42.722 INFO:teuthology.orchestra.run.smithi152.stderr:bluefs files list 2024-09-16T04:40:42.722 INFO:teuthology.orchestra.run.smithi152.stderr:bluefs stats 2024-09-16T04:40:42.722 INFO:teuthology.orchestra.run.smithi152.stderr:bluestore allocator dump block 2024-09-16T04:40:42.722 INFO:teuthology.orchestra.run.smithi152.stderr:bluestore allocator fragmentation block 2024-09-16T04:40:42.722 INFO:teuthology.orchestra.run.smithi152.stderr:admin_socket: invalid command 2024-09-16T04:40:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:40:42.725 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-16T04:40:42.820 INFO:tasks.ceph.osd.3.smithi152.stderr:2024-09-16T04:40:42.819+0000 7fcc06d5d640 -1 osd.3 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:40:47.726 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:40:47.843 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:40:47.843 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [], 2024-09-16T04:40:47.843 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 0 2024-09-16T04:40:47.843 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:40:47.852 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.3 to be up 2024-09-16T04:40:47.853 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:40:48.129 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:48.129 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":27,"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","created":"2024-09-16T04:39:47.674616+0000","modified":"2024-09-16T04:40:44.822649+0000","last_up_change":"2024-09-16T04:40:43.812561+0000","last_in_change":"2024-09-16T04:39:49.620271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-16T04:40:10.631384+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"d8b076bc-f4ae-4100-8cdf-78560225fea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":26,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4015439555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6801","nonce":4015439555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":4015439555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4015439555}]},"public_addr":"172.21.15.152:6800/4015439555","cluster_addr":"172.21.15.152:6801/4015439555","heartbeat_back_addr":"172.21.15.152:6803/4015439555","heartbeat_front_addr":"172.21.15.152:6802/4015439555","state":["exists","up"]},{"osd":1,"uuid":"023d4748-7485-4d8b-8594-54a238b67b71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1844087228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":1844087228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6811","nonce":1844087228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1844087228}]},"public_addr":"172.21.15.152:6808/1844087228","cluster_addr":"172.21.15.152:6809/1844087228","heartbeat_back_addr":"172.21.15.152:6811/1844087228","heartbeat_front_addr":"172.21.15.152:6810/1844087228","state":["exists"]},{"osd":2,"uuid":"c7bfb282-86b4-4bdc-902d-0a0cfbadf5ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2854825795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2854825795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":2854825795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2854825795}]},"public_addr":"172.21.15.152:6804/2854825795","cluster_addr":"172.21.15.152:6805/2854825795","heartbeat_back_addr":"172.21.15.152:6807/2854825795","heartbeat_front_addr":"172.21.15.152:6806/2854825795","state":["exists","up"]},{"osd":3,"uuid":"f82045bc-6fe9-4755-8c25-ea7738b90d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":26,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1506610745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":1506610745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6811","nonce":1506610745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1506610745}]},"public_addr":"172.21.15.152:6808/1506610745","cluster_addr":"172.21.15.152:6809/1506610745","heartbeat_back_addr":"172.21.15.152:6811/1506610745","heartbeat_front_addr":"172.21.15.152:6810/1506610745","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.429729+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.175556+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.243970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.373375+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:40:48.142 INFO:tasks.ec_lost_unfound.ceph_manager:osd.3 is up 2024-09-16T04:40:48.142 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T04:40:48.143 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T04:40:48.143 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T04:40:48.282 INFO:teuthology.orchestra.run.smithi152.stdout:98784247812 2024-09-16T04:40:48.282 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:40:48.285 INFO:teuthology.orchestra.run.smithi152.stdout:111669149699 2024-09-16T04:40:48.285 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:40:48.286 INFO:teuthology.orchestra.run.smithi152.stdout:38654705678 2024-09-16T04:40:48.286 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:40:48.579 INFO:teuthology.orchestra.run.smithi152.stdout:98784247811 2024-09-16T04:40:48.592 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247812 got 98784247811 for osd.0 2024-09-16T04:40:48.619 INFO:teuthology.orchestra.run.smithi152.stdout:1 2024-09-16T04:40:48.625 INFO:teuthology.orchestra.run.smithi152.stdout:38654705677 2024-09-16T04:40:48.633 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149699 got 1 for osd.3 2024-09-16T04:40:48.638 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705678 got 38654705677 for osd.2 2024-09-16T04:40:49.593 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:40:49.634 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:40:49.638 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:40:49.900 INFO:teuthology.orchestra.run.smithi152.stdout:98784247812 2024-09-16T04:40:49.913 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247812 got 98784247812 for osd.0 2024-09-16T04:40:49.913 DEBUG:teuthology.parallel:result is None 2024-09-16T04:40:49.937 INFO:teuthology.orchestra.run.smithi152.stdout:38654705678 2024-09-16T04:40:49.945 INFO:teuthology.orchestra.run.smithi152.stdout:111669149699 2024-09-16T04:40:49.950 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705678 got 38654705678 for osd.2 2024-09-16T04:40:49.950 DEBUG:teuthology.parallel:result is None 2024-09-16T04:40:49.958 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149699 got 111669149699 for osd.3 2024-09-16T04:40:49.958 DEBUG:teuthology.parallel:result is None 2024-09-16T04:40:49.959 INFO:tasks.ec_lost_unfound.ceph_manager:waiting till active 2024-09-16T04:40:49.959 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:40:50.273 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:50.273 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:40:50.284 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":49,"stamp":"2024-09-16T04:40:48.550015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":110124,"kb_used_data":1032,"kb_used_omap":11,"kb_used_meta":108980,"kb_avail":377377236,"statfs":{"total":386547056640,"available":386434289664,"internally_reserved":0,"allocated":1056768,"data_stored":495923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11993,"internal_metadata":111595815},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001664"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+undersized","last_fresh":"2024-09-16T04:40:44.842322+0000","last_change":"2024-09-16T04:40:44.842163+0000","last_active":"2024-09-16T04:40:44.842322+0000","last_peered":"2024-09-16T04:40:44.842322+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:40:44.842163+0000","last_became_peered":"2024-09-16T04:40:44.842163+0000","last_unstale":"2024-09-16T04:40:44.842322+0000","last_undegraded":"2024-09-16T04:40:44.842322+0000","last_fullsized":"2024-09-16T04:40:44.833950+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"19'4","reported_seq":43,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:40:44.853190+0000","last_change":"2024-09-16T04:40:44.853621+0000","last_active":"2024-09-16T04:40:44.853190+0000","last_peered":"2024-09-16T04:40:44.853190+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:40:44.848800+0000","last_became_peered":"2024-09-16T04:40:44.848800+0000","last_unstale":"2024-09-16T04:40:44.853190+0000","last_undegraded":"2024-09-16T04:40:44.837670+0000","last_fullsized":"2024-09-16T04:40:44.837294+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"19'3","reported_seq":54,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:40:44.846595+0000","last_change":"2024-09-16T04:40:44.846856+0000","last_active":"2024-09-16T04:40:44.846595+0000","last_peered":"2024-09-16T04:40:44.846595+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:40:44.839558+0000","last_became_peered":"2024-09-16T04:40:44.839558+0000","last_unstale":"2024-09-16T04:40:44.846595+0000","last_undegraded":"2024-09-16T04:40:44.832928+0000","last_fullsized":"2024-09-16T04:40:44.832628+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4","reported_seq":43,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:40:44.850354+0000","last_change":"2024-09-16T04:40:44.850757+0000","last_active":"2024-09-16T04:40:44.850354+0000","last_peered":"2024-09-16T04:40:44.850354+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:40:44.848625+0000","last_became_peered":"2024-09-16T04:40:44.848625+0000","last_unstale":"2024-09-16T04:40:44.850354+0000","last_undegraded":"2024-09-16T04:40:44.837318+0000","last_fullsized":"2024-09-16T04:40:44.835886+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3","reported_seq":31,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:40:44.851070+0000","last_change":"2024-09-16T04:40:44.851339+0000","last_active":"2024-09-16T04:40:44.851070+0000","last_peered":"2024-09-16T04:40:44.851070+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:40:44.843541+0000","last_became_peered":"2024-09-16T04:40:44.843541+0000","last_unstale":"2024-09-16T04:40:44.851070+0000","last_undegraded":"2024-09-16T04:40:44.835930+0000","last_fullsized":"2024-09-16T04:40:44.834220+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+undersized","last_fresh":"2024-09-16T04:40:44.842527+0000","last_change":"2024-09-16T04:40:44.842247+0000","last_active":"2024-09-16T04:40:44.842527+0000","last_peered":"2024-09-16T04:40:44.842527+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:40:44.842247+0000","last_became_peered":"2024-09-16T04:40:44.842247+0000","last_unstale":"2024-09-16T04:40:44.842527+0000","last_undegraded":"2024-09-16T04:40:44.842527+0000","last_fullsized":"2024-09-16T04:40:44.833664+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"19'6","reported_seq":45,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:40:44.845139+0000","last_change":"2024-09-16T04:40:44.845464+0000","last_active":"2024-09-16T04:40:44.845139+0000","last_peered":"2024-09-16T04:40:44.845139+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:40:44.842032+0000","last_became_peered":"2024-09-16T04:40:44.842032+0000","last_unstale":"2024-09-16T04:40:44.845139+0000","last_undegraded":"2024-09-16T04:40:44.835742+0000","last_fullsized":"2024-09-16T04:40:44.835328+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'5","reported_seq":44,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:40:44.851914+0000","last_change":"2024-09-16T04:40:44.852220+0000","last_active":"2024-09-16T04:40:44.851914+0000","last_peered":"2024-09-16T04:40:44.851914+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:40:44.848808+0000","last_became_peered":"2024-09-16T04:40:44.848808+0000","last_unstale":"2024-09-16T04:40:44.851914+0000","last_undegraded":"2024-09-16T04:40:44.836618+0000","last_fullsized":"2024-09-16T04:40:44.836113+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'9","reported_seq":36,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:40:44.847377+0000","last_change":"2024-09-16T04:40:44.847852+0000","last_active":"2024-09-16T04:40:44.847377+0000","last_peered":"2024-09-16T04:40:44.847377+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:40:44.842734+0000","last_became_peered":"2024-09-16T04:40:44.842734+0000","last_unstale":"2024-09-16T04:40:44.847377+0000","last_undegraded":"2024-09-16T04:40:44.833363+0000","last_fullsized":"2024-09-16T04:40:44.833064+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"19'2","reported_seq":52,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:40:44.849502+0000","last_change":"2024-09-16T04:40:44.849992+0000","last_active":"2024-09-16T04:40:44.849502+0000","last_peered":"2024-09-16T04:40:44.849502+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:40:44.839905+0000","last_became_peered":"2024-09-16T04:40:44.839905+0000","last_unstale":"2024-09-16T04:40:44.849502+0000","last_undegraded":"2024-09-16T04:40:44.833540+0000","last_fullsized":"2024-09-16T04:40:44.833235+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'5","reported_seq":32,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:40:44.845757+0000","last_change":"2024-09-16T04:40:44.846078+0000","last_active":"2024-09-16T04:40:44.845757+0000","last_peered":"2024-09-16T04:40:44.845757+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:40:44.841868+0000","last_became_peered":"2024-09-16T04:40:44.841868+0000","last_unstale":"2024-09-16T04:40:44.845757+0000","last_undegraded":"2024-09-16T04:40:44.835179+0000","last_fullsized":"2024-09-16T04:40:44.834838+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"19'1","reported_seq":17,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:40:44.854206+0000","last_change":"2024-09-16T04:40:44.854206+0000","last_active":"2024-09-16T04:40:44.854206+0000","last_peered":"2024-09-16T04:40:44.854206+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:40:44.843370+0000","last_became_peered":"2024-09-16T04:40:44.843370+0000","last_unstale":"2024-09-16T04:40:44.854206+0000","last_undegraded":"2024-09-16T04:40:44.835077+0000","last_fullsized":"2024-09-16T04:40:44.834737+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"19'1","reported_seq":52,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:40:44.844141+0000","last_change":"2024-09-16T04:40:44.844542+0000","last_active":"2024-09-16T04:40:44.844141+0000","last_peered":"2024-09-16T04:40:44.844141+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:40:44.839353+0000","last_became_peered":"2024-09-16T04:40:44.839353+0000","last_unstale":"2024-09-16T04:40:44.844141+0000","last_undegraded":"2024-09-16T04:40:44.832429+0000","last_fullsized":"2024-09-16T04:40:44.832101+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"19'4","reported_seq":31,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:40:44.855685+0000","last_change":"2024-09-16T04:40:44.855685+0000","last_active":"2024-09-16T04:40:44.855685+0000","last_peered":"2024-09-16T04:40:44.855685+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:40:44.844454+0000","last_became_peered":"2024-09-16T04:40:44.844454+0000","last_unstale":"2024-09-16T04:40:44.855685+0000","last_undegraded":"2024-09-16T04:40:44.836539+0000","last_fullsized":"2024-09-16T04:40:44.836255+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"19'3","reported_seq":55,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:40:44.852522+0000","last_change":"2024-09-16T04:40:44.852882+0000","last_active":"2024-09-16T04:40:44.852522+0000","last_peered":"2024-09-16T04:40:44.852522+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:40:44.840598+0000","last_became_peered":"2024-09-16T04:40:44.840598+0000","last_unstale":"2024-09-16T04:40:44.852522+0000","last_undegraded":"2024-09-16T04:40:44.833574+0000","last_fullsized":"2024-09-16T04:40:44.833243+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"19'5","reported_seq":44,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:40:44.854769+0000","last_change":"2024-09-16T04:40:44.855046+0000","last_active":"2024-09-16T04:40:44.854769+0000","last_peered":"2024-09-16T04:40:44.854769+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:40:44.848727+0000","last_became_peered":"2024-09-16T04:40:44.848727+0000","last_unstale":"2024-09-16T04:40:44.854769+0000","last_undegraded":"2024-09-16T04:40:44.835805+0000","last_fullsized":"2024-09-16T04:40:44.835392+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":237568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":264,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":270336,"data_stored":116951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":28176079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705678,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":344064,"data_stored":190679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":172032,"data_stored":71342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":264,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":270336,"data_stored":116951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":28110542},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:40:50.286 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:40:50.622 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:50.622 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":62,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":27,"num_osds":4,"num_up_osds":3,"osd_up_since":1726461643,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":112766976,"bytes_avail":386434289664,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-16T04:39:47.675528+0000","services":{}},"progress_events":{}} 2024-09-16T04:40:50.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': '47829f7b-b43c-44e0-b4e2-77e6a22c542c', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 62, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 27, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1726461643, 'num_in_osds': 4, 'osd_in_since': 1726461589, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 190120, 'bytes_used': 112766976, 'bytes_avail': 386434289664, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'btime': '2024-09-16T04:39:47:671911+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-09-16T04:39:47.675528+0000', 'services': {}}, 'progress_events': {}} 2024-09-16T04:40:50.635 INFO:tasks.ec_lost_unfound.ceph_manager:active! 2024-09-16T04:40:50.635 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T04:40:50.636 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T04:40:50.636 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T04:40:50.774 INFO:teuthology.orchestra.run.smithi152.stdout:98784247814 2024-09-16T04:40:50.774 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:40:50.786 INFO:teuthology.orchestra.run.smithi152.stdout:111669149700 2024-09-16T04:40:50.786 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:40:50.787 INFO:teuthology.orchestra.run.smithi152.stdout:38654705680 2024-09-16T04:40:50.787 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:40:51.073 INFO:teuthology.orchestra.run.smithi152.stdout:98784247813 2024-09-16T04:40:51.086 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247814 got 98784247813 for osd.0 2024-09-16T04:40:51.118 INFO:teuthology.orchestra.run.smithi152.stdout:111669149699 2024-09-16T04:40:51.120 INFO:teuthology.orchestra.run.smithi152.stdout:38654705679 2024-09-16T04:40:51.133 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149700 got 111669149699 for osd.3 2024-09-16T04:40:51.134 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705680 got 38654705679 for osd.2 2024-09-16T04:40:52.087 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:40:52.134 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:40:52.135 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:40:52.396 INFO:teuthology.orchestra.run.smithi152.stdout:98784247813 2024-09-16T04:40:52.409 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247814 got 98784247813 for osd.0 2024-09-16T04:40:52.434 INFO:teuthology.orchestra.run.smithi152.stdout:111669149699 2024-09-16T04:40:52.435 INFO:teuthology.orchestra.run.smithi152.stdout:38654705679 2024-09-16T04:40:52.447 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149700 got 111669149699 for osd.3 2024-09-16T04:40:52.449 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705680 got 38654705679 for osd.2 2024-09-16T04:40:53.410 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:40:53.448 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:40:53.450 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:40:53.741 INFO:teuthology.orchestra.run.smithi152.stdout:98784247814 2024-09-16T04:40:53.753 INFO:teuthology.orchestra.run.smithi152.stdout:111669149700 2024-09-16T04:40:53.754 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247814 got 98784247814 for osd.0 2024-09-16T04:40:53.754 DEBUG:teuthology.parallel:result is None 2024-09-16T04:40:53.771 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149700 got 111669149700 for osd.3 2024-09-16T04:40:53.771 DEBUG:teuthology.parallel:result is None 2024-09-16T04:40:53.787 INFO:teuthology.orchestra.run.smithi152.stdout:38654705680 2024-09-16T04:40:53.800 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705680 got 38654705680 for osd.2 2024-09-16T04:40:53.800 DEBUG:teuthology.parallel:result is None 2024-09-16T04:40:53.801 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:40:54.177 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:40:54.177 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":66,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":27,"num_osds":4,"num_up_osds":3,"osd_up_since":1726461643,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":112766976,"bytes_avail":386434289664,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-16T04:39:47.675528+0000","services":{}},"progress_events":{}} 2024-09-16T04:40:54.190 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': '47829f7b-b43c-44e0-b4e2-77e6a22c542c', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 66, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 27, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1726461643, 'num_in_osds': 4, 'osd_in_since': 1726461589, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 190120, 'bytes_used': 112766976, 'bytes_avail': 386434289664, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'btime': '2024-09-16T04:39:47:671911+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-09-16T04:39:47.675528+0000', 'services': {}}, 'progress_events': {}} 2024-09-16T04:40:54.190 INFO:tasks.ec_lost_unfound:there are 27 unfound objects 2024-09-16T04:40:54.191 DEBUG:teuthology.orchestra.run.smithi152:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.admin -b 4096 -p unique_pool_0 -t 20 bench 240 write' 2024-09-16T04:40:54.242 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stderr:INFO: op_size has been rounded to 8192 2024-09-16T04:40:54.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:hints = 1 2024-09-16T04:40:54.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Maintaining 20 concurrent writes of 8192 bytes to objects of size 8192 for up to 240 seconds or 0 objects 2024-09-16T04:40:54.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Object prefix: benchmark_data_smithi152_43478 2024-09-16T04:40:54.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T04:40:54.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 0 0 0 0 0 0 - 0 2024-09-16T04:40:55.242 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 1 20 1673 1653 12.9133 12.9141 0.0223488 0.0119831 2024-09-16T04:40:56.242 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 2 20 3283 3263 12.7452 12.5781 0.0120138 0.0122175 2024-09-16T04:40:57.242 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 3 20 4854 4834 12.5876 12.2734 0.00976209 0.0123866 2024-09-16T04:40:58.242 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 4 20 6377 6357 12.4151 11.8984 0.0103855 0.0125604 2024-09-16T04:40:59.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 5 20 7852 7832 12.2366 11.5234 0.0121124 0.0127464 2024-09-16T04:41:00.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 6 20 9293 9273 12.0733 11.2578 0.0126711 0.0129239 2024-09-16T04:41:01.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 7 20 10692 10672 11.9095 10.9297 0.0120665 0.0130982 2024-09-16T04:41:02.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 8 20 12057 12037 11.7531 10.6641 0.0132647 0.0132791 2024-09-16T04:41:03.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 9 20 13369 13349 11.586 10.25 0.0157231 0.0134661 2024-09-16T04:41:04.194 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:04.243 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 10 20 14659 14639 11.4351 10.0781 0.0134838 0.0136456 2024-09-16T04:41:04.720 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:04.721 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:04.742 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2024-09-16T04:41:04.554872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":110110376,"num_objects":13446,"num_object_clones":0,"num_object_copies":53784,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":13500,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":13446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13473,"num_write_kb":107666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13473,"ondisk_log_size":13473,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":392616,"kb_used_data":158632,"kb_used_omap":11,"kb_used_meta":233908,"kb_avail":377094744,"statfs":{"total":386547056640,"available":386145017856,"internally_reserved":0,"allocated":162439168,"data_stored":161878323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11993,"internal_metadata":239522087},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,11,17,11,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":109920256,"num_objects":13418,"num_object_clones":0,"num_object_copies":53672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13418,"num_write_kb":107344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003663"},"pg_stats":[{"pgid":"1.f","version":"27'798","reported_seq":825,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:02.773982+0000","last_change":"2024-09-16T04:40:54.254864+0000","last_active":"2024-09-16T04:41:02.773982+0000","last_peered":"2024-09-16T04:41:02.773982+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:40:44.842163+0000","last_became_peered":"2024-09-16T04:40:44.842163+0000","last_unstale":"2024-09-16T04:41:02.773982+0000","last_undegraded":"2024-09-16T04:40:54.254576+0000","last_fullsized":"2024-09-16T04:40:44.833950+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":798,"log_dups_size":0,"ondisk_log_size":798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:15.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6537216,"num_objects":798,"num_object_clones":0,"num_object_copies":3192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":798,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":798,"num_write_kb":6384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":798}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'878","reported_seq":917,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:41:03.731277+0000","last_change":"2024-09-16T04:40:44.853621+0000","last_active":"2024-09-16T04:41:03.731277+0000","last_peered":"2024-09-16T04:41:03.731277+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:40:44.848800+0000","last_became_peered":"2024-09-16T04:40:44.848800+0000","last_unstale":"2024-09-16T04:41:03.731277+0000","last_undegraded":"2024-09-16T04:40:44.837670+0000","last_fullsized":"2024-09-16T04:40:44.837294+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7173388,"num_objects":876,"num_object_clones":0,"num_object_copies":3504,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":880,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":878,"num_write_kb":7020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":874},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'856","reported_seq":907,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:41:03.525408+0000","last_change":"2024-09-16T04:40:44.846856+0000","last_active":"2024-09-16T04:41:03.525408+0000","last_peered":"2024-09-16T04:41:03.525408+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:40:44.839558+0000","last_became_peered":"2024-09-16T04:40:44.839558+0000","last_unstale":"2024-09-16T04:41:03.525408+0000","last_undegraded":"2024-09-16T04:40:44.832928+0000","last_fullsized":"2024-09-16T04:40:44.832628+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":856,"log_dups_size":0,"ondisk_log_size":856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6994566,"num_objects":854,"num_object_clones":0,"num_object_copies":3416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":856,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":856,"num_write_kb":6838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":853},{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'909","reported_seq":947,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:41:03.741023+0000","last_change":"2024-09-16T04:40:44.850757+0000","last_active":"2024-09-16T04:41:03.741023+0000","last_peered":"2024-09-16T04:41:03.741023+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:40:44.848625+0000","last_became_peered":"2024-09-16T04:40:44.848625+0000","last_unstale":"2024-09-16T04:41:03.741023+0000","last_undegraded":"2024-09-16T04:40:44.837318+0000","last_fullsized":"2024-09-16T04:40:44.835886+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7427340,"num_objects":907,"num_object_clones":0,"num_object_copies":3628,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":911,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":909,"num_write_kb":7261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":905},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'859","reported_seq":885,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:41:02.828361+0000","last_change":"2024-09-16T04:40:44.851339+0000","last_active":"2024-09-16T04:41:02.828361+0000","last_peered":"2024-09-16T04:41:02.828361+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:40:44.843541+0000","last_became_peered":"2024-09-16T04:40:44.843541+0000","last_unstale":"2024-09-16T04:41:02.828361+0000","last_undegraded":"2024-09-16T04:40:44.835930+0000","last_fullsized":"2024-09-16T04:40:44.834220+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":859,"log_dups_size":0,"ondisk_log_size":859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7025932,"num_objects":858,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":862,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":859,"num_write_kb":6869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":856},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'806","reported_seq":833,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:02.803273+0000","last_change":"2024-09-16T04:40:54.254573+0000","last_active":"2024-09-16T04:41:02.803273+0000","last_peered":"2024-09-16T04:41:02.803273+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:40:44.842247+0000","last_became_peered":"2024-09-16T04:40:44.842247+0000","last_unstale":"2024-09-16T04:41:02.803273+0000","last_undegraded":"2024-09-16T04:40:54.249307+0000","last_fullsized":"2024-09-16T04:40:44.833664+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":806,"log_dups_size":0,"ondisk_log_size":806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:54.335765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6602752,"num_objects":806,"num_object_clones":0,"num_object_copies":3224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":806,"num_write_kb":6448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"27'886","reported_seq":924,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:41:03.729694+0000","last_change":"2024-09-16T04:40:44.845464+0000","last_active":"2024-09-16T04:41:03.729694+0000","last_peered":"2024-09-16T04:41:03.729694+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:40:44.842032+0000","last_became_peered":"2024-09-16T04:40:44.842032+0000","last_unstale":"2024-09-16T04:41:03.729694+0000","last_undegraded":"2024-09-16T04:40:44.835742+0000","last_fullsized":"2024-09-16T04:40:44.835328+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":886,"log_dups_size":0,"ondisk_log_size":886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7222540,"num_objects":882,"num_object_clones":0,"num_object_copies":3528,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":886,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":886,"num_write_kb":7068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":880},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'901","reported_seq":939,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:41:03.741737+0000","last_change":"2024-09-16T04:40:44.852220+0000","last_active":"2024-09-16T04:41:03.741737+0000","last_peered":"2024-09-16T04:41:03.741737+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:40:44.848808+0000","last_became_peered":"2024-09-16T04:40:44.848808+0000","last_unstale":"2024-09-16T04:41:03.741737+0000","last_undegraded":"2024-09-16T04:40:44.836618+0000","last_fullsized":"2024-09-16T04:40:44.836113+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":901,"log_dups_size":0,"ondisk_log_size":901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7353612,"num_objects":898,"num_object_clones":0,"num_object_copies":3592,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":902,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":901,"num_write_kb":7196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":896},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'848","reported_seq":875,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:41:02.812931+0000","last_change":"2024-09-16T04:40:44.847852+0000","last_active":"2024-09-16T04:41:02.812931+0000","last_peered":"2024-09-16T04:41:02.812931+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:40:44.842734+0000","last_became_peered":"2024-09-16T04:40:44.842734+0000","last_unstale":"2024-09-16T04:41:02.812931+0000","last_undegraded":"2024-09-16T04:40:44.833363+0000","last_fullsized":"2024-09-16T04:40:44.833064+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":848,"log_dups_size":0,"ondisk_log_size":848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6900248,"num_objects":843,"num_object_clones":0,"num_object_copies":3372,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":849,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":848,"num_write_kb":6761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":840},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'865","reported_seq":915,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:41:03.524323+0000","last_change":"2024-09-16T04:40:44.849992+0000","last_active":"2024-09-16T04:41:03.524323+0000","last_peered":"2024-09-16T04:41:03.524323+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:40:44.839905+0000","last_became_peered":"2024-09-16T04:40:44.839905+0000","last_unstale":"2024-09-16T04:41:03.524323+0000","last_undegraded":"2024-09-16T04:40:44.833540+0000","last_fullsized":"2024-09-16T04:40:44.833235+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":865,"log_dups_size":0,"ondisk_log_size":865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7083276,"num_objects":865,"num_object_clones":0,"num_object_copies":3460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":869,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":865,"num_write_kb":6918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":863},{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'781","reported_seq":808,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:41:02.794317+0000","last_change":"2024-09-16T04:40:44.846078+0000","last_active":"2024-09-16T04:41:02.794317+0000","last_peered":"2024-09-16T04:41:02.794317+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:40:44.841868+0000","last_became_peered":"2024-09-16T04:40:44.841868+0000","last_unstale":"2024-09-16T04:41:02.794317+0000","last_undegraded":"2024-09-16T04:40:44.835179+0000","last_fullsized":"2024-09-16T04:40:44.834838+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":781,"log_dups_size":0,"ondisk_log_size":781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6377362,"num_objects":779,"num_object_clones":0,"num_object_copies":3116,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":785,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":781,"num_write_kb":6236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":776},{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'744","reported_seq":760,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:41:02.817516+0000","last_change":"2024-09-16T04:40:44.854206+0000","last_active":"2024-09-16T04:41:02.817516+0000","last_peered":"2024-09-16T04:41:02.817516+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:40:44.843370+0000","last_became_peered":"2024-09-16T04:40:44.843370+0000","last_unstale":"2024-09-16T04:41:02.817516+0000","last_undegraded":"2024-09-16T04:40:44.835077+0000","last_fullsized":"2024-09-16T04:40:44.834737+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":744,"log_dups_size":0,"ondisk_log_size":744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6093446,"num_objects":744,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":746,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":744,"num_write_kb":5951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":743},{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'837","reported_seq":888,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:41:03.536108+0000","last_change":"2024-09-16T04:40:44.844542+0000","last_active":"2024-09-16T04:41:03.536108+0000","last_peered":"2024-09-16T04:41:03.536108+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:40:44.839353+0000","last_became_peered":"2024-09-16T04:40:44.839353+0000","last_unstale":"2024-09-16T04:41:03.536108+0000","last_undegraded":"2024-09-16T04:40:44.832429+0000","last_fullsized":"2024-09-16T04:40:44.832101+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6855302,"num_objects":837,"num_object_clones":0,"num_object_copies":3348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":839,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":837,"num_write_kb":6695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":836},{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'771","reported_seq":798,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:41:02.809970+0000","last_change":"2024-09-16T04:40:44.855685+0000","last_active":"2024-09-16T04:41:02.809970+0000","last_peered":"2024-09-16T04:41:02.809970+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:40:44.844454+0000","last_became_peered":"2024-09-16T04:40:44.844454+0000","last_unstale":"2024-09-16T04:41:02.809970+0000","last_undegraded":"2024-09-16T04:40:44.836539+0000","last_fullsized":"2024-09-16T04:40:44.836255+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":771,"log_dups_size":0,"ondisk_log_size":771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6296844,"num_objects":769,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":773,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":771,"num_write_kb":6164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":767},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'840","reported_seq":892,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:41:03.535205+0000","last_change":"2024-09-16T04:40:44.852882+0000","last_active":"2024-09-16T04:41:03.535205+0000","last_peered":"2024-09-16T04:41:03.535205+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:40:44.840598+0000","last_became_peered":"2024-09-16T04:40:44.840598+0000","last_unstale":"2024-09-16T04:41:03.535205+0000","last_undegraded":"2024-09-16T04:40:44.833574+0000","last_fullsized":"2024-09-16T04:40:44.833243+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":840,"log_dups_size":0,"ondisk_log_size":840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6870284,"num_objects":839,"num_object_clones":0,"num_object_copies":3356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":843,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":840,"num_write_kb":6717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":837},{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'894","reported_seq":933,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-16T04:41:03.727889+0000","last_change":"2024-09-16T04:40:44.855046+0000","last_active":"2024-09-16T04:41:03.727889+0000","last_peered":"2024-09-16T04:41:03.727889+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:40:44.848727+0000","last_became_peered":"2024-09-16T04:40:44.848727+0000","last_unstale":"2024-09-16T04:41:03.727889+0000","last_undegraded":"2024-09-16T04:40:44.835805+0000","last_fullsized":"2024-09-16T04:40:44.835392+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7296268,"num_objects":891,"num_object_clones":0,"num_object_copies":3564,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":895,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":894,"num_write_kb":7140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":889},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":110110376,"num_objects":13446,"num_object_clones":0,"num_object_copies":53784,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":13500,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":13446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13473,"num_write_kb":107666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166694912,"data_stored":166694912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13473,"ondisk_log_size":13473,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125992,"kb_used_data":49880,"kb_used_omap":4,"kb_used_meta":76091,"kb_avail":94245848,"statfs":{"total":96636764160,"available":96507748352,"internally_reserved":0,"allocated":51077120,"data_stored":50923735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":77917903},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,5,4,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121068,"kb_used_data":53788,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94250772,"statfs":{"total":96636764160,"available":96512790528,"internally_reserved":0,"allocated":55078912,"data_stored":54925527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,6,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":172032,"data_stored":71342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118364,"kb_used_data":54796,"kb_used_omap":4,"kb_used_meta":63547,"kb_avail":94253476,"statfs":{"total":96636764160,"available":96515559424,"internally_reserved":0,"allocated":56111104,"data_stored":55957719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":65072846},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,7,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57458688,"data_stored":57458688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":56586240,"data_stored":56586240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":52609024,"data_stored":52609024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:04.743 INFO:tasks.ec_lost_unfound:no unfound in 1.f 2024-09-16T04:41:04.743 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.e state active+recovery_unfound+undersized+degraded 2024-09-16T04:41:04.743 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.e list_unfound '{}' 2024-09-16T04:41:04.971 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:41:04.971 INFO:teuthology.orchestra.run.smithi152.stdout: "offset": { 2024-09-16T04:41:04.971 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "", 2024-09-16T04:41:04.971 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:04.971 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": 0, 2024-09-16T04:41:04.971 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 0, 2024-09-16T04:41:04.972 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:04.972 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": -9223372036854775808, 2024-09-16T04:41:04.972 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:04.972 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:04.972 INFO:teuthology.orchestra.run.smithi152.stdout: "num_missing": 2, 2024-09-16T04:41:04.972 INFO:teuthology.orchestra.run.smithi152.stdout: "num_unfound": 2, 2024-09-16T04:41:04.972 INFO:teuthology.orchestra.run.smithi152.stdout: "objects": [ 2024-09-16T04:41:04.972 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:04.972 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:04.972 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existing_4", 2024-09-16T04:41:04.972 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:04.972 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:04.973 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 1224592654, 2024-09-16T04:41:04.973 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:04.973 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:04.973 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:04.973 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:04.973 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'3", 2024-09-16T04:41:04.973 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "16'1", 2024-09-16T04:41:04.973 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:04.973 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-16T04:41:04.973 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:04.973 INFO:teuthology.orchestra.run.smithi152.stdout: "2(3)" 2024-09-16T04:41:04.973 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:04.974 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:04.974 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:04.974 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:04.974 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existing_6", 2024-09-16T04:41:04.974 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:04.974 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:04.974 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 3101443614, 2024-09-16T04:41:04.974 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:04.974 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:04.974 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:04.974 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:04.974 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'4", 2024-09-16T04:41:04.975 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "16'2", 2024-09-16T04:41:04.975 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:04.975 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-16T04:41:04.975 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:04.975 INFO:teuthology.orchestra.run.smithi152.stdout: "2(3)" 2024-09-16T04:41:04.975 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:04.975 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:04.975 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:04.975 INFO:teuthology.orchestra.run.smithi152.stdout: "state": "NotRecovering", 2024-09-16T04:41:04.975 INFO:teuthology.orchestra.run.smithi152.stdout: "available_might_have_unfound": true, 2024-09-16T04:41:04.975 INFO:teuthology.orchestra.run.smithi152.stdout: "might_have_unfound": [ 2024-09-16T04:41:04.975 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:04.976 INFO:teuthology.orchestra.run.smithi152.stdout: "osd": "1(0)", 2024-09-16T04:41:04.977 INFO:teuthology.orchestra.run.smithi152.stdout: "status": "osd is down" 2024-09-16T04:41:04.977 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:04.977 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:04.977 INFO:teuthology.orchestra.run.smithi152.stdout: "more": false 2024-09-16T04:41:04.977 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:41:04.995 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_4', 'key': '', 'snapid': -2, 'hash': 1224592654, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(3)']}, {'oid': {'oid': 'existing_6', 'key': '', 'snapid': -2, 'hash': 3101443614, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-16T04:41:04.996 INFO:tasks.ec_lost_unfound:reverting unfound in 1.e 2024-09-16T04:41:04.996 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.e mark_unfound_lost delete 2024-09-16T04:41:05.244 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 11 20 15753 15733 11.1725 8.54688 0.021813 0.0139691 2024-09-16T04:41:05.263 INFO:teuthology.orchestra.run.smithi152.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T04:41:05.284 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.d state active+recovery_unfound+undersized+degraded 2024-09-16T04:41:05.285 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.d list_unfound '{}' 2024-09-16T04:41:05.515 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:41:05.515 INFO:teuthology.orchestra.run.smithi152.stdout: "offset": { 2024-09-16T04:41:05.515 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "", 2024-09-16T04:41:05.516 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:05.516 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": 0, 2024-09-16T04:41:05.516 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 0, 2024-09-16T04:41:05.516 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:05.516 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": -9223372036854775808, 2024-09-16T04:41:05.516 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:05.516 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:05.516 INFO:teuthology.orchestra.run.smithi152.stdout: "num_missing": 1, 2024-09-16T04:41:05.516 INFO:teuthology.orchestra.run.smithi152.stdout: "num_unfound": 1, 2024-09-16T04:41:05.516 INFO:teuthology.orchestra.run.smithi152.stdout: "objects": [ 2024-09-16T04:41:05.516 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:05.516 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:05.516 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existed_7", 2024-09-16T04:41:05.517 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:05.517 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:05.517 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 2358579165, 2024-09-16T04:41:05.517 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:05.517 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:05.517 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:05.517 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:05.517 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'3", 2024-09-16T04:41:05.517 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:05.517 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:05.517 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:05.517 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:05.517 INFO:teuthology.orchestra.run.smithi152.stdout: "2(0)" 2024-09-16T04:41:05.518 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:05.518 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:05.518 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:05.518 INFO:teuthology.orchestra.run.smithi152.stdout: "state": "NotRecovering", 2024-09-16T04:41:05.518 INFO:teuthology.orchestra.run.smithi152.stdout: "available_might_have_unfound": true, 2024-09-16T04:41:05.518 INFO:teuthology.orchestra.run.smithi152.stdout: "might_have_unfound": [ 2024-09-16T04:41:05.518 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:05.518 INFO:teuthology.orchestra.run.smithi152.stdout: "osd": "1(1)", 2024-09-16T04:41:05.518 INFO:teuthology.orchestra.run.smithi152.stdout: "status": "osd is down" 2024-09-16T04:41:05.518 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:05.518 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:05.518 INFO:teuthology.orchestra.run.smithi152.stdout: "more": false 2024-09-16T04:41:05.518 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:41:05.545 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'existed_7', 'key': '', 'snapid': -2, 'hash': 2358579165, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-09-16T04:41:05.545 INFO:tasks.ec_lost_unfound:reverting unfound in 1.d 2024-09-16T04:41:05.545 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.d mark_unfound_lost delete 2024-09-16T04:41:05.789 INFO:teuthology.orchestra.run.smithi152.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-16T04:41:05.816 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.0 state active+recovery_unfound+undersized+degraded 2024-09-16T04:41:05.817 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.0 list_unfound '{}' 2024-09-16T04:41:06.050 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:41:06.050 INFO:teuthology.orchestra.run.smithi152.stdout: "offset": { 2024-09-16T04:41:06.050 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "", 2024-09-16T04:41:06.050 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:06.050 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": 0, 2024-09-16T04:41:06.050 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 0, 2024-09-16T04:41:06.050 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:06.050 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": -9223372036854775808, 2024-09-16T04:41:06.051 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:06.051 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:06.051 INFO:teuthology.orchestra.run.smithi152.stdout: "num_missing": 2, 2024-09-16T04:41:06.051 INFO:teuthology.orchestra.run.smithi152.stdout: "num_unfound": 2, 2024-09-16T04:41:06.051 INFO:teuthology.orchestra.run.smithi152.stdout: "objects": [ 2024-09-16T04:41:06.051 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:06.051 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:06.051 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "new_5", 2024-09-16T04:41:06.051 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:06.051 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:06.051 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 4248945696, 2024-09-16T04:41:06.051 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:06.051 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:06.052 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:06.052 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:06.052 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'3", 2024-09-16T04:41:06.052 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:06.052 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:06.052 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:06.052 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:06.052 INFO:teuthology.orchestra.run.smithi152.stdout: "2(2)" 2024-09-16T04:41:06.052 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:06.052 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:06.052 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:06.052 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:06.053 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existed_5", 2024-09-16T04:41:06.053 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:06.053 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:06.053 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 4212789968, 2024-09-16T04:41:06.053 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:06.053 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:06.053 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:06.053 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:06.053 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'4", 2024-09-16T04:41:06.053 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:06.053 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:06.053 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:06.053 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:06.053 INFO:teuthology.orchestra.run.smithi152.stdout: "2(2)" 2024-09-16T04:41:06.054 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:06.054 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:06.054 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:06.054 INFO:teuthology.orchestra.run.smithi152.stdout: "state": "NotRecovering", 2024-09-16T04:41:06.054 INFO:teuthology.orchestra.run.smithi152.stdout: "available_might_have_unfound": true, 2024-09-16T04:41:06.054 INFO:teuthology.orchestra.run.smithi152.stdout: "might_have_unfound": [ 2024-09-16T04:41:06.054 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:06.054 INFO:teuthology.orchestra.run.smithi152.stdout: "osd": "1(0)", 2024-09-16T04:41:06.054 INFO:teuthology.orchestra.run.smithi152.stdout: "status": "osd is down" 2024-09-16T04:41:06.054 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:06.054 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:06.054 INFO:teuthology.orchestra.run.smithi152.stdout: "more": false 2024-09-16T04:41:06.054 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:41:06.075 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_5', 'key': '', 'snapid': -2, 'hash': 4248945696, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_5', 'key': '', 'snapid': -2, 'hash': 4212789968, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-16T04:41:06.075 INFO:tasks.ec_lost_unfound:reverting unfound in 1.0 2024-09-16T04:41:06.076 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.0 mark_unfound_lost delete 2024-09-16T04:41:06.244 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 12 20 16845 16825 10.9524 8.53125 0.02357 0.0142545 2024-09-16T04:41:06.328 INFO:teuthology.orchestra.run.smithi152.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T04:41:06.346 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.1 state active+recovery_unfound+undersized+degraded 2024-09-16T04:41:06.347 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.1 list_unfound '{}' 2024-09-16T04:41:06.578 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:41:06.578 INFO:teuthology.orchestra.run.smithi152.stdout: "offset": { 2024-09-16T04:41:06.578 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "", 2024-09-16T04:41:06.578 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:06.578 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": 0, 2024-09-16T04:41:06.578 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 0, 2024-09-16T04:41:06.578 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:06.578 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": -9223372036854775808, 2024-09-16T04:41:06.579 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:06.579 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:06.579 INFO:teuthology.orchestra.run.smithi152.stdout: "num_missing": 2, 2024-09-16T04:41:06.579 INFO:teuthology.orchestra.run.smithi152.stdout: "num_unfound": 2, 2024-09-16T04:41:06.579 INFO:teuthology.orchestra.run.smithi152.stdout: "objects": [ 2024-09-16T04:41:06.579 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:06.579 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:06.579 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existing_3", 2024-09-16T04:41:06.579 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:06.579 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:06.579 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 459768081, 2024-09-16T04:41:06.579 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:06.579 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:06.580 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:06.580 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:06.580 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'3", 2024-09-16T04:41:06.580 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "16'1", 2024-09-16T04:41:06.580 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:06.580 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-16T04:41:06.580 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:06.580 INFO:teuthology.orchestra.run.smithi152.stdout: "2(3)" 2024-09-16T04:41:06.580 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:06.580 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:06.580 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:06.581 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:06.581 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "new_3", 2024-09-16T04:41:06.581 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:06.581 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:06.581 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 480221105, 2024-09-16T04:41:06.581 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:06.581 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:06.581 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:06.581 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:06.581 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'2", 2024-09-16T04:41:06.581 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:06.581 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:06.582 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:06.582 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:06.582 INFO:teuthology.orchestra.run.smithi152.stdout: "2(3)" 2024-09-16T04:41:06.582 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:06.582 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:06.582 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:06.582 INFO:teuthology.orchestra.run.smithi152.stdout: "state": "NotRecovering", 2024-09-16T04:41:06.582 INFO:teuthology.orchestra.run.smithi152.stdout: "available_might_have_unfound": true, 2024-09-16T04:41:06.582 INFO:teuthology.orchestra.run.smithi152.stdout: "might_have_unfound": [ 2024-09-16T04:41:06.583 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:06.583 INFO:teuthology.orchestra.run.smithi152.stdout: "osd": "1(1)", 2024-09-16T04:41:06.583 INFO:teuthology.orchestra.run.smithi152.stdout: "status": "osd is down" 2024-09-16T04:41:06.583 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:06.583 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:06.583 INFO:teuthology.orchestra.run.smithi152.stdout: "more": false 2024-09-16T04:41:06.583 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:41:06.605 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_3', 'key': '', 'snapid': -2, 'hash': 459768081, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(3)']}, {'oid': {'oid': 'new_3', 'key': '', 'snapid': -2, 'hash': 480221105, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-09-16T04:41:06.605 INFO:tasks.ec_lost_unfound:reverting unfound in 1.1 2024-09-16T04:41:06.605 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.1 mark_unfound_lost delete 2024-09-16T04:41:06.859 INFO:teuthology.orchestra.run.smithi152.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T04:41:06.877 INFO:tasks.ec_lost_unfound:no unfound in 1.c 2024-09-16T04:41:06.877 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.2 state active+recovery_unfound+undersized+degraded 2024-09-16T04:41:06.878 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.2 list_unfound '{}' 2024-09-16T04:41:07.104 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:41:07.104 INFO:teuthology.orchestra.run.smithi152.stdout: "offset": { 2024-09-16T04:41:07.104 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "", 2024-09-16T04:41:07.104 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:07.104 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": 0, 2024-09-16T04:41:07.104 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 0, 2024-09-16T04:41:07.104 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:07.104 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": -9223372036854775808, 2024-09-16T04:41:07.104 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:07.105 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:07.105 INFO:teuthology.orchestra.run.smithi152.stdout: "num_missing": 2, 2024-09-16T04:41:07.105 INFO:teuthology.orchestra.run.smithi152.stdout: "num_unfound": 2, 2024-09-16T04:41:07.105 INFO:teuthology.orchestra.run.smithi152.stdout: "objects": [ 2024-09-16T04:41:07.105 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:07.105 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:07.105 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existed_6", 2024-09-16T04:41:07.105 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:07.105 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:07.105 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 1708673474, 2024-09-16T04:41:07.105 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:07.105 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:07.105 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:07.106 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:07.106 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'5", 2024-09-16T04:41:07.106 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:07.106 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:07.106 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:07.106 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:07.106 INFO:teuthology.orchestra.run.smithi152.stdout: "2(3)" 2024-09-16T04:41:07.106 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:07.106 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:07.106 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:07.106 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:07.106 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existed_9", 2024-09-16T04:41:07.106 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:07.107 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:07.107 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 831402034, 2024-09-16T04:41:07.107 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:07.107 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:07.107 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:07.107 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:07.107 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'6", 2024-09-16T04:41:07.107 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:07.107 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:07.107 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:07.107 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:07.107 INFO:teuthology.orchestra.run.smithi152.stdout: "2(3)" 2024-09-16T04:41:07.107 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:07.107 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:07.108 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:07.108 INFO:teuthology.orchestra.run.smithi152.stdout: "state": "NotRecovering", 2024-09-16T04:41:07.108 INFO:teuthology.orchestra.run.smithi152.stdout: "available_might_have_unfound": true, 2024-09-16T04:41:07.108 INFO:teuthology.orchestra.run.smithi152.stdout: "might_have_unfound": [ 2024-09-16T04:41:07.108 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:07.108 INFO:teuthology.orchestra.run.smithi152.stdout: "osd": "1(2)", 2024-09-16T04:41:07.108 INFO:teuthology.orchestra.run.smithi152.stdout: "status": "osd is down" 2024-09-16T04:41:07.108 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:07.108 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:07.108 INFO:teuthology.orchestra.run.smithi152.stdout: "more": false 2024-09-16T04:41:07.108 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:41:07.126 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_6', 'key': '', 'snapid': -2, 'hash': 1708673474, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'existed_9', 'key': '', 'snapid': -2, 'hash': 831402034, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'6", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-09-16T04:41:07.127 INFO:tasks.ec_lost_unfound:reverting unfound in 1.2 2024-09-16T04:41:07.127 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.2 mark_unfound_lost delete 2024-09-16T04:41:07.244 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 13 20 17901 17881 10.7444 8.25 0.0250854 0.0145296 2024-09-16T04:41:07.373 INFO:teuthology.orchestra.run.smithi152.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T04:41:07.391 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.3 state active+recovery_unfound+undersized+degraded 2024-09-16T04:41:07.392 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.3 list_unfound '{}' 2024-09-16T04:41:07.626 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:41:07.626 INFO:teuthology.orchestra.run.smithi152.stdout: "offset": { 2024-09-16T04:41:07.626 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "", 2024-09-16T04:41:07.627 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:07.627 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": 0, 2024-09-16T04:41:07.627 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 0, 2024-09-16T04:41:07.627 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:07.627 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": -9223372036854775808, 2024-09-16T04:41:07.627 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:07.627 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:07.627 INFO:teuthology.orchestra.run.smithi152.stdout: "num_missing": 2, 2024-09-16T04:41:07.627 INFO:teuthology.orchestra.run.smithi152.stdout: "num_unfound": 2, 2024-09-16T04:41:07.627 INFO:teuthology.orchestra.run.smithi152.stdout: "objects": [ 2024-09-16T04:41:07.627 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:07.627 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:07.627 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existing_8", 2024-09-16T04:41:07.627 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:07.628 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:07.628 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 1901787523, 2024-09-16T04:41:07.628 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:07.628 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:07.628 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:07.628 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:07.628 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'5", 2024-09-16T04:41:07.628 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "16'3", 2024-09-16T04:41:07.628 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:07.628 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-16T04:41:07.628 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:07.628 INFO:teuthology.orchestra.run.smithi152.stdout: "2(2)" 2024-09-16T04:41:07.628 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:07.629 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:07.629 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:07.629 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:07.629 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existed_1", 2024-09-16T04:41:07.629 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:07.629 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:07.629 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 3122379363, 2024-09-16T04:41:07.629 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:07.629 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:07.629 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:07.629 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:07.629 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'4", 2024-09-16T04:41:07.630 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:07.630 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:07.630 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:07.630 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:07.630 INFO:teuthology.orchestra.run.smithi152.stdout: "2(2)" 2024-09-16T04:41:07.630 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:07.630 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:07.631 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:07.631 INFO:teuthology.orchestra.run.smithi152.stdout: "state": "NotRecovering", 2024-09-16T04:41:07.631 INFO:teuthology.orchestra.run.smithi152.stdout: "available_might_have_unfound": true, 2024-09-16T04:41:07.631 INFO:teuthology.orchestra.run.smithi152.stdout: "might_have_unfound": [ 2024-09-16T04:41:07.631 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:07.631 INFO:teuthology.orchestra.run.smithi152.stdout: "osd": "1(0)", 2024-09-16T04:41:07.631 INFO:teuthology.orchestra.run.smithi152.stdout: "status": "osd is down" 2024-09-16T04:41:07.631 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:07.631 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:07.631 INFO:teuthology.orchestra.run.smithi152.stdout: "more": false 2024-09-16T04:41:07.631 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:41:07.651 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_8', 'key': '', 'snapid': -2, 'hash': 1901787523, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "16'3", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_1', 'key': '', 'snapid': -2, 'hash': 3122379363, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-16T04:41:07.652 INFO:tasks.ec_lost_unfound:reverting unfound in 1.3 2024-09-16T04:41:07.652 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.3 mark_unfound_lost delete 2024-09-16T04:41:07.909 INFO:teuthology.orchestra.run.smithi152.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T04:41:07.931 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.4 state active+recovery_unfound+undersized+degraded 2024-09-16T04:41:07.931 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.4 list_unfound '{}' 2024-09-16T04:41:08.177 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:41:08.177 INFO:teuthology.orchestra.run.smithi152.stdout: "offset": { 2024-09-16T04:41:08.177 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "", 2024-09-16T04:41:08.177 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:08.177 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": 0, 2024-09-16T04:41:08.177 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 0, 2024-09-16T04:41:08.177 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:08.177 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": -9223372036854775808, 2024-09-16T04:41:08.177 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:08.177 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:08.178 INFO:teuthology.orchestra.run.smithi152.stdout: "num_missing": 3, 2024-09-16T04:41:08.178 INFO:teuthology.orchestra.run.smithi152.stdout: "num_unfound": 3, 2024-09-16T04:41:08.178 INFO:teuthology.orchestra.run.smithi152.stdout: "objects": [ 2024-09-16T04:41:08.178 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:08.178 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:08.178 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existed_4", 2024-09-16T04:41:08.178 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:08.178 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:08.178 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 873891716, 2024-09-16T04:41:08.178 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:08.178 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:08.178 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:08.178 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:08.179 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'8", 2024-09-16T04:41:08.179 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:08.179 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:08.179 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:08.179 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:08.179 INFO:teuthology.orchestra.run.smithi152.stdout: "2(2)" 2024-09-16T04:41:08.179 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:08.179 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:08.179 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:08.179 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:08.179 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existing_7", 2024-09-16T04:41:08.179 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:08.179 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:08.179 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 3996155460, 2024-09-16T04:41:08.180 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:08.180 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:08.180 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:08.180 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:08.180 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'9", 2024-09-16T04:41:08.180 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "16'6", 2024-09-16T04:41:08.180 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:08.180 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-16T04:41:08.180 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:08.180 INFO:teuthology.orchestra.run.smithi152.stdout: "2(2)" 2024-09-16T04:41:08.180 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:08.180 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:08.180 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:08.181 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:08.181 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existed_2", 2024-09-16T04:41:08.181 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:08.181 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:08.181 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 1767667508, 2024-09-16T04:41:08.181 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:08.181 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:08.181 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:08.181 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:08.181 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'7", 2024-09-16T04:41:08.181 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:08.181 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:08.181 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:08.181 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:08.182 INFO:teuthology.orchestra.run.smithi152.stdout: "2(2)" 2024-09-16T04:41:08.182 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:08.182 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:08.182 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:08.182 INFO:teuthology.orchestra.run.smithi152.stdout: "state": "NotRecovering", 2024-09-16T04:41:08.182 INFO:teuthology.orchestra.run.smithi152.stdout: "available_might_have_unfound": true, 2024-09-16T04:41:08.182 INFO:teuthology.orchestra.run.smithi152.stdout: "might_have_unfound": [ 2024-09-16T04:41:08.182 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:08.182 INFO:teuthology.orchestra.run.smithi152.stdout: "osd": "1(3)", 2024-09-16T04:41:08.182 INFO:teuthology.orchestra.run.smithi152.stdout: "status": "osd is down" 2024-09-16T04:41:08.182 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:08.182 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:08.182 INFO:teuthology.orchestra.run.smithi152.stdout: "more": false 2024-09-16T04:41:08.182 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:41:08.201 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_4', 'key': '', 'snapid': -2, 'hash': 873891716, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'8", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_7', 'key': '', 'snapid': -2, 'hash': 3996155460, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'9", 'have': "16'6", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_2', 'key': '', 'snapid': -2, 'hash': 1767667508, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'7", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-09-16T04:41:08.201 INFO:tasks.ec_lost_unfound:reverting unfound in 1.4 2024-09-16T04:41:08.201 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.4 mark_unfound_lost delete 2024-09-16T04:41:08.244 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 14 20 18932 18912 10.5523 8.05469 0.0274668 0.014793 2024-09-16T04:41:08.458 INFO:teuthology.orchestra.run.smithi152.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-16T04:41:08.482 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.5 state active+recovery_unfound+undersized+degraded 2024-09-16T04:41:08.482 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.5 list_unfound '{}' 2024-09-16T04:41:08.755 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:41:08.755 INFO:teuthology.orchestra.run.smithi152.stdout: "offset": { 2024-09-16T04:41:08.755 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "", 2024-09-16T04:41:08.755 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:08.755 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": 0, 2024-09-16T04:41:08.755 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 0, 2024-09-16T04:41:08.755 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:08.755 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": -9223372036854775808, 2024-09-16T04:41:08.755 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:08.756 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:08.756 INFO:teuthology.orchestra.run.smithi152.stdout: "num_missing": 2, 2024-09-16T04:41:08.756 INFO:teuthology.orchestra.run.smithi152.stdout: "num_unfound": 2, 2024-09-16T04:41:08.756 INFO:teuthology.orchestra.run.smithi152.stdout: "objects": [ 2024-09-16T04:41:08.756 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:08.756 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:08.756 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "new_1", 2024-09-16T04:41:08.756 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:08.756 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:08.756 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 26281365, 2024-09-16T04:41:08.756 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:08.756 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:08.757 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:08.757 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:08.757 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'1", 2024-09-16T04:41:08.757 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:08.757 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:08.757 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:08.757 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:08.757 INFO:teuthology.orchestra.run.smithi152.stdout: "2(0)" 2024-09-16T04:41:08.757 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:08.757 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:08.757 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:08.757 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:08.757 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "new_2", 2024-09-16T04:41:08.758 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:08.758 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:08.758 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 3196340981, 2024-09-16T04:41:08.758 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:08.758 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:08.758 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:08.758 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:08.758 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'2", 2024-09-16T04:41:08.758 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:08.758 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:08.758 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:08.758 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:08.758 INFO:teuthology.orchestra.run.smithi152.stdout: "2(0)" 2024-09-16T04:41:08.758 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:08.759 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:08.759 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:08.759 INFO:teuthology.orchestra.run.smithi152.stdout: "state": "NotRecovering", 2024-09-16T04:41:08.759 INFO:teuthology.orchestra.run.smithi152.stdout: "available_might_have_unfound": true, 2024-09-16T04:41:08.759 INFO:teuthology.orchestra.run.smithi152.stdout: "might_have_unfound": [ 2024-09-16T04:41:08.759 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:08.759 INFO:teuthology.orchestra.run.smithi152.stdout: "osd": "1(3)", 2024-09-16T04:41:08.759 INFO:teuthology.orchestra.run.smithi152.stdout: "status": "osd is down" 2024-09-16T04:41:08.759 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:08.759 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:08.759 INFO:teuthology.orchestra.run.smithi152.stdout: "more": false 2024-09-16T04:41:08.759 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:41:08.777 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_1', 'key': '', 'snapid': -2, 'hash': 26281365, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}, {'oid': {'oid': 'new_2', 'key': '', 'snapid': -2, 'hash': 3196340981, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-09-16T04:41:08.777 INFO:tasks.ec_lost_unfound:reverting unfound in 1.5 2024-09-16T04:41:08.778 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.5 mark_unfound_lost delete 2024-09-16T04:41:09.014 INFO:tasks.ceph.mon.a.smithi152.stderr:2024-09-16T04:41:09.012+0000 7f6a84663640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 13 pgs recovery_unfound (PG_DAMAGED) 2024-09-16T04:41:09.034 INFO:teuthology.orchestra.run.smithi152.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T04:41:09.057 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.6 state active+recovery_unfound+undersized+degraded 2024-09-16T04:41:09.058 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.6 list_unfound '{}' 2024-09-16T04:41:09.244 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 15 20 19947 19927 10.3774 7.92969 0.0184703 0.0150412 2024-09-16T04:41:09.293 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:41:09.294 INFO:teuthology.orchestra.run.smithi152.stdout: "offset": { 2024-09-16T04:41:09.294 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "", 2024-09-16T04:41:09.294 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:09.294 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": 0, 2024-09-16T04:41:09.294 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 0, 2024-09-16T04:41:09.294 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:09.294 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": -9223372036854775808, 2024-09-16T04:41:09.294 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:09.294 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:09.294 INFO:teuthology.orchestra.run.smithi152.stdout: "num_missing": 3, 2024-09-16T04:41:09.294 INFO:teuthology.orchestra.run.smithi152.stdout: "num_unfound": 3, 2024-09-16T04:41:09.295 INFO:teuthology.orchestra.run.smithi152.stdout: "objects": [ 2024-09-16T04:41:09.295 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:09.295 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:09.295 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existed_8", 2024-09-16T04:41:09.295 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:09.295 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:09.295 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 1861424966, 2024-09-16T04:41:09.295 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:09.295 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:09.295 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:09.295 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:09.295 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'5", 2024-09-16T04:41:09.295 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:09.296 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:09.296 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:09.296 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:09.296 INFO:teuthology.orchestra.run.smithi152.stdout: "2(3)" 2024-09-16T04:41:09.296 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:09.296 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:09.296 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:09.296 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:09.296 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "new_6", 2024-09-16T04:41:09.296 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:09.296 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:09.296 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 3910131350, 2024-09-16T04:41:09.296 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:09.297 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:09.297 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:09.297 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:09.297 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'4", 2024-09-16T04:41:09.297 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:09.297 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:09.297 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:09.297 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:09.297 INFO:teuthology.orchestra.run.smithi152.stdout: "2(3)" 2024-09-16T04:41:09.297 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:09.297 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:09.297 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:09.297 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:09.297 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "new_4", 2024-09-16T04:41:09.298 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:09.298 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:09.298 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 2787839990, 2024-09-16T04:41:09.298 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:09.298 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:09.298 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:09.298 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:09.298 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'3", 2024-09-16T04:41:09.298 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:09.298 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:09.298 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:09.298 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:09.298 INFO:teuthology.orchestra.run.smithi152.stdout: "2(3)" 2024-09-16T04:41:09.298 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:09.299 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:09.299 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:09.299 INFO:teuthology.orchestra.run.smithi152.stdout: "state": "NotRecovering", 2024-09-16T04:41:09.299 INFO:teuthology.orchestra.run.smithi152.stdout: "available_might_have_unfound": true, 2024-09-16T04:41:09.299 INFO:teuthology.orchestra.run.smithi152.stdout: "might_have_unfound": [ 2024-09-16T04:41:09.299 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:09.299 INFO:teuthology.orchestra.run.smithi152.stdout: "osd": "1(2)", 2024-09-16T04:41:09.299 INFO:teuthology.orchestra.run.smithi152.stdout: "status": "osd is down" 2024-09-16T04:41:09.299 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:09.299 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:09.299 INFO:teuthology.orchestra.run.smithi152.stdout: "more": false 2024-09-16T04:41:09.299 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:41:09.314 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_8', 'key': '', 'snapid': -2, 'hash': 1861424966, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'new_6', 'key': '', 'snapid': -2, 'hash': 3910131350, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'new_4', 'key': '', 'snapid': -2, 'hash': 2787839990, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-09-16T04:41:09.315 INFO:tasks.ec_lost_unfound:reverting unfound in 1.6 2024-09-16T04:41:09.315 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.6 mark_unfound_lost delete 2024-09-16T04:41:09.566 INFO:teuthology.orchestra.run.smithi152.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-16T04:41:09.594 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.7 state active+recovery_unfound+undersized+degraded 2024-09-16T04:41:09.595 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.7 list_unfound '{}' 2024-09-16T04:41:09.828 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:41:09.828 INFO:teuthology.orchestra.run.smithi152.stdout: "offset": { 2024-09-16T04:41:09.829 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "", 2024-09-16T04:41:09.829 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:09.829 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": 0, 2024-09-16T04:41:09.829 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 0, 2024-09-16T04:41:09.829 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:09.829 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": -9223372036854775808, 2024-09-16T04:41:09.829 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:09.829 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:09.829 INFO:teuthology.orchestra.run.smithi152.stdout: "num_missing": 1, 2024-09-16T04:41:09.829 INFO:teuthology.orchestra.run.smithi152.stdout: "num_unfound": 1, 2024-09-16T04:41:09.829 INFO:teuthology.orchestra.run.smithi152.stdout: "objects": [ 2024-09-16T04:41:09.829 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:09.829 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:09.829 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "new_7", 2024-09-16T04:41:09.830 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:09.830 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:09.830 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 3761017527, 2024-09-16T04:41:09.830 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:09.830 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:09.830 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:09.830 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:09.830 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'1", 2024-09-16T04:41:09.830 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:09.830 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:09.830 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:09.830 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:09.830 INFO:teuthology.orchestra.run.smithi152.stdout: "2(2)" 2024-09-16T04:41:09.831 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:09.831 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:09.831 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:09.831 INFO:teuthology.orchestra.run.smithi152.stdout: "state": "NotRecovering", 2024-09-16T04:41:09.831 INFO:teuthology.orchestra.run.smithi152.stdout: "available_might_have_unfound": true, 2024-09-16T04:41:09.831 INFO:teuthology.orchestra.run.smithi152.stdout: "might_have_unfound": [ 2024-09-16T04:41:09.831 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:09.831 INFO:teuthology.orchestra.run.smithi152.stdout: "osd": "1(0)", 2024-09-16T04:41:09.831 INFO:teuthology.orchestra.run.smithi152.stdout: "status": "osd is down" 2024-09-16T04:41:09.831 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:09.831 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:09.831 INFO:teuthology.orchestra.run.smithi152.stdout: "more": false 2024-09-16T04:41:09.831 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:41:09.855 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_7', 'key': '', 'snapid': -2, 'hash': 3761017527, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-16T04:41:09.855 INFO:tasks.ec_lost_unfound:reverting unfound in 1.7 2024-09-16T04:41:09.856 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.7 mark_unfound_lost delete 2024-09-16T04:41:10.109 INFO:teuthology.orchestra.run.smithi152.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-16T04:41:10.130 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.8 state active+recovery_unfound+undersized+degraded 2024-09-16T04:41:10.130 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.8 list_unfound '{}' 2024-09-16T04:41:10.244 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 16 20 20962 20942 10.2244 7.92969 0.0136162 0.0152677 2024-09-16T04:41:10.361 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:41:10.361 INFO:teuthology.orchestra.run.smithi152.stdout: "offset": { 2024-09-16T04:41:10.361 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "", 2024-09-16T04:41:10.361 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:10.362 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": 0, 2024-09-16T04:41:10.362 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 0, 2024-09-16T04:41:10.362 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:10.362 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": -9223372036854775808, 2024-09-16T04:41:10.362 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:10.362 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:10.362 INFO:teuthology.orchestra.run.smithi152.stdout: "num_missing": 1, 2024-09-16T04:41:10.362 INFO:teuthology.orchestra.run.smithi152.stdout: "num_unfound": 1, 2024-09-16T04:41:10.362 INFO:teuthology.orchestra.run.smithi152.stdout: "objects": [ 2024-09-16T04:41:10.362 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:10.362 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:10.362 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "new_8", 2024-09-16T04:41:10.362 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:10.362 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:10.363 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 2730327960, 2024-09-16T04:41:10.363 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:10.363 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:10.363 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:10.363 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:10.363 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'1", 2024-09-16T04:41:10.363 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:10.363 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:10.363 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:10.363 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:10.363 INFO:teuthology.orchestra.run.smithi152.stdout: "2(1)" 2024-09-16T04:41:10.363 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:10.363 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:10.363 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:10.364 INFO:teuthology.orchestra.run.smithi152.stdout: "state": "NotRecovering", 2024-09-16T04:41:10.364 INFO:teuthology.orchestra.run.smithi152.stdout: "available_might_have_unfound": true, 2024-09-16T04:41:10.364 INFO:teuthology.orchestra.run.smithi152.stdout: "might_have_unfound": [ 2024-09-16T04:41:10.364 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:10.364 INFO:teuthology.orchestra.run.smithi152.stdout: "osd": "1(0)", 2024-09-16T04:41:10.364 INFO:teuthology.orchestra.run.smithi152.stdout: "status": "osd is down" 2024-09-16T04:41:10.364 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:10.364 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:10.364 INFO:teuthology.orchestra.run.smithi152.stdout: "more": false 2024-09-16T04:41:10.364 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:41:10.385 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_8', 'key': '', 'snapid': -2, 'hash': 2730327960, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-16T04:41:10.385 INFO:tasks.ec_lost_unfound:reverting unfound in 1.8 2024-09-16T04:41:10.385 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.8 mark_unfound_lost delete 2024-09-16T04:41:10.594 INFO:teuthology.orchestra.run.smithi152.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-16T04:41:10.617 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.9 state active+recovery_unfound+undersized+degraded 2024-09-16T04:41:10.618 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.9 list_unfound '{}' 2024-09-16T04:41:10.838 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:41:10.838 INFO:teuthology.orchestra.run.smithi152.stdout: "offset": { 2024-09-16T04:41:10.838 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "", 2024-09-16T04:41:10.838 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:10.838 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": 0, 2024-09-16T04:41:10.839 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 0, 2024-09-16T04:41:10.839 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:10.839 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": -9223372036854775808, 2024-09-16T04:41:10.839 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:10.839 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:10.839 INFO:teuthology.orchestra.run.smithi152.stdout: "num_missing": 2, 2024-09-16T04:41:10.839 INFO:teuthology.orchestra.run.smithi152.stdout: "num_unfound": 2, 2024-09-16T04:41:10.839 INFO:teuthology.orchestra.run.smithi152.stdout: "objects": [ 2024-09-16T04:41:10.839 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:10.839 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:10.839 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existing_5", 2024-09-16T04:41:10.839 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:10.839 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:10.839 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 2998549577, 2024-09-16T04:41:10.840 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:10.840 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:10.840 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:10.840 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:10.840 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'4", 2024-09-16T04:41:10.840 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "16'2", 2024-09-16T04:41:10.840 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:10.840 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-16T04:41:10.840 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:10.840 INFO:teuthology.orchestra.run.smithi152.stdout: "2(2)" 2024-09-16T04:41:10.840 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:10.840 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:10.840 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:10.840 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:10.841 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existing_2", 2024-09-16T04:41:10.841 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:10.841 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:10.841 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 4032093161, 2024-09-16T04:41:10.841 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:10.841 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:10.841 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:10.841 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:10.841 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'3", 2024-09-16T04:41:10.841 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "16'1", 2024-09-16T04:41:10.841 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:10.841 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-16T04:41:10.841 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:10.841 INFO:teuthology.orchestra.run.smithi152.stdout: "2(2)" 2024-09-16T04:41:10.842 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:10.842 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:10.842 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:10.842 INFO:teuthology.orchestra.run.smithi152.stdout: "state": "NotRecovering", 2024-09-16T04:41:10.842 INFO:teuthology.orchestra.run.smithi152.stdout: "available_might_have_unfound": true, 2024-09-16T04:41:10.842 INFO:teuthology.orchestra.run.smithi152.stdout: "might_have_unfound": [ 2024-09-16T04:41:10.842 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:10.842 INFO:teuthology.orchestra.run.smithi152.stdout: "osd": "1(0)", 2024-09-16T04:41:10.842 INFO:teuthology.orchestra.run.smithi152.stdout: "status": "osd is down" 2024-09-16T04:41:10.842 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:10.842 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:10.842 INFO:teuthology.orchestra.run.smithi152.stdout: "more": false 2024-09-16T04:41:10.842 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:41:10.859 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_5', 'key': '', 'snapid': -2, 'hash': 2998549577, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_2', 'key': '', 'snapid': -2, 'hash': 4032093161, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-16T04:41:10.860 INFO:tasks.ec_lost_unfound:reverting unfound in 1.9 2024-09-16T04:41:10.860 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.9 mark_unfound_lost delete 2024-09-16T04:41:11.121 INFO:teuthology.orchestra.run.smithi152.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T04:41:11.144 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.a state active+recovery_unfound+undersized+degraded 2024-09-16T04:41:11.145 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.a list_unfound '{}' 2024-09-16T04:41:11.244 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 17 20 21889 21869 10.0489 7.24219 0.021744 0.0155334 2024-09-16T04:41:11.364 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:41:11.364 INFO:teuthology.orchestra.run.smithi152.stdout: "offset": { 2024-09-16T04:41:11.364 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "", 2024-09-16T04:41:11.365 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:11.365 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": 0, 2024-09-16T04:41:11.365 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 0, 2024-09-16T04:41:11.365 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:11.365 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": -9223372036854775808, 2024-09-16T04:41:11.365 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:11.365 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:11.365 INFO:teuthology.orchestra.run.smithi152.stdout: "num_missing": 2, 2024-09-16T04:41:11.365 INFO:teuthology.orchestra.run.smithi152.stdout: "num_unfound": 2, 2024-09-16T04:41:11.365 INFO:teuthology.orchestra.run.smithi152.stdout: "objects": [ 2024-09-16T04:41:11.365 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:11.365 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:11.365 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existing_9", 2024-09-16T04:41:11.366 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:11.366 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:11.366 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 1217469130, 2024-09-16T04:41:11.366 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:11.366 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:11.366 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:11.366 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:11.366 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'3", 2024-09-16T04:41:11.366 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "16'1", 2024-09-16T04:41:11.366 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:11.366 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-16T04:41:11.366 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:11.366 INFO:teuthology.orchestra.run.smithi152.stdout: "2(1)" 2024-09-16T04:41:11.367 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:11.367 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:11.367 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:11.367 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:11.367 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "new_9", 2024-09-16T04:41:11.367 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:11.367 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:11.367 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 170713514, 2024-09-16T04:41:11.367 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:11.367 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:11.367 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:11.367 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:11.367 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'2", 2024-09-16T04:41:11.367 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:11.368 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:11.368 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:11.368 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:11.368 INFO:teuthology.orchestra.run.smithi152.stdout: "2(1)" 2024-09-16T04:41:11.368 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:11.368 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:11.368 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:11.368 INFO:teuthology.orchestra.run.smithi152.stdout: "state": "NotRecovering", 2024-09-16T04:41:11.368 INFO:teuthology.orchestra.run.smithi152.stdout: "available_might_have_unfound": true, 2024-09-16T04:41:11.368 INFO:teuthology.orchestra.run.smithi152.stdout: "might_have_unfound": [ 2024-09-16T04:41:11.369 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:11.369 INFO:teuthology.orchestra.run.smithi152.stdout: "osd": "1(0)", 2024-09-16T04:41:11.369 INFO:teuthology.orchestra.run.smithi152.stdout: "status": "osd is down" 2024-09-16T04:41:11.369 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:11.369 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:11.369 INFO:teuthology.orchestra.run.smithi152.stdout: "more": false 2024-09-16T04:41:11.369 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:41:11.388 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_9', 'key': '', 'snapid': -2, 'hash': 1217469130, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(1)']}, {'oid': {'oid': 'new_9', 'key': '', 'snapid': -2, 'hash': 170713514, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-16T04:41:11.389 INFO:tasks.ec_lost_unfound:reverting unfound in 1.a 2024-09-16T04:41:11.389 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.a mark_unfound_lost delete 2024-09-16T04:41:11.636 INFO:teuthology.orchestra.run.smithi152.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T04:41:11.657 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.b state active+recovery_unfound+undersized+degraded 2024-09-16T04:41:11.657 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.b list_unfound '{}' 2024-09-16T04:41:11.901 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:41:11.901 INFO:teuthology.orchestra.run.smithi152.stdout: "offset": { 2024-09-16T04:41:11.901 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "", 2024-09-16T04:41:11.901 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:11.901 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": 0, 2024-09-16T04:41:11.901 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 0, 2024-09-16T04:41:11.901 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:11.901 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": -9223372036854775808, 2024-09-16T04:41:11.901 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:11.901 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:11.902 INFO:teuthology.orchestra.run.smithi152.stdout: "num_missing": 2, 2024-09-16T04:41:11.902 INFO:teuthology.orchestra.run.smithi152.stdout: "num_unfound": 2, 2024-09-16T04:41:11.902 INFO:teuthology.orchestra.run.smithi152.stdout: "objects": [ 2024-09-16T04:41:11.902 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:11.902 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:11.902 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existed_3", 2024-09-16T04:41:11.902 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:11.902 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:11.902 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 943960171, 2024-09-16T04:41:11.902 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:11.902 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:11.902 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:11.902 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:11.903 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'5", 2024-09-16T04:41:11.903 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "0'0", 2024-09-16T04:41:11.903 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:11.903 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-16T04:41:11.903 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:11.903 INFO:teuthology.orchestra.run.smithi152.stdout: "2(2)" 2024-09-16T04:41:11.903 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:11.903 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:11.903 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:11.903 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": { 2024-09-16T04:41:11.903 INFO:teuthology.orchestra.run.smithi152.stdout: "oid": "existing_1", 2024-09-16T04:41:11.903 INFO:teuthology.orchestra.run.smithi152.stdout: "key": "", 2024-09-16T04:41:11.903 INFO:teuthology.orchestra.run.smithi152.stdout: "snapid": -2, 2024-09-16T04:41:11.903 INFO:teuthology.orchestra.run.smithi152.stdout: "hash": 2885884443, 2024-09-16T04:41:11.904 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 0, 2024-09-16T04:41:11.904 INFO:teuthology.orchestra.run.smithi152.stdout: "pool": 1, 2024-09-16T04:41:11.904 INFO:teuthology.orchestra.run.smithi152.stdout: "namespace": "" 2024-09-16T04:41:11.904 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:11.904 INFO:teuthology.orchestra.run.smithi152.stdout: "need": "19'4", 2024-09-16T04:41:11.904 INFO:teuthology.orchestra.run.smithi152.stdout: "have": "16'1", 2024-09-16T04:41:11.904 INFO:teuthology.orchestra.run.smithi152.stdout: "flags": "none", 2024-09-16T04:41:11.904 INFO:teuthology.orchestra.run.smithi152.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-16T04:41:11.904 INFO:teuthology.orchestra.run.smithi152.stdout: "locations": [ 2024-09-16T04:41:11.904 INFO:teuthology.orchestra.run.smithi152.stdout: "2(2)" 2024-09-16T04:41:11.904 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:11.904 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:11.904 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:11.905 INFO:teuthology.orchestra.run.smithi152.stdout: "state": "NotRecovering", 2024-09-16T04:41:11.905 INFO:teuthology.orchestra.run.smithi152.stdout: "available_might_have_unfound": true, 2024-09-16T04:41:11.905 INFO:teuthology.orchestra.run.smithi152.stdout: "might_have_unfound": [ 2024-09-16T04:41:11.905 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:11.905 INFO:teuthology.orchestra.run.smithi152.stdout: "osd": "1(0)", 2024-09-16T04:41:11.905 INFO:teuthology.orchestra.run.smithi152.stdout: "status": "osd is down" 2024-09-16T04:41:11.905 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:11.905 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:11.905 INFO:teuthology.orchestra.run.smithi152.stdout: "more": false 2024-09-16T04:41:11.905 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:41:11.923 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_3', 'key': '', 'snapid': -2, 'hash': 943960171, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_1', 'key': '', 'snapid': -2, 'hash': 2885884443, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-16T04:41:11.924 INFO:tasks.ec_lost_unfound:reverting unfound in 1.b 2024-09-16T04:41:11.924 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.b mark_unfound_lost delete 2024-09-16T04:41:12.164 INFO:teuthology.orchestra.run.smithi152.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T04:41:12.195 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq 5 2024-09-16T04:41:12.245 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 18 20 22870 22850 9.9161 7.66406 0.011908 0.0157431 2024-09-16T04:41:12.427 INFO:teuthology.orchestra.run.smithi152.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-16T04:41:12.453 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq 5 2024-09-16T04:41:12.703 INFO:teuthology.orchestra.run.smithi152.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-16T04:41:12.726 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 debug kick_recovery_wq 5 2024-09-16T04:41:12.963 INFO:teuthology.orchestra.run.smithi152.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-16T04:41:12.987 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T04:41:12.987 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T04:41:12.988 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T04:41:13.245 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 19 20 23763 23743 9.76137 6.97656 0.0230801 0.0159923 2024-09-16T04:41:13.270 INFO:teuthology.orchestra.run.smithi152.stdout:38654705685 2024-09-16T04:41:13.271 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:41:13.274 INFO:teuthology.orchestra.run.smithi152.stdout:98784247819 2024-09-16T04:41:13.275 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:41:13.280 INFO:teuthology.orchestra.run.smithi152.stdout:111669149706 2024-09-16T04:41:13.280 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:41:13.818 INFO:teuthology.orchestra.run.smithi152.stdout:38654705684 2024-09-16T04:41:13.841 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705685 got 38654705684 for osd.2 2024-09-16T04:41:13.907 INFO:teuthology.orchestra.run.smithi152.stdout:111669149704 2024-09-16T04:41:13.909 INFO:teuthology.orchestra.run.smithi152.stdout:98784247818 2024-09-16T04:41:13.932 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247819 got 98784247818 for osd.0 2024-09-16T04:41:13.933 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149706 got 111669149704 for osd.3 2024-09-16T04:41:14.245 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:2024-09-16T04:41:14.244742+0000 min lat: 0.00269411 max lat: 0.101645 avg lat: 0.0162559 2024-09-16T04:41:14.245 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T04:41:14.245 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 20 20 24611 24591 9.60452 6.625 0.0132389 0.0162559 2024-09-16T04:41:14.842 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:41:14.933 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:41:14.933 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:41:15.245 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 21 20 25570 25550 9.5039 7.49219 0.0397185 0.0164251 2024-09-16T04:41:15.436 INFO:teuthology.orchestra.run.smithi152.stdout:38654705686 2024-09-16T04:41:15.459 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705685 got 38654705686 for osd.2 2024-09-16T04:41:15.459 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:15.497 INFO:teuthology.orchestra.run.smithi152.stdout:111669149706 2024-09-16T04:41:15.523 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149706 got 111669149706 for osd.3 2024-09-16T04:41:15.523 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:15.532 INFO:teuthology.orchestra.run.smithi152.stdout:98784247820 2024-09-16T04:41:15.554 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247819 got 98784247820 for osd.0 2024-09-16T04:41:15.554 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:15.555 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-09-16T04:41:15.555 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:16.100 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:16.101 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:16.123 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-09-16T04:41:14.557614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":196257448,"num_objects":23962,"num_object_clones":0,"num_object_copies":95848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23989,"num_write_kb":191794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24016,"ondisk_log_size":24016,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":580420,"kb_used_data":283968,"kb_used_omap":11,"kb_used_meta":296372,"kb_avail":376906940,"statfs":{"total":386547056640,"available":385952706560,"internally_reserved":0,"allocated":290783232,"data_stored":290222387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11993,"internal_metadata":303485223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,7,4,7,9,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":143163392,"num_objects":17476,"num_object_clones":0,"num_object_copies":69904,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":17422,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":17476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17476,"num_write_kb":139808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003264"},"pg_stats":[{"pgid":"1.f","version":"27'1475","reported_seq":1502,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.234920+0000","last_change":"2024-09-16T04:40:54.254864+0000","last_active":"2024-09-16T04:41:13.234920+0000","last_peered":"2024-09-16T04:41:13.234920+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:40:44.842163+0000","last_became_peered":"2024-09-16T04:40:44.842163+0000","last_unstale":"2024-09-16T04:41:13.234920+0000","last_undegraded":"2024-09-16T04:40:54.254576+0000","last_fullsized":"2024-09-16T04:40:44.833950+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1475,"log_dups_size":0,"ondisk_log_size":1475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:15.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12083200,"num_objects":1475,"num_object_clones":0,"num_object_copies":5900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1475,"num_write_kb":11800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1475}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1517","reported_seq":1565,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.725964+0000","last_change":"2024-09-16T04:41:05.344762+0000","last_active":"2024-09-16T04:41:13.725964+0000","last_peered":"2024-09-16T04:41:13.725964+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:40:44.848800+0000","last_became_peered":"2024-09-16T04:40:44.848800+0000","last_unstale":"2024-09-16T04:41:13.725964+0000","last_undegraded":"2024-09-16T04:40:44.837670+0000","last_fullsized":"2024-09-16T04:40:44.837294+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1517,"log_dups_size":0,"ondisk_log_size":1517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:16.459367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12391692,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1515,"num_write_kb":12116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1513}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'1471","reported_seq":1529,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.568493+0000","last_change":"2024-09-16T04:41:05.820801+0000","last_active":"2024-09-16T04:41:13.568493+0000","last_peered":"2024-09-16T04:41:13.568493+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:40:44.839558+0000","last_became_peered":"2024-09-16T04:40:44.839558+0000","last_unstale":"2024-09-16T04:41:13.568493+0000","last_undegraded":"2024-09-16T04:40:44.832928+0000","last_fullsized":"2024-09-16T04:40:44.832628+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1471,"log_dups_size":0,"ondisk_log_size":1471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:16.384111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12024454,"num_objects":1468,"num_object_clones":0,"num_object_copies":5872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1470,"num_write_kb":11750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1468}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1509","reported_seq":1556,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.684873+0000","last_change":"2024-09-16T04:41:06.403977+0000","last_active":"2024-09-16T04:41:13.684873+0000","last_peered":"2024-09-16T04:41:13.684873+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:40:44.848625+0000","last_became_peered":"2024-09-16T04:40:44.848625+0000","last_unstale":"2024-09-16T04:41:13.684873+0000","last_undegraded":"2024-09-16T04:40:44.837318+0000","last_fullsized":"2024-09-16T04:40:44.835886+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1509,"log_dups_size":0,"ondisk_log_size":1509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:16.459390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12326156,"num_objects":1505,"num_object_clones":0,"num_object_copies":6020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1507,"num_write_kb":12045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1505}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'1524","reported_seq":1562,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.196915+0000","last_change":"2024-09-16T04:41:06.936664+0000","last_active":"2024-09-16T04:41:13.196915+0000","last_peered":"2024-09-16T04:41:13.196915+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:40:44.843541+0000","last_became_peered":"2024-09-16T04:40:44.843541+0000","last_unstale":"2024-09-16T04:41:13.196915+0000","last_undegraded":"2024-09-16T04:40:44.835930+0000","last_fullsized":"2024-09-16T04:40:44.834220+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1524,"log_dups_size":0,"ondisk_log_size":1524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12457228,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1522,"num_write_kb":12173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1521}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'1487","reported_seq":1514,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.200228+0000","last_change":"2024-09-16T04:40:54.254573+0000","last_active":"2024-09-16T04:41:13.200228+0000","last_peered":"2024-09-16T04:41:13.200228+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:40:44.842247+0000","last_became_peered":"2024-09-16T04:40:44.842247+0000","last_unstale":"2024-09-16T04:41:13.200228+0000","last_undegraded":"2024-09-16T04:40:54.249307+0000","last_fullsized":"2024-09-16T04:40:44.833664+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1487,"log_dups_size":0,"ondisk_log_size":1487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:54.335765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12181504,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1487,"num_write_kb":11896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"27'1500","reported_seq":1549,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.661441+0000","last_change":"2024-09-16T04:41:07.459632+0000","last_active":"2024-09-16T04:41:13.661441+0000","last_peered":"2024-09-16T04:41:13.661441+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:40:44.842032+0000","last_became_peered":"2024-09-16T04:40:44.842032+0000","last_unstale":"2024-09-16T04:41:13.661441+0000","last_undegraded":"2024-09-16T04:40:44.835742+0000","last_fullsized":"2024-09-16T04:40:44.835328+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1500,"log_dups_size":0,"ondisk_log_size":1500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12236044,"num_objects":1494,"num_object_clones":0,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1498,"num_write_kb":11964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1494}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1557","reported_seq":1606,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.697610+0000","last_change":"2024-09-16T04:41:07.993869+0000","last_active":"2024-09-16T04:41:13.697610+0000","last_peered":"2024-09-16T04:41:13.697610+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:40:44.848808+0000","last_became_peered":"2024-09-16T04:40:44.848808+0000","last_unstale":"2024-09-16T04:41:13.697610+0000","last_undegraded":"2024-09-16T04:40:44.836618+0000","last_fullsized":"2024-09-16T04:40:44.836113+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1557,"log_dups_size":0,"ondisk_log_size":1557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12711180,"num_objects":1552,"num_object_clones":0,"num_object_copies":6208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1555,"num_write_kb":12428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1552}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'1628","reported_seq":1666,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.212452+0000","last_change":"2024-09-16T04:41:08.588210+0000","last_active":"2024-09-16T04:41:13.212452+0000","last_peered":"2024-09-16T04:41:13.212452+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:40:44.842734+0000","last_became_peered":"2024-09-16T04:40:44.842734+0000","last_unstale":"2024-09-16T04:41:13.212452+0000","last_undegraded":"2024-09-16T04:40:44.833363+0000","last_fullsized":"2024-09-16T04:40:44.833064+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1628,"log_dups_size":0,"ondisk_log_size":1628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:14.417827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13265432,"num_objects":1620,"num_object_clones":0,"num_object_copies":6480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1625,"num_write_kb":12977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'1492","reported_seq":1552,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.559393+0000","last_change":"2024-09-16T04:41:09.133153+0000","last_active":"2024-09-16T04:41:13.559393+0000","last_peered":"2024-09-16T04:41:13.559393+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:40:44.839905+0000","last_became_peered":"2024-09-16T04:40:44.839905+0000","last_unstale":"2024-09-16T04:41:13.559393+0000","last_undegraded":"2024-09-16T04:40:44.833540+0000","last_fullsized":"2024-09-16T04:40:44.833235+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1492,"log_dups_size":0,"ondisk_log_size":1492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:14.331059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12203276,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1490,"num_write_kb":11918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1490}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'1496","reported_seq":1535,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.224196+0000","last_change":"2024-09-16T04:41:09.700204+0000","last_active":"2024-09-16T04:41:13.224196+0000","last_peered":"2024-09-16T04:41:13.224196+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:40:44.841868+0000","last_became_peered":"2024-09-16T04:40:44.841868+0000","last_unstale":"2024-09-16T04:41:13.224196+0000","last_undegraded":"2024-09-16T04:40:44.835179+0000","last_fullsized":"2024-09-16T04:40:44.834838+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1496,"log_dups_size":0,"ondisk_log_size":1496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:15.413550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12210066,"num_objects":1491,"num_object_clones":0,"num_object_copies":5964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1493,"num_write_kb":11932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1407","reported_seq":1430,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.171361+0000","last_change":"2024-09-16T04:41:10.154641+0000","last_active":"2024-09-16T04:41:13.171361+0000","last_peered":"2024-09-16T04:41:13.171361+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:40:44.843370+0000","last_became_peered":"2024-09-16T04:40:44.843370+0000","last_unstale":"2024-09-16T04:41:13.171361+0000","last_undegraded":"2024-09-16T04:40:44.835077+0000","last_fullsized":"2024-09-16T04:40:44.834737+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1407,"log_dups_size":0,"ondisk_log_size":1407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:15.413577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11516550,"num_objects":1406,"num_object_clones":0,"num_object_copies":5624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1406,"num_write_kb":11247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1503","reported_seq":1560,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.503963+0000","last_change":"2024-09-16T04:41:10.629432+0000","last_active":"2024-09-16T04:41:13.503963+0000","last_peered":"2024-09-16T04:41:13.503963+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:40:44.839353+0000","last_became_peered":"2024-09-16T04:40:44.839353+0000","last_unstale":"2024-09-16T04:41:13.503963+0000","last_undegraded":"2024-09-16T04:40:44.832429+0000","last_fullsized":"2024-09-16T04:40:44.832101+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1503,"log_dups_size":0,"ondisk_log_size":1503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:16.384132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12302982,"num_objects":1502,"num_object_clones":0,"num_object_copies":6008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1502,"num_write_kb":12015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'1476","reported_seq":1513,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.244234+0000","last_change":"2024-09-16T04:41:11.214366+0000","last_active":"2024-09-16T04:41:13.244234+0000","last_peered":"2024-09-16T04:41:13.244234+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:40:44.844454+0000","last_became_peered":"2024-09-16T04:40:44.844454+0000","last_unstale":"2024-09-16T04:41:13.244234+0000","last_undegraded":"2024-09-16T04:40:44.836539+0000","last_fullsized":"2024-09-16T04:40:44.836255+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:16.409985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055820,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1474,"num_write_kb":11788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1477","reported_seq":1539,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.449079+0000","last_change":"2024-09-16T04:41:11.767326+0000","last_active":"2024-09-16T04:41:13.449079+0000","last_peered":"2024-09-16T04:41:13.449079+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:40:44.840598+0000","last_became_peered":"2024-09-16T04:40:44.840598+0000","last_unstale":"2024-09-16T04:41:13.449079+0000","last_undegraded":"2024-09-16T04:40:44.833574+0000","last_fullsized":"2024-09-16T04:40:44.833243+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1477,"log_dups_size":0,"ondisk_log_size":1477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:17.433371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12072204,"num_objects":1474,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1474,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1475,"num_write_kb":11797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1474}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1497","reported_seq":1546,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.735816+0000","last_change":"2024-09-16T04:41:12.269614+0000","last_active":"2024-09-16T04:41:13.735816+0000","last_peered":"2024-09-16T04:41:13.735816+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:40:44.848727+0000","last_became_peered":"2024-09-16T04:40:44.848727+0000","last_unstale":"2024-09-16T04:41:13.735816+0000","last_undegraded":"2024-09-16T04:40:44.835805+0000","last_fullsized":"2024-09-16T04:40:44.835392+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1497,"log_dups_size":0,"ondisk_log_size":1497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:17.440381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12219660,"num_objects":1492,"num_object_clones":0,"num_object_copies":5968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1495,"num_write_kb":11948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1492}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":196257448,"num_objects":23962,"num_object_clones":0,"num_object_copies":95848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23989,"num_write_kb":191794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294588416,"data_stored":294588416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24016,"ondisk_log_size":24016,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182264,"kb_used_data":92260,"kb_used_omap":4,"kb_used_meta":89979,"kb_avail":94189576,"statfs":{"total":96636764160,"available":96450125824,"internally_reserved":0,"allocated":94474240,"data_stored":94320855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":92139215},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181088,"kb_used_data":95760,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94190752,"statfs":{"total":96636764160,"available":96451330048,"internally_reserved":0,"allocated":98058240,"data_stored":97904855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":172032,"data_stored":71342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189876,"kb_used_data":95780,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94181964,"statfs":{"total":96636764160,"available":96442331136,"internally_reserved":0,"allocated":98078720,"data_stored":97925335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":96333518},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,3,5,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":98848768,"data_stored":98848768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98353152,"data_stored":98353152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":97345536,"data_stored":97345536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:16.125 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:16.245 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 22 20 26367 26347 9.35492 6.22656 0.0223599 0.0166891 2024-09-16T04:41:16.662 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:16.662 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:16.691 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-09-16T04:41:16.558109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":196257448,"num_objects":23962,"num_object_clones":0,"num_object_copies":95848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23989,"num_write_kb":191794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24016,"ondisk_log_size":24016,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":580420,"kb_used_data":283968,"kb_used_omap":11,"kb_used_meta":296372,"kb_avail":376906940,"statfs":{"total":386547056640,"available":385952706560,"internally_reserved":0,"allocated":290783232,"data_stored":290222387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11993,"internal_metadata":303485223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,7,4,7,9,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":86147072,"num_objects":10516,"num_object_clones":0,"num_object_copies":42064,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":10462,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":10516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10516,"num_write_kb":84128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003236"},"pg_stats":[{"pgid":"1.f","version":"27'1475","reported_seq":1502,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.234920+0000","last_change":"2024-09-16T04:40:54.254864+0000","last_active":"2024-09-16T04:41:13.234920+0000","last_peered":"2024-09-16T04:41:13.234920+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:40:44.842163+0000","last_became_peered":"2024-09-16T04:40:44.842163+0000","last_unstale":"2024-09-16T04:41:13.234920+0000","last_undegraded":"2024-09-16T04:40:54.254576+0000","last_fullsized":"2024-09-16T04:40:44.833950+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1475,"log_dups_size":0,"ondisk_log_size":1475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:15.462412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12083200,"num_objects":1475,"num_object_clones":0,"num_object_copies":5900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1475,"num_write_kb":11800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1475}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1517","reported_seq":1565,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.725964+0000","last_change":"2024-09-16T04:41:05.344762+0000","last_active":"2024-09-16T04:41:13.725964+0000","last_peered":"2024-09-16T04:41:13.725964+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:40:44.848800+0000","last_became_peered":"2024-09-16T04:40:44.848800+0000","last_unstale":"2024-09-16T04:41:13.725964+0000","last_undegraded":"2024-09-16T04:40:44.837670+0000","last_fullsized":"2024-09-16T04:40:44.837294+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1517,"log_dups_size":0,"ondisk_log_size":1517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:16.459367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12391692,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1515,"num_write_kb":12116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1513}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'1471","reported_seq":1529,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.568493+0000","last_change":"2024-09-16T04:41:05.820801+0000","last_active":"2024-09-16T04:41:13.568493+0000","last_peered":"2024-09-16T04:41:13.568493+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:40:44.839558+0000","last_became_peered":"2024-09-16T04:40:44.839558+0000","last_unstale":"2024-09-16T04:41:13.568493+0000","last_undegraded":"2024-09-16T04:40:44.832928+0000","last_fullsized":"2024-09-16T04:40:44.832628+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1471,"log_dups_size":0,"ondisk_log_size":1471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:16.384111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12024454,"num_objects":1468,"num_object_clones":0,"num_object_copies":5872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1470,"num_write_kb":11750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1468}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1509","reported_seq":1556,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.684873+0000","last_change":"2024-09-16T04:41:06.403977+0000","last_active":"2024-09-16T04:41:13.684873+0000","last_peered":"2024-09-16T04:41:13.684873+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:40:44.848625+0000","last_became_peered":"2024-09-16T04:40:44.848625+0000","last_unstale":"2024-09-16T04:41:13.684873+0000","last_undegraded":"2024-09-16T04:40:44.837318+0000","last_fullsized":"2024-09-16T04:40:44.835886+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1509,"log_dups_size":0,"ondisk_log_size":1509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:16.459390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12326156,"num_objects":1505,"num_object_clones":0,"num_object_copies":6020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1507,"num_write_kb":12045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1505}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'1524","reported_seq":1562,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.196915+0000","last_change":"2024-09-16T04:41:06.936664+0000","last_active":"2024-09-16T04:41:13.196915+0000","last_peered":"2024-09-16T04:41:13.196915+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:40:44.843541+0000","last_became_peered":"2024-09-16T04:40:44.843541+0000","last_unstale":"2024-09-16T04:41:13.196915+0000","last_undegraded":"2024-09-16T04:40:44.835930+0000","last_fullsized":"2024-09-16T04:40:44.834220+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1524,"log_dups_size":0,"ondisk_log_size":1524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12457228,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1522,"num_write_kb":12173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1521}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'1487","reported_seq":1514,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.200228+0000","last_change":"2024-09-16T04:40:54.254573+0000","last_active":"2024-09-16T04:41:13.200228+0000","last_peered":"2024-09-16T04:41:13.200228+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:40:44.842247+0000","last_became_peered":"2024-09-16T04:40:44.842247+0000","last_unstale":"2024-09-16T04:41:13.200228+0000","last_undegraded":"2024-09-16T04:40:54.249307+0000","last_fullsized":"2024-09-16T04:40:44.833664+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1487,"log_dups_size":0,"ondisk_log_size":1487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:54.335765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12181504,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1487,"num_write_kb":11896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1487}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"27'1500","reported_seq":1549,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.661441+0000","last_change":"2024-09-16T04:41:07.459632+0000","last_active":"2024-09-16T04:41:13.661441+0000","last_peered":"2024-09-16T04:41:13.661441+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:40:44.842032+0000","last_became_peered":"2024-09-16T04:40:44.842032+0000","last_unstale":"2024-09-16T04:41:13.661441+0000","last_undegraded":"2024-09-16T04:40:44.835742+0000","last_fullsized":"2024-09-16T04:40:44.835328+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1500,"log_dups_size":0,"ondisk_log_size":1500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12236044,"num_objects":1494,"num_object_clones":0,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1498,"num_write_kb":11964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1494}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1557","reported_seq":1606,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.697610+0000","last_change":"2024-09-16T04:41:07.993869+0000","last_active":"2024-09-16T04:41:13.697610+0000","last_peered":"2024-09-16T04:41:13.697610+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:40:44.848808+0000","last_became_peered":"2024-09-16T04:40:44.848808+0000","last_unstale":"2024-09-16T04:41:13.697610+0000","last_undegraded":"2024-09-16T04:40:44.836618+0000","last_fullsized":"2024-09-16T04:40:44.836113+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1557,"log_dups_size":0,"ondisk_log_size":1557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12711180,"num_objects":1552,"num_object_clones":0,"num_object_copies":6208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1555,"num_write_kb":12428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1552}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'1628","reported_seq":1666,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.212452+0000","last_change":"2024-09-16T04:41:08.588210+0000","last_active":"2024-09-16T04:41:13.212452+0000","last_peered":"2024-09-16T04:41:13.212452+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:40:44.842734+0000","last_became_peered":"2024-09-16T04:40:44.842734+0000","last_unstale":"2024-09-16T04:41:13.212452+0000","last_undegraded":"2024-09-16T04:40:44.833363+0000","last_fullsized":"2024-09-16T04:40:44.833064+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1628,"log_dups_size":0,"ondisk_log_size":1628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:14.417827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13265432,"num_objects":1620,"num_object_clones":0,"num_object_copies":6480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1625,"num_write_kb":12977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'1492","reported_seq":1552,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.559393+0000","last_change":"2024-09-16T04:41:09.133153+0000","last_active":"2024-09-16T04:41:13.559393+0000","last_peered":"2024-09-16T04:41:13.559393+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:40:44.839905+0000","last_became_peered":"2024-09-16T04:40:44.839905+0000","last_unstale":"2024-09-16T04:41:13.559393+0000","last_undegraded":"2024-09-16T04:40:44.833540+0000","last_fullsized":"2024-09-16T04:40:44.833235+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1492,"log_dups_size":0,"ondisk_log_size":1492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:14.331059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12203276,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1490,"num_write_kb":11918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1490}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'1496","reported_seq":1535,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.224196+0000","last_change":"2024-09-16T04:41:09.700204+0000","last_active":"2024-09-16T04:41:13.224196+0000","last_peered":"2024-09-16T04:41:13.224196+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:40:44.841868+0000","last_became_peered":"2024-09-16T04:40:44.841868+0000","last_unstale":"2024-09-16T04:41:13.224196+0000","last_undegraded":"2024-09-16T04:40:44.835179+0000","last_fullsized":"2024-09-16T04:40:44.834838+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1496,"log_dups_size":0,"ondisk_log_size":1496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:15.413550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12210066,"num_objects":1491,"num_object_clones":0,"num_object_copies":5964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1493,"num_write_kb":11932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1407","reported_seq":1430,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.171361+0000","last_change":"2024-09-16T04:41:10.154641+0000","last_active":"2024-09-16T04:41:13.171361+0000","last_peered":"2024-09-16T04:41:13.171361+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:40:44.843370+0000","last_became_peered":"2024-09-16T04:40:44.843370+0000","last_unstale":"2024-09-16T04:41:13.171361+0000","last_undegraded":"2024-09-16T04:40:44.835077+0000","last_fullsized":"2024-09-16T04:40:44.834737+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1407,"log_dups_size":0,"ondisk_log_size":1407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:15.413577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11516550,"num_objects":1406,"num_object_clones":0,"num_object_copies":5624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1406,"num_write_kb":11247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1503","reported_seq":1560,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.503963+0000","last_change":"2024-09-16T04:41:10.629432+0000","last_active":"2024-09-16T04:41:13.503963+0000","last_peered":"2024-09-16T04:41:13.503963+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:40:44.839353+0000","last_became_peered":"2024-09-16T04:40:44.839353+0000","last_unstale":"2024-09-16T04:41:13.503963+0000","last_undegraded":"2024-09-16T04:40:44.832429+0000","last_fullsized":"2024-09-16T04:40:44.832101+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1503,"log_dups_size":0,"ondisk_log_size":1503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:16.384132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12302982,"num_objects":1502,"num_object_clones":0,"num_object_copies":6008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1502,"num_write_kb":12015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'1476","reported_seq":1513,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.244234+0000","last_change":"2024-09-16T04:41:11.214366+0000","last_active":"2024-09-16T04:41:13.244234+0000","last_peered":"2024-09-16T04:41:13.244234+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:40:44.844454+0000","last_became_peered":"2024-09-16T04:40:44.844454+0000","last_unstale":"2024-09-16T04:41:13.244234+0000","last_undegraded":"2024-09-16T04:40:44.836539+0000","last_fullsized":"2024-09-16T04:40:44.836255+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:16.409985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055820,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1474,"num_write_kb":11788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1477","reported_seq":1539,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.449079+0000","last_change":"2024-09-16T04:41:11.767326+0000","last_active":"2024-09-16T04:41:13.449079+0000","last_peered":"2024-09-16T04:41:13.449079+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:40:44.840598+0000","last_became_peered":"2024-09-16T04:40:44.840598+0000","last_unstale":"2024-09-16T04:41:13.449079+0000","last_undegraded":"2024-09-16T04:40:44.833574+0000","last_fullsized":"2024-09-16T04:40:44.833243+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1477,"log_dups_size":0,"ondisk_log_size":1477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:17.433371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12072204,"num_objects":1474,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1474,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1475,"num_write_kb":11797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1474}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1497","reported_seq":1546,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-16T04:41:13.735816+0000","last_change":"2024-09-16T04:41:12.269614+0000","last_active":"2024-09-16T04:41:13.735816+0000","last_peered":"2024-09-16T04:41:13.735816+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:40:44.848727+0000","last_became_peered":"2024-09-16T04:40:44.848727+0000","last_unstale":"2024-09-16T04:41:13.735816+0000","last_undegraded":"2024-09-16T04:40:44.835805+0000","last_fullsized":"2024-09-16T04:40:44.835392+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":1497,"log_dups_size":0,"ondisk_log_size":1497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:17.440381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12219660,"num_objects":1492,"num_object_clones":0,"num_object_copies":5968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1495,"num_write_kb":11948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1492}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":196257448,"num_objects":23962,"num_object_clones":0,"num_object_copies":95848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23989,"num_write_kb":191794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294588416,"data_stored":294588416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24016,"ondisk_log_size":24016,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182264,"kb_used_data":92260,"kb_used_omap":4,"kb_used_meta":89979,"kb_avail":94189576,"statfs":{"total":96636764160,"available":96450125824,"internally_reserved":0,"allocated":94474240,"data_stored":94320855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":92139215},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181088,"kb_used_data":95760,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94190752,"statfs":{"total":96636764160,"available":96451330048,"internally_reserved":0,"allocated":98058240,"data_stored":97904855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":172032,"data_stored":71342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189876,"kb_used_data":95780,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94181964,"statfs":{"total":96636764160,"available":96442331136,"internally_reserved":0,"allocated":98078720,"data_stored":97925335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":96333518},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,3,5,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":98848768,"data_stored":98848768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98353152,"data_stored":98353152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":97345536,"data_stored":97345536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:16.692 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-09-16T04:41:16.692 INFO:tasks.util.rados:rados -p unique_pool_0 get new_1 - 2024-09-16T04:41:16.692 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_1 - 2024-09-16T04:41:16.798 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/new_1: (2) No such file or directory 2024-09-16T04:41:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:16.807 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_1 - 2024-09-16T04:41:16.808 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_1 - 2024-09-16T04:41:16.897 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existed_1: (2) No such file or directory 2024-09-16T04:41:16.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:16.906 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_1 - 2024-09-16T04:41:16.907 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_1 - 2024-09-16T04:41:17.009 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existing_1: (2) No such file or directory 2024-09-16T04:41:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:17.019 INFO:tasks.util.rados:rados -p unique_pool_0 get new_2 - 2024-09-16T04:41:17.019 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_2 - 2024-09-16T04:41:17.126 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/new_2: (2) No such file or directory 2024-09-16T04:41:17.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:17.138 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_2 - 2024-09-16T04:41:17.139 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_2 - 2024-09-16T04:41:17.243 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existed_2: (2) No such file or directory 2024-09-16T04:41:17.245 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 23 20 27287 27267 9.26066 7.1875 0.0252302 0.0168604 2024-09-16T04:41:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:17.255 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_2 - 2024-09-16T04:41:17.256 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_2 - 2024-09-16T04:41:17.353 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existing_2: (2) No such file or directory 2024-09-16T04:41:17.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:17.359 INFO:tasks.util.rados:rados -p unique_pool_0 get new_3 - 2024-09-16T04:41:17.359 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_3 - 2024-09-16T04:41:17.454 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/new_3: (2) No such file or directory 2024-09-16T04:41:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:17.465 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_3 - 2024-09-16T04:41:17.465 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_3 - 2024-09-16T04:41:17.569 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existed_3: (2) No such file or directory 2024-09-16T04:41:17.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:17.579 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_3 - 2024-09-16T04:41:17.579 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_3 - 2024-09-16T04:41:17.677 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existing_3: (2) No such file or directory 2024-09-16T04:41:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:17.686 INFO:tasks.util.rados:rados -p unique_pool_0 get new_4 - 2024-09-16T04:41:17.686 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_4 - 2024-09-16T04:41:17.794 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/new_4: (2) No such file or directory 2024-09-16T04:41:17.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:17.809 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_4 - 2024-09-16T04:41:17.809 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_4 - 2024-09-16T04:41:17.904 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existed_4: (2) No such file or directory 2024-09-16T04:41:17.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:17.909 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_4 - 2024-09-16T04:41:17.910 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_4 - 2024-09-16T04:41:18.018 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existing_4: (2) No such file or directory 2024-09-16T04:41:18.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:18.030 INFO:tasks.util.rados:rados -p unique_pool_0 get new_5 - 2024-09-16T04:41:18.030 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_5 - 2024-09-16T04:41:18.132 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/new_5: (2) No such file or directory 2024-09-16T04:41:18.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:18.140 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_5 - 2024-09-16T04:41:18.140 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_5 - 2024-09-16T04:41:18.242 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existed_5: (2) No such file or directory 2024-09-16T04:41:18.246 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 24 20 28195 28175 9.17035 7.09375 0.0191957 0.0170275 2024-09-16T04:41:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:18.252 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_5 - 2024-09-16T04:41:18.252 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_5 - 2024-09-16T04:41:18.355 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existing_5: (2) No such file or directory 2024-09-16T04:41:18.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:18.361 INFO:tasks.util.rados:rados -p unique_pool_0 get new_6 - 2024-09-16T04:41:18.366 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_6 - 2024-09-16T04:41:18.468 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/new_6: (2) No such file or directory 2024-09-16T04:41:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:18.474 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_6 - 2024-09-16T04:41:18.475 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_6 - 2024-09-16T04:41:18.573 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existed_6: (2) No such file or directory 2024-09-16T04:41:18.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:18.579 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_6 - 2024-09-16T04:41:18.579 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_6 - 2024-09-16T04:41:18.683 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existing_6: (2) No such file or directory 2024-09-16T04:41:18.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:18.689 INFO:tasks.util.rados:rados -p unique_pool_0 get new_7 - 2024-09-16T04:41:18.690 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_7 - 2024-09-16T04:41:18.792 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/new_7: (2) No such file or directory 2024-09-16T04:41:18.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:18.804 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_7 - 2024-09-16T04:41:18.804 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_7 - 2024-09-16T04:41:18.907 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existed_7: (2) No such file or directory 2024-09-16T04:41:18.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:18.914 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_7 - 2024-09-16T04:41:18.914 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_7 - 2024-09-16T04:41:19.008 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existing_7: (2) No such file or directory 2024-09-16T04:41:19.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:19.020 INFO:tasks.util.rados:rados -p unique_pool_0 get new_8 - 2024-09-16T04:41:19.020 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_8 - 2024-09-16T04:41:19.124 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/new_8: (2) No such file or directory 2024-09-16T04:41:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:19.129 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_8 - 2024-09-16T04:41:19.129 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_8 - 2024-09-16T04:41:19.231 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existed_8: (2) No such file or directory 2024-09-16T04:41:19.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:19.238 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_8 - 2024-09-16T04:41:19.238 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_8 - 2024-09-16T04:41:19.247 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 25 20 29102 29082 9.08651 7.08594 0.0171477 0.0171834 2024-09-16T04:41:19.333 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existing_8: (2) No such file or directory 2024-09-16T04:41:19.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:19.343 INFO:tasks.util.rados:rados -p unique_pool_0 get new_9 - 2024-09-16T04:41:19.344 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_9 - 2024-09-16T04:41:19.493 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/new_9: (2) No such file or directory 2024-09-16T04:41:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:19.505 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_9 - 2024-09-16T04:41:19.505 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_9 - 2024-09-16T04:41:19.605 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existed_9: (2) No such file or directory 2024-09-16T04:41:19.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:19.610 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_9 - 2024-09-16T04:41:19.610 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_9 - 2024-09-16T04:41:19.716 INFO:teuthology.orchestra.run.smithi152.stderr:error getting unique_pool_0/existing_9: (2) No such file or directory 2024-09-16T04:41:19.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:41:19.721 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T04:41:19.721 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T04:41:19.723 INFO:tasks.ceph.osd.1:Started 2024-09-16T04:41:19.724 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:41:19.891 INFO:teuthology.orchestra.run.smithi152.stderr:no valid command found; 10 closest matches: 2024-09-16T04:41:19.891 INFO:teuthology.orchestra.run.smithi152.stderr:0 2024-09-16T04:41:19.891 INFO:teuthology.orchestra.run.smithi152.stderr:1 2024-09-16T04:41:19.891 INFO:teuthology.orchestra.run.smithi152.stderr:2 2024-09-16T04:41:19.892 INFO:teuthology.orchestra.run.smithi152.stderr:abort 2024-09-16T04:41:19.892 INFO:teuthology.orchestra.run.smithi152.stderr:assert 2024-09-16T04:41:19.892 INFO:teuthology.orchestra.run.smithi152.stderr:config diff 2024-09-16T04:41:19.892 INFO:teuthology.orchestra.run.smithi152.stderr:config diff get 2024-09-16T04:41:19.892 INFO:teuthology.orchestra.run.smithi152.stderr:config get 2024-09-16T04:41:19.892 INFO:teuthology.orchestra.run.smithi152.stderr:config help [] 2024-09-16T04:41:19.892 INFO:teuthology.orchestra.run.smithi152.stderr:config set ... 2024-09-16T04:41:19.892 INFO:teuthology.orchestra.run.smithi152.stderr:admin_socket: invalid command 2024-09-16T04:41:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:41:19.895 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-16T04:41:20.247 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 26 20 30010 29990 9.0096 7.09375 0.0190697 0.0173322 2024-09-16T04:41:20.950 INFO:tasks.ceph.osd.1.smithi152.stderr:2024-09-16T04:41:20.942+0000 7f4fae611640 -1 Falling back to public interface 2024-09-16T04:41:21.247 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 27 20 31007 30987 8.96438 7.78906 0.0175495 0.0174196 2024-09-16T04:41:22.248 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 28 20 32005 31985 8.92266 7.79688 0.0146815 0.0175014 2024-09-16T04:41:23.248 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 29 20 32952 32932 8.87009 7.39844 0.0173888 0.0176048 2024-09-16T04:41:24.248 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 30 20 33788 33768 8.79212 6.53125 0.0192072 0.0177616 2024-09-16T04:41:24.895 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:41:25.073 INFO:teuthology.orchestra.run.smithi152.stderr:no valid command found; 10 closest matches: 2024-09-16T04:41:25.073 INFO:teuthology.orchestra.run.smithi152.stderr:0 2024-09-16T04:41:25.073 INFO:teuthology.orchestra.run.smithi152.stderr:1 2024-09-16T04:41:25.073 INFO:teuthology.orchestra.run.smithi152.stderr:2 2024-09-16T04:41:25.073 INFO:teuthology.orchestra.run.smithi152.stderr:abort 2024-09-16T04:41:25.073 INFO:teuthology.orchestra.run.smithi152.stderr:assert 2024-09-16T04:41:25.073 INFO:teuthology.orchestra.run.smithi152.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:41:25.073 INFO:teuthology.orchestra.run.smithi152.stderr:bluefs files list 2024-09-16T04:41:25.073 INFO:teuthology.orchestra.run.smithi152.stderr:bluefs stats 2024-09-16T04:41:25.073 INFO:teuthology.orchestra.run.smithi152.stderr:bluestore allocator dump block 2024-09-16T04:41:25.074 INFO:teuthology.orchestra.run.smithi152.stderr:bluestore allocator fragmentation block 2024-09-16T04:41:25.074 INFO:teuthology.orchestra.run.smithi152.stderr:admin_socket: invalid command 2024-09-16T04:41:25.075 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:41:25.075 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-16T04:41:25.094 INFO:tasks.ceph.osd.1.smithi152.stderr:2024-09-16T04:41:25.090+0000 7f4fae611640 -1 osd.1 19 log_to_monitors true 2024-09-16T04:41:25.187 INFO:tasks.ceph.osd.1.smithi152.stderr:2024-09-16T04:41:25.184+0000 7f4fa1d0d640 -1 osd.1 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:25.248 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 31 20 34676 34656 8.73228 6.9375 0.0162552 0.0178849 2024-09-16T04:41:26.248 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 32 20 35535 35515 8.66909 6.71094 0.0287786 0.0179786 2024-09-16T04:41:27.248 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 33 20 35535 35515 8.40639 0 - 0.0179786 2024-09-16T04:41:28.248 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 34 20 35721 35701 8.2019 0.726562 0.0759861 0.019032 2024-09-16T04:41:29.248 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 35 20 36172 36152 8.06823 3.52344 0.0588441 0.0193539 2024-09-16T04:41:30.077 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:41:30.248 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 36 20 36598 36578 7.93657 3.32812 0.10849 0.0196645 2024-09-16T04:41:30.280 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:41:30.280 INFO:teuthology.orchestra.run.smithi152.stdout: "ops": [ 2024-09-16T04:41:30.280 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.281 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(1.ds1 31/30 ECSubWrite(tid=12747, reqid=client.4991.0:36593, at_version=31'2293, trim_to=0'0, roll_forward_to=31'2292))", 2024-09-16T04:41:30.281 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-16T04:41:30.243018+0000", 2024-09-16T04:41:30.281 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.034042627999999998, 2024-09-16T04:41:30.281 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.034067074000000003, 2024-09-16T04:41:30.281 INFO:teuthology.orchestra.run.smithi152.stdout: "continuous": false, 2024-09-16T04:41:30.281 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-16T04:41:30.281 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-16T04:41:30.281 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-16T04:41:30.281 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.281 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-16T04:41:30.281 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.243018+0000", 2024-09-16T04:41:30.281 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.281 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.282 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.282 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-16T04:41:30.282 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.243018+0000", 2024-09-16T04:41:30.282 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.282 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.282 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.282 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-16T04:41:30.282 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.243018+0000", 2024-09-16T04:41:30.282 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 5.9100000000000004e-07 2024-09-16T04:41:30.282 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.282 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.282 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-16T04:41:30.283 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.243039+0000", 2024-09-16T04:41:30.283 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.0336999999999999e-05 2024-09-16T04:41:30.283 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.283 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.283 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-16T04:41:30.283 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.243047+0000", 2024-09-16T04:41:30.283 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.568e-06 2024-09-16T04:41:30.283 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.283 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.283 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-16T04:41:30.283 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.243054+0000", 2024-09-16T04:41:30.283 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.6900000000000003e-06 2024-09-16T04:41:30.284 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.284 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.284 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-16T04:41:30.284 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.246985+0000", 2024-09-16T04:41:30.284 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0039308329999999999 2024-09-16T04:41:30.284 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.284 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.284 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-16T04:41:30.284 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.246995+0000", 2024-09-16T04:41:30.284 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0295999999999999e-05 2024-09-16T04:41:30.284 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.284 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:30.285 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.285 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.285 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.285 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(1.5s3 31/30 ECSubWrite(tid=12748, reqid=client.4991.0:36584, at_version=0'0, trim_to=0'0, roll_forward_to=31'2272))", 2024-09-16T04:41:30.285 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-16T04:41:30.252021+0000", 2024-09-16T04:41:30.285 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.025039511, 2024-09-16T04:41:30.285 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.025124388000000001, 2024-09-16T04:41:30.285 INFO:teuthology.orchestra.run.smithi152.stdout: "continuous": false, 2024-09-16T04:41:30.285 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-16T04:41:30.285 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-16T04:41:30.285 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-16T04:41:30.285 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.285 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-16T04:41:30.286 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252021+0000", 2024-09-16T04:41:30.286 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.286 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.286 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.286 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-16T04:41:30.286 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252021+0000", 2024-09-16T04:41:30.286 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.286 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.286 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.286 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-16T04:41:30.286 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252022+0000", 2024-09-16T04:41:30.286 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2780000000000001e-06 2024-09-16T04:41:30.287 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.287 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.287 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-16T04:41:30.287 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252037+0000", 2024-09-16T04:41:30.287 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.4959e-05 2024-09-16T04:41:30.287 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.287 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.287 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-16T04:41:30.287 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252050+0000", 2024-09-16T04:41:30.287 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2942000000000001e-05 2024-09-16T04:41:30.287 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.287 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.287 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-16T04:41:30.287 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252059+0000", 2024-09-16T04:41:30.288 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.363e-06 2024-09-16T04:41:30.288 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.288 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.288 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-16T04:41:30.288 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252234+0000", 2024-09-16T04:41:30.288 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00017406600000000001 2024-09-16T04:41:30.288 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.288 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.288 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-16T04:41:30.288 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252244+0000", 2024-09-16T04:41:30.288 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0256e-05 2024-09-16T04:41:30.288 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.289 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:30.289 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.289 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.289 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.289 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(1.5s3 31/30 ECSubWrite(tid=12750, reqid=client.4991.0:36597, at_version=31'2273, trim_to=0'0, roll_forward_to=31'2272))", 2024-09-16T04:41:30.289 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-16T04:41:30.252120+0000", 2024-09-16T04:41:30.289 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.024940560000000001, 2024-09-16T04:41:30.289 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.025078538000000001, 2024-09-16T04:41:30.289 INFO:teuthology.orchestra.run.smithi152.stdout: "continuous": false, 2024-09-16T04:41:30.289 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-16T04:41:30.289 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-16T04:41:30.289 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-16T04:41:30.289 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.290 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-16T04:41:30.290 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252120+0000", 2024-09-16T04:41:30.290 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.290 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.290 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.290 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-16T04:41:30.290 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252120+0000", 2024-09-16T04:41:30.290 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.290 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.290 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.290 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-16T04:41:30.290 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252120+0000", 2024-09-16T04:41:30.290 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.2000000000000001e-07 2024-09-16T04:41:30.290 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.290 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.291 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-16T04:41:30.291 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252142+0000", 2024-09-16T04:41:30.291 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.1895e-05 2024-09-16T04:41:30.291 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.291 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.291 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-16T04:41:30.291 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252150+0000", 2024-09-16T04:41:30.291 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.1030000000000002e-06 2024-09-16T04:41:30.291 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.291 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.291 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-16T04:41:30.291 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252157+0000", 2024-09-16T04:41:30.291 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.3980000000000004e-06 2024-09-16T04:41:30.291 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.292 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.292 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-16T04:41:30.292 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.255354+0000", 2024-09-16T04:41:30.292 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0031970330000000002 2024-09-16T04:41:30.292 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.292 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.292 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-16T04:41:30.292 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.255362+0000", 2024-09-16T04:41:30.292 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4470000000000006e-06 2024-09-16T04:41:30.292 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.292 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:30.292 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.292 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.293 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.293 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(1.0s0 31/30 ECSubWrite(tid=16106, reqid=client.4991.0:36585, at_version=0'0, trim_to=0'0, roll_forward_to=31'2307))", 2024-09-16T04:41:30.293 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-16T04:41:30.252891+0000", 2024-09-16T04:41:30.293 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.024169499000000001, 2024-09-16T04:41:30.293 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.024359440999999999, 2024-09-16T04:41:30.293 INFO:teuthology.orchestra.run.smithi152.stdout: "continuous": false, 2024-09-16T04:41:30.293 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-16T04:41:30.293 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-16T04:41:30.293 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-16T04:41:30.293 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.293 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-16T04:41:30.293 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252891+0000", 2024-09-16T04:41:30.293 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.294 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.294 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.294 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-16T04:41:30.294 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252891+0000", 2024-09-16T04:41:30.294 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.294 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.294 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.294 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-16T04:41:30.294 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252892+0000", 2024-09-16T04:41:30.294 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.0800000000000004e-07 2024-09-16T04:41:30.294 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.294 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.294 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-16T04:41:30.295 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252905+0000", 2024-09-16T04:41:30.295 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.3263e-05 2024-09-16T04:41:30.295 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.295 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.295 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-16T04:41:30.295 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252916+0000", 2024-09-16T04:41:30.295 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0747999999999999e-05 2024-09-16T04:41:30.295 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.295 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.295 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-16T04:41:30.295 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.252924+0000", 2024-09-16T04:41:30.295 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.7999999999999999e-06 2024-09-16T04:41:30.295 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.296 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.296 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-16T04:41:30.296 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.253198+0000", 2024-09-16T04:41:30.296 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00027429400000000001 2024-09-16T04:41:30.296 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.296 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.296 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-16T04:41:30.296 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.253206+0000", 2024-09-16T04:41:30.296 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.2719999999999997e-06 2024-09-16T04:41:30.296 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.296 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:30.297 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.297 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.297 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.297 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(1.ds1 31/30 ECSubWrite(tid=12751, reqid=client.4991.0:36601, at_version=31'2294, trim_to=0'0, roll_forward_to=31'2292))", 2024-09-16T04:41:30.297 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-16T04:41:30.255108+0000", 2024-09-16T04:41:30.297 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.021952231999999999, 2024-09-16T04:41:30.297 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.022207201999999999, 2024-09-16T04:41:30.297 INFO:teuthology.orchestra.run.smithi152.stdout: "continuous": false, 2024-09-16T04:41:30.297 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-16T04:41:30.297 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-16T04:41:30.297 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-16T04:41:30.297 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.297 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-16T04:41:30.298 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.255108+0000", 2024-09-16T04:41:30.298 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.298 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.298 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.298 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-16T04:41:30.298 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.255108+0000", 2024-09-16T04:41:30.298 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.298 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.298 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.298 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-16T04:41:30.298 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.255109+0000", 2024-09-16T04:41:30.298 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2580000000000001e-06 2024-09-16T04:41:30.298 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.298 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.299 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-16T04:41:30.299 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.255137+0000", 2024-09-16T04:41:30.299 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7608000000000001e-05 2024-09-16T04:41:30.299 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.299 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.299 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-16T04:41:30.299 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.255149+0000", 2024-09-16T04:41:30.299 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2296999999999999e-05 2024-09-16T04:41:30.299 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.299 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.300 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-16T04:41:30.300 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.255159+0000", 2024-09-16T04:41:30.300 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.4839999999999993e-06 2024-09-16T04:41:30.300 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.300 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.300 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-16T04:41:30.300 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.257766+0000", 2024-09-16T04:41:30.300 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.002607416 2024-09-16T04:41:30.300 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.300 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.300 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-16T04:41:30.300 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.257776+0000", 2024-09-16T04:41:30.300 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0069e-05 2024-09-16T04:41:30.301 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.301 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:30.301 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.301 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.301 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.301 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(1.ds1 31/30 ECSubWrite(tid=12752, reqid=client.4991.0:36602, at_version=31'2295, trim_to=0'0, roll_forward_to=31'2292))", 2024-09-16T04:41:30.301 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-16T04:41:30.260111+0000", 2024-09-16T04:41:30.301 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.016949255999999999, 2024-09-16T04:41:30.301 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.017256736000000002, 2024-09-16T04:41:30.301 INFO:teuthology.orchestra.run.smithi152.stdout: "continuous": false, 2024-09-16T04:41:30.301 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-16T04:41:30.301 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-16T04:41:30.301 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-16T04:41:30.301 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.302 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-16T04:41:30.302 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.260111+0000", 2024-09-16T04:41:30.302 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.302 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.302 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.302 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-16T04:41:30.302 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.260111+0000", 2024-09-16T04:41:30.302 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.302 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.302 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.302 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-16T04:41:30.302 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.260113+0000", 2024-09-16T04:41:30.302 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.75e-06 2024-09-16T04:41:30.303 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.303 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.303 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-16T04:41:30.303 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.260137+0000", 2024-09-16T04:41:30.303 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.3643999999999999e-05 2024-09-16T04:41:30.303 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.303 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.303 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-16T04:41:30.303 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.260149+0000", 2024-09-16T04:41:30.303 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2002e-05 2024-09-16T04:41:30.303 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.303 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.303 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-16T04:41:30.304 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.260158+0000", 2024-09-16T04:41:30.304 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.0729999999999994e-06 2024-09-16T04:41:30.304 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.304 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.304 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-16T04:41:30.304 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.266065+0000", 2024-09-16T04:41:30.304 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0059071510000000002 2024-09-16T04:41:30.304 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.304 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.304 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-16T04:41:30.304 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.266074+0000", 2024-09-16T04:41:30.304 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.5510000000000005e-06 2024-09-16T04:41:30.304 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.304 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:30.305 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.305 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.305 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.305 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(1.0s0 31/30 ECSubWrite(tid=16108, reqid=client.4991.0:36599, at_version=31'2308, trim_to=0'0, roll_forward_to=31'2307))", 2024-09-16T04:41:30.305 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-16T04:41:30.261387+0000", 2024-09-16T04:41:30.305 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.015673853000000001, 2024-09-16T04:41:30.305 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.016035964, 2024-09-16T04:41:30.305 INFO:teuthology.orchestra.run.smithi152.stdout: "continuous": false, 2024-09-16T04:41:30.305 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-16T04:41:30.305 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-16T04:41:30.305 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-16T04:41:30.305 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.305 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-16T04:41:30.305 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.261387+0000", 2024-09-16T04:41:30.306 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.306 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.306 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.306 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-16T04:41:30.306 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.261387+0000", 2024-09-16T04:41:30.306 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.306 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.306 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.306 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-16T04:41:30.306 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.261387+0000", 2024-09-16T04:41:30.306 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.75e-07 2024-09-16T04:41:30.306 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.306 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.306 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-16T04:41:30.307 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.261411+0000", 2024-09-16T04:41:30.307 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.4060999999999999e-05 2024-09-16T04:41:30.307 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.307 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.307 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-16T04:41:30.307 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.261424+0000", 2024-09-16T04:41:30.307 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2330999999999999e-05 2024-09-16T04:41:30.307 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.307 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.307 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-16T04:41:30.307 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.261431+0000", 2024-09-16T04:41:30.307 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6199999999999999e-06 2024-09-16T04:41:30.307 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.307 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.308 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-16T04:41:30.308 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.261505+0000", 2024-09-16T04:41:30.308 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.4220000000000004e-05 2024-09-16T04:41:30.308 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.308 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.308 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-16T04:41:30.308 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.261516+0000", 2024-09-16T04:41:30.308 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0151e-05 2024-09-16T04:41:30.308 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.308 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:30.308 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.308 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.308 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.308 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(1.5s3 31/30 ECSubWrite(tid=12753, reqid=client.4991.0:36603, at_version=31'2274, trim_to=0'0, roll_forward_to=31'2272))", 2024-09-16T04:41:30.309 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-16T04:41:30.263550+0000", 2024-09-16T04:41:30.309 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.013510476, 2024-09-16T04:41:30.309 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.013924324, 2024-09-16T04:41:30.309 INFO:teuthology.orchestra.run.smithi152.stdout: "continuous": false, 2024-09-16T04:41:30.309 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-16T04:41:30.309 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-16T04:41:30.309 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-16T04:41:30.309 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.309 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-16T04:41:30.309 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.263550+0000", 2024-09-16T04:41:30.309 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.309 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.309 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.309 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-16T04:41:30.310 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.263550+0000", 2024-09-16T04:41:30.310 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.310 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.310 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.310 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-16T04:41:30.310 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.263551+0000", 2024-09-16T04:41:30.310 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.0499999999999999e-06 2024-09-16T04:41:30.310 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.310 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.310 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-16T04:41:30.310 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.263577+0000", 2024-09-16T04:41:30.310 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.5772e-05 2024-09-16T04:41:30.310 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.311 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.311 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-16T04:41:30.311 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.263589+0000", 2024-09-16T04:41:30.311 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.2075999999999999e-05 2024-09-16T04:41:30.311 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.311 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.311 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-16T04:41:30.311 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.263597+0000", 2024-09-16T04:41:30.311 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4710000000000007e-06 2024-09-16T04:41:30.311 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.311 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.311 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-16T04:41:30.311 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.265605+0000", 2024-09-16T04:41:30.312 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0020081420000000001 2024-09-16T04:41:30.312 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.312 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.312 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-16T04:41:30.312 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.265617+0000", 2024-09-16T04:41:30.312 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.1804000000000001e-05 2024-09-16T04:41:30.312 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.312 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:30.312 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.312 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.312 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.312 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(1.0s0 31/30 ECSubWrite(tid=16110, reqid=client.4991.0:36605, at_version=31'2309, trim_to=0'0, roll_forward_to=31'2307))", 2024-09-16T04:41:30.312 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-16T04:41:30.270774+0000", 2024-09-16T04:41:30.313 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.0062865769999999998, 2024-09-16T04:41:30.313 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.00675423, 2024-09-16T04:41:30.313 INFO:teuthology.orchestra.run.smithi152.stdout: "continuous": false, 2024-09-16T04:41:30.313 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-16T04:41:30.313 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-16T04:41:30.313 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-16T04:41:30.313 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.313 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-16T04:41:30.313 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.270774+0000", 2024-09-16T04:41:30.313 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.313 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.313 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.313 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-16T04:41:30.313 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.270774+0000", 2024-09-16T04:41:30.314 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.314 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.314 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.314 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-16T04:41:30.314 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.270774+0000", 2024-09-16T04:41:30.314 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 3.5999999999999999e-07 2024-09-16T04:41:30.314 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.314 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.314 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-16T04:41:30.314 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.270794+0000", 2024-09-16T04:41:30.314 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 1.9939e-05 2024-09-16T04:41:30.314 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.314 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.314 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-16T04:41:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.270802+0000", 2024-09-16T04:41:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.7349999999999996e-06 2024-09-16T04:41:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-16T04:41:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.270808+0000", 2024-09-16T04:41:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.4030000000000004e-06 2024-09-16T04:41:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-16T04:41:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.274646+0000", 2024-09-16T04:41:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0038373719999999999 2024-09-16T04:41:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-16T04:41:30.316 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.274655+0000", 2024-09-16T04:41:30.316 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.6409999999999997e-06 2024-09-16T04:41:30.316 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.316 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:30.316 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.316 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.316 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.316 INFO:teuthology.orchestra.run.smithi152.stdout: "description": "MOSDECSubOpWrite(1.4s3 31/30 ECSubWrite(tid=22407, reqid=client.4991.0:36568, at_version=0'0, trim_to=0'0, roll_forward_to=31'2415))", 2024-09-16T04:41:30.316 INFO:teuthology.orchestra.run.smithi152.stdout: "initiated_at": "2024-09-16T04:41:30.271900+0000", 2024-09-16T04:41:30.316 INFO:teuthology.orchestra.run.smithi152.stdout: "age": 0.0051609250000000002, 2024-09-16T04:41:30.316 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0056799550000000004, 2024-09-16T04:41:30.316 INFO:teuthology.orchestra.run.smithi152.stdout: "continuous": false, 2024-09-16T04:41:30.316 INFO:teuthology.orchestra.run.smithi152.stdout: "type_data": { 2024-09-16T04:41:30.316 INFO:teuthology.orchestra.run.smithi152.stdout: "flag_point": "reached pg", 2024-09-16T04:41:30.317 INFO:teuthology.orchestra.run.smithi152.stdout: "events": [ 2024-09-16T04:41:30.317 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.317 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "initiated", 2024-09-16T04:41:30.317 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.271900+0000", 2024-09-16T04:41:30.317 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.317 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.317 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.317 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "header_read", 2024-09-16T04:41:30.317 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.271900+0000", 2024-09-16T04:41:30.317 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0 2024-09-16T04:41:30.317 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.317 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.317 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "throttled", 2024-09-16T04:41:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.271900+0000", 2024-09-16T04:41:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 2.7300000000000002e-07 2024-09-16T04:41:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "all_read", 2024-09-16T04:41:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.271908+0000", 2024-09-16T04:41:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 8.4020000000000002e-06 2024-09-16T04:41:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "dispatched", 2024-09-16T04:41:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.271914+0000", 2024-09-16T04:41:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 6.2049999999999999e-06 2024-09-16T04:41:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.318 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "queued_for_pg", 2024-09-16T04:41:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.271922+0000", 2024-09-16T04:41:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 7.6450000000000005e-06 2024-09-16T04:41:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "reached_pg", 2024-09-16T04:41:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.274173+0000", 2024-09-16T04:41:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 0.0022512029999999998 2024-09-16T04:41:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-09-16T04:41:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: { 2024-09-16T04:41:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: "event": "sub_op_started", 2024-09-16T04:41:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: "time": "2024-09-16T04:41:30.274183+0000", 2024-09-16T04:41:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: "duration": 9.4949999999999997e-06 2024-09-16T04:41:30.319 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.320 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-09-16T04:41:30.320 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.320 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-09-16T04:41:30.320 INFO:teuthology.orchestra.run.smithi152.stdout: ], 2024-09-16T04:41:30.320 INFO:teuthology.orchestra.run.smithi152.stdout: "num_ops": 10 2024-09-16T04:41:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:41:30.321 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.1 to be up 2024-09-16T04:41:30.321 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:41:30.862 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:30.862 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":31,"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","created":"2024-09-16T04:39:47.674616+0000","modified":"2024-09-16T04:41:27.471515+0000","last_up_change":"2024-09-16T04:41:26.169553+0000","last_in_change":"2024-09-16T04:39:49.620271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-16T04:40:10.631384+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"d8b076bc-f4ae-4100-8cdf-78560225fea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":30,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4015439555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6801","nonce":4015439555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":4015439555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4015439555}]},"public_addr":"172.21.15.152:6800/4015439555","cluster_addr":"172.21.15.152:6801/4015439555","heartbeat_back_addr":"172.21.15.152:6803/4015439555","heartbeat_front_addr":"172.21.15.152:6802/4015439555","state":["exists","up"]},{"osd":1,"uuid":"023d4748-7485-4d8b-8594-54a238b67b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":29,"up_thru":29,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1865775950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6813","nonce":1865775950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":1865775950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1865775950}]},"public_addr":"172.21.15.152:6812/1865775950","cluster_addr":"172.21.15.152:6813/1865775950","heartbeat_back_addr":"172.21.15.152:6815/1865775950","heartbeat_front_addr":"172.21.15.152:6814/1865775950","state":["exists","up"]},{"osd":2,"uuid":"c7bfb282-86b4-4bdc-902d-0a0cfbadf5ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2854825795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2854825795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":2854825795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2854825795}]},"public_addr":"172.21.15.152:6804/2854825795","cluster_addr":"172.21.15.152:6805/2854825795","heartbeat_back_addr":"172.21.15.152:6807/2854825795","heartbeat_front_addr":"172.21.15.152:6806/2854825795","state":["exists","up"]},{"osd":3,"uuid":"f82045bc-6fe9-4755-8c25-ea7738b90d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":30,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1506610745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":1506610745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6811","nonce":1506610745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1506610745}]},"public_addr":"172.21.15.152:6808/1506610745","cluster_addr":"172.21.15.152:6809/1506610745","heartbeat_back_addr":"172.21.15.152:6811/1506610745","heartbeat_front_addr":"172.21.15.152:6810/1506610745","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.429729+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.175556+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.243970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.373375+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.0","osds":[2147483647,0,2,3]},{"pgid":"1.1","osds":[3,2147483647,0,2]},{"pgid":"1.2","osds":[0,3,2147483647,2]},{"pgid":"1.3","osds":[2147483647,0,2,3]},{"pgid":"1.4","osds":[3,0,2,2147483647]},{"pgid":"1.5","osds":[2,0,3,2147483647]},{"pgid":"1.6","osds":[3,0,2147483647,2]},{"pgid":"1.7","osds":[2147483647,3,2,0]},{"pgid":"1.8","osds":[2147483647,2,3,0]},{"pgid":"1.9","osds":[2147483647,3,2,0]},{"pgid":"1.a","osds":[2147483647,2,3,0]},{"pgid":"1.b","osds":[2147483647,0,2,3]},{"pgid":"1.c","osds":[3,0,2,2147483647]},{"pgid":"1.d","osds":[2,2147483647,3,0]},{"pgid":"1.e","osds":[2147483647,0,3,2]},{"pgid":"1.f","osds":[3,0,2147483647,2]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:41:30.896 INFO:tasks.ec_lost_unfound.ceph_manager:osd.1 is up 2024-09-16T04:41:30.897 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-16T04:41:30.897 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:31.249 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 37 20 36973 36953 7.80125 2.92969 0.0638258 0.0200153 2024-09-16T04:41:31.504 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:31.504 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:31.529 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-09-16T04:41:30.564775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":293463720,"num_objects":35828,"num_object_clones":0,"num_object_copies":143312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35855,"num_write_kb":286722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35882,"ondisk_log_size":35882,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":864884,"kb_used_data":433256,"kb_used_omap":14,"kb_used_meta":431537,"kb_avail":376622476,"statfs":{"total":386547056640,"available":385661415424,"internally_reserved":0,"allocated":443654144,"data_stored":442975441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":441894449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,7,8,11,2,0,5,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22470656,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2452,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2743,"num_write_kb":21944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.386594"},"pg_stats":[{"pgid":"1.f","version":"31'2222","reported_seq":2261,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.885279+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:27.885279+0000","last_peered":"2024-09-16T04:41:27.885279+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:27.885279+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2222,"log_dups_size":0,"ondisk_log_size":2222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18202624,"num_objects":2222,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2222,"num_write_kb":17776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2251","reported_seq":2312,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.834774+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:28.834774+0000","last_peered":"2024-09-16T04:41:28.834774+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:28.834774+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2251,"log_dups_size":0,"ondisk_log_size":2251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18404620,"num_objects":2247,"num_object_clones":0,"num_object_copies":8988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2249,"num_write_kb":17988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":36}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2258","reported_seq":2327,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.614577+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:28.614577+0000","last_peered":"2024-09-16T04:41:28.614577+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:28.614577+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2258,"log_dups_size":0,"ondisk_log_size":2258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18471558,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2257,"num_write_kb":18046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":28}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2266","reported_seq":2327,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.791392+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:28.791392+0000","last_peered":"2024-09-16T04:41:28.791392+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:28.791392+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2266,"log_dups_size":0,"ondisk_log_size":2266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18527500,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2264,"num_write_kb":18101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":33}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2293","reported_seq":2342,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.926319+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:27.926319+0000","last_peered":"2024-09-16T04:41:27.926319+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:27.926319+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2293,"log_dups_size":0,"ondisk_log_size":2293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18756876,"num_objects":2290,"num_object_clones":0,"num_object_copies":9160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2291,"num_write_kb":18325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2210","reported_seq":2249,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.822165+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:27.822165+0000","last_peered":"2024-09-16T04:41:27.822165+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:27.822165+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2210,"log_dups_size":0,"ondisk_log_size":2210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18104320,"num_objects":2210,"num_object_clones":0,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2210,"num_write_kb":17680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2268","reported_seq":2328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.744367+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:28.744367+0000","last_peered":"2024-09-16T04:41:28.744367+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:28.744367+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2268,"log_dups_size":0,"ondisk_log_size":2268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18527500,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2266,"num_write_kb":18108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":36}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2274","reported_seq":2366,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.805003+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:28.805003+0000","last_peered":"2024-09-16T04:41:28.805003+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:28.805003+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2274,"log_dups_size":0,"ondisk_log_size":2274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18584844,"num_objects":2269,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2272,"num_write_kb":18164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":47}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2353","reported_seq":2405,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.770135+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:27.770135+0000","last_peered":"2024-09-16T04:41:27.770135+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:27.770135+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2353,"log_dups_size":0,"ondisk_log_size":2353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19204632,"num_objects":2345,"num_object_clones":0,"num_object_copies":9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2350,"num_write_kb":18777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2227","reported_seq":2297,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.609308+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:28.609308+0000","last_peered":"2024-09-16T04:41:28.609308+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:28.609308+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2227,"log_dups_size":0,"ondisk_log_size":2227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18224396,"num_objects":2225,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2225,"num_write_kb":17798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":25}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2198","reported_seq":2251,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.873131+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:27.873131+0000","last_peered":"2024-09-16T04:41:27.873131+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:27.873131+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2198,"log_dups_size":0,"ondisk_log_size":2198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17960850,"num_objects":2193,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2195,"num_write_kb":17548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2133","reported_seq":2168,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.877880+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:27.877880+0000","last_peered":"2024-09-16T04:41:27.877880+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:27.877880+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2133,"log_dups_size":0,"ondisk_log_size":2133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17463942,"num_objects":2132,"num_object_clones":0,"num_object_copies":8528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":17055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2256","reported_seq":2325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.511847+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:28.511847+0000","last_peered":"2024-09-16T04:41:28.511847+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:28.511847+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2256,"log_dups_size":0,"ondisk_log_size":2256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18471558,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2255,"num_write_kb":18039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":30}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2245","reported_seq":2292,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.902819+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:27.902819+0000","last_peered":"2024-09-16T04:41:27.902819+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:27.902819+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2245,"log_dups_size":0,"ondisk_log_size":2245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355468,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2243,"num_write_kb":17940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2156","reported_seq":2230,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.556082+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:28.556082+0000","last_peered":"2024-09-16T04:41:28.556082+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:28.556082+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2156,"log_dups_size":0,"ondisk_log_size":2156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17634572,"num_objects":2153,"num_object_clones":0,"num_object_copies":8612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2154,"num_write_kb":17229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2272","reported_seq":2333,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.780049+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:28.780049+0000","last_peered":"2024-09-16T04:41:28.780049+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:28.780049+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2272,"log_dups_size":0,"ondisk_log_size":2272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18568460,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2270,"num_write_kb":18148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":293463720,"num_objects":35828,"num_object_clones":0,"num_object_copies":143312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35855,"num_write_kb":286722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":445079552,"data_stored":445079552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35882,"ondisk_log_size":35882,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":272632,"kb_used_data":142364,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94099208,"statfs":{"total":96636764160,"available":96357588992,"internally_reserved":0,"allocated":145780736,"data_stored":145614463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133361359},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":264876,"kb_used_data":143452,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94106964,"statfs":{"total":96636764160,"available":96365531136,"internally_reserved":0,"allocated":146894848,"data_stored":146724038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,5,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54048,"kb_used_data":4304,"kb_used_omap":4,"kb_used_meta":49723,"kb_avail":94317792,"statfs":{"total":96636764160,"available":96581419008,"internally_reserved":0,"allocated":4407296,"data_stored":4236486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":50917071},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":273328,"kb_used_data":143136,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94098512,"statfs":{"total":96636764160,"available":96356876288,"internally_reserved":0,"allocated":146571264,"data_stored":146400454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":133295822},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":147431424,"data_stored":147431424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4857856,"data_stored":4857856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":147046400,"data_stored":147046400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":145743872,"data_stored":145743872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:31.531 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:32.133 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:32.134 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:32.160 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-09-16T04:41:30.564775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":293463720,"num_objects":35828,"num_object_clones":0,"num_object_copies":143312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35855,"num_write_kb":286722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35882,"ondisk_log_size":35882,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":864884,"kb_used_data":433256,"kb_used_omap":14,"kb_used_meta":431537,"kb_avail":376622476,"statfs":{"total":386547056640,"available":385661415424,"internally_reserved":0,"allocated":443654144,"data_stored":442975441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":441894449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,7,8,11,2,0,5,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22470656,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2452,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2743,"num_write_kb":21944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.386594"},"pg_stats":[{"pgid":"1.f","version":"31'2222","reported_seq":2261,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.885279+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:27.885279+0000","last_peered":"2024-09-16T04:41:27.885279+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:27.885279+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2222,"log_dups_size":0,"ondisk_log_size":2222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18202624,"num_objects":2222,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2222,"num_write_kb":17776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2251","reported_seq":2312,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.834774+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:28.834774+0000","last_peered":"2024-09-16T04:41:28.834774+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:28.834774+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2251,"log_dups_size":0,"ondisk_log_size":2251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18404620,"num_objects":2247,"num_object_clones":0,"num_object_copies":8988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2249,"num_write_kb":17988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":36}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2258","reported_seq":2327,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.614577+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:28.614577+0000","last_peered":"2024-09-16T04:41:28.614577+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:28.614577+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2258,"log_dups_size":0,"ondisk_log_size":2258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18471558,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2257,"num_write_kb":18046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":28}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2266","reported_seq":2327,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.791392+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:28.791392+0000","last_peered":"2024-09-16T04:41:28.791392+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:28.791392+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2266,"log_dups_size":0,"ondisk_log_size":2266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18527500,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2264,"num_write_kb":18101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":33}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2293","reported_seq":2342,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.926319+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:27.926319+0000","last_peered":"2024-09-16T04:41:27.926319+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:27.926319+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2293,"log_dups_size":0,"ondisk_log_size":2293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18756876,"num_objects":2290,"num_object_clones":0,"num_object_copies":9160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2291,"num_write_kb":18325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2210","reported_seq":2249,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.822165+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:27.822165+0000","last_peered":"2024-09-16T04:41:27.822165+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:27.822165+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2210,"log_dups_size":0,"ondisk_log_size":2210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18104320,"num_objects":2210,"num_object_clones":0,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2210,"num_write_kb":17680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2268","reported_seq":2328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.744367+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:28.744367+0000","last_peered":"2024-09-16T04:41:28.744367+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:28.744367+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2268,"log_dups_size":0,"ondisk_log_size":2268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18527500,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2266,"num_write_kb":18108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":36}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2274","reported_seq":2366,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.805003+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:28.805003+0000","last_peered":"2024-09-16T04:41:28.805003+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:28.805003+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2274,"log_dups_size":0,"ondisk_log_size":2274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18584844,"num_objects":2269,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2272,"num_write_kb":18164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":47}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2353","reported_seq":2405,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.770135+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:27.770135+0000","last_peered":"2024-09-16T04:41:27.770135+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:27.770135+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2353,"log_dups_size":0,"ondisk_log_size":2353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19204632,"num_objects":2345,"num_object_clones":0,"num_object_copies":9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2350,"num_write_kb":18777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2227","reported_seq":2297,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.609308+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:28.609308+0000","last_peered":"2024-09-16T04:41:28.609308+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:28.609308+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2227,"log_dups_size":0,"ondisk_log_size":2227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18224396,"num_objects":2225,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2225,"num_write_kb":17798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":25}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2198","reported_seq":2251,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.873131+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:27.873131+0000","last_peered":"2024-09-16T04:41:27.873131+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:27.873131+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2198,"log_dups_size":0,"ondisk_log_size":2198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17960850,"num_objects":2193,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2195,"num_write_kb":17548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2133","reported_seq":2168,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.877880+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:27.877880+0000","last_peered":"2024-09-16T04:41:27.877880+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:27.877880+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2133,"log_dups_size":0,"ondisk_log_size":2133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17463942,"num_objects":2132,"num_object_clones":0,"num_object_copies":8528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":17055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2256","reported_seq":2325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.511847+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:28.511847+0000","last_peered":"2024-09-16T04:41:28.511847+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:28.511847+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2256,"log_dups_size":0,"ondisk_log_size":2256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18471558,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2255,"num_write_kb":18039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":30}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2245","reported_seq":2292,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.902819+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:27.902819+0000","last_peered":"2024-09-16T04:41:27.902819+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:27.902819+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2245,"log_dups_size":0,"ondisk_log_size":2245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355468,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2243,"num_write_kb":17940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2156","reported_seq":2230,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.556082+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:28.556082+0000","last_peered":"2024-09-16T04:41:28.556082+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:28.556082+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2156,"log_dups_size":0,"ondisk_log_size":2156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17634572,"num_objects":2153,"num_object_clones":0,"num_object_copies":8612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2154,"num_write_kb":17229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2272","reported_seq":2333,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.780049+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:28.780049+0000","last_peered":"2024-09-16T04:41:28.780049+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:28.780049+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2272,"log_dups_size":0,"ondisk_log_size":2272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18568460,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2270,"num_write_kb":18148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":293463720,"num_objects":35828,"num_object_clones":0,"num_object_copies":143312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35855,"num_write_kb":286722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":445079552,"data_stored":445079552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35882,"ondisk_log_size":35882,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":272632,"kb_used_data":142364,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94099208,"statfs":{"total":96636764160,"available":96357588992,"internally_reserved":0,"allocated":145780736,"data_stored":145614463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133361359},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":264876,"kb_used_data":143452,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94106964,"statfs":{"total":96636764160,"available":96365531136,"internally_reserved":0,"allocated":146894848,"data_stored":146724038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,5,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54048,"kb_used_data":4304,"kb_used_omap":4,"kb_used_meta":49723,"kb_avail":94317792,"statfs":{"total":96636764160,"available":96581419008,"internally_reserved":0,"allocated":4407296,"data_stored":4236486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":50917071},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":273328,"kb_used_data":143136,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94098512,"statfs":{"total":96636764160,"available":96356876288,"internally_reserved":0,"allocated":146571264,"data_stored":146400454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":133295822},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":147431424,"data_stored":147431424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4857856,"data_stored":4857856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":147046400,"data_stored":147046400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":145743872,"data_stored":145743872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:32.161 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:32.250 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 38 20 37356 37336 7.67433 2.99219 0.0735717 0.0203369 2024-09-16T04:41:32.759 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:32.760 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:32.783 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-09-16T04:41:32.565247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":293463720,"num_objects":35828,"num_object_clones":0,"num_object_copies":143312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35855,"num_write_kb":286722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35882,"ondisk_log_size":35882,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":864884,"kb_used_data":433256,"kb_used_omap":14,"kb_used_meta":431537,"kb_avail":376622476,"statfs":{"total":386547056640,"available":385661415424,"internally_reserved":0,"allocated":443654144,"data_stored":442975441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":441894449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,7,8,11,2,0,5,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22470656,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2452,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2743,"num_write_kb":21944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.387444"},"pg_stats":[{"pgid":"1.f","version":"31'2222","reported_seq":2261,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.885279+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:27.885279+0000","last_peered":"2024-09-16T04:41:27.885279+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:27.885279+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2222,"log_dups_size":0,"ondisk_log_size":2222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18202624,"num_objects":2222,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2222,"num_write_kb":17776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2251","reported_seq":2312,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.834774+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:28.834774+0000","last_peered":"2024-09-16T04:41:28.834774+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:28.834774+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2251,"log_dups_size":0,"ondisk_log_size":2251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18404620,"num_objects":2247,"num_object_clones":0,"num_object_copies":8988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2249,"num_write_kb":17988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":36}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2258","reported_seq":2327,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.614577+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:28.614577+0000","last_peered":"2024-09-16T04:41:28.614577+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:28.614577+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2258,"log_dups_size":0,"ondisk_log_size":2258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18471558,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2257,"num_write_kb":18046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":28}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2266","reported_seq":2327,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.791392+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:28.791392+0000","last_peered":"2024-09-16T04:41:28.791392+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:28.791392+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2266,"log_dups_size":0,"ondisk_log_size":2266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18527500,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2264,"num_write_kb":18101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":33}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2293","reported_seq":2342,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.926319+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:27.926319+0000","last_peered":"2024-09-16T04:41:27.926319+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:27.926319+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2293,"log_dups_size":0,"ondisk_log_size":2293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18756876,"num_objects":2290,"num_object_clones":0,"num_object_copies":9160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2291,"num_write_kb":18325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2210","reported_seq":2249,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.822165+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:27.822165+0000","last_peered":"2024-09-16T04:41:27.822165+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:27.822165+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2210,"log_dups_size":0,"ondisk_log_size":2210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18104320,"num_objects":2210,"num_object_clones":0,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2210,"num_write_kb":17680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2268","reported_seq":2328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.744367+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:28.744367+0000","last_peered":"2024-09-16T04:41:28.744367+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:28.744367+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2268,"log_dups_size":0,"ondisk_log_size":2268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18527500,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2266,"num_write_kb":18108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":36}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2274","reported_seq":2366,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.805003+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:28.805003+0000","last_peered":"2024-09-16T04:41:28.805003+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:28.805003+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2274,"log_dups_size":0,"ondisk_log_size":2274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18584844,"num_objects":2269,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2272,"num_write_kb":18164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":47}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2353","reported_seq":2405,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.770135+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:27.770135+0000","last_peered":"2024-09-16T04:41:27.770135+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:27.770135+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2353,"log_dups_size":0,"ondisk_log_size":2353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19204632,"num_objects":2345,"num_object_clones":0,"num_object_copies":9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2350,"num_write_kb":18777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2227","reported_seq":2297,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.609308+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:28.609308+0000","last_peered":"2024-09-16T04:41:28.609308+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:28.609308+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2227,"log_dups_size":0,"ondisk_log_size":2227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18224396,"num_objects":2225,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2225,"num_write_kb":17798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":25}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2198","reported_seq":2251,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.873131+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:27.873131+0000","last_peered":"2024-09-16T04:41:27.873131+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:27.873131+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2198,"log_dups_size":0,"ondisk_log_size":2198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17960850,"num_objects":2193,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2195,"num_write_kb":17548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2133","reported_seq":2168,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.877880+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:27.877880+0000","last_peered":"2024-09-16T04:41:27.877880+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:27.877880+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2133,"log_dups_size":0,"ondisk_log_size":2133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17463942,"num_objects":2132,"num_object_clones":0,"num_object_copies":8528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":17055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2256","reported_seq":2325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.511847+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:28.511847+0000","last_peered":"2024-09-16T04:41:28.511847+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:28.511847+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2256,"log_dups_size":0,"ondisk_log_size":2256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18471558,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2255,"num_write_kb":18039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":30}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2245","reported_seq":2292,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.902819+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:27.902819+0000","last_peered":"2024-09-16T04:41:27.902819+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:27.902819+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2245,"log_dups_size":0,"ondisk_log_size":2245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355468,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2243,"num_write_kb":17940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2156","reported_seq":2230,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.556082+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:28.556082+0000","last_peered":"2024-09-16T04:41:28.556082+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:28.556082+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2156,"log_dups_size":0,"ondisk_log_size":2156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17634572,"num_objects":2153,"num_object_clones":0,"num_object_copies":8612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2154,"num_write_kb":17229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2272","reported_seq":2333,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.780049+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:28.780049+0000","last_peered":"2024-09-16T04:41:28.780049+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:28.780049+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2272,"log_dups_size":0,"ondisk_log_size":2272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18568460,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2270,"num_write_kb":18148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":293463720,"num_objects":35828,"num_object_clones":0,"num_object_copies":143312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35855,"num_write_kb":286722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":445079552,"data_stored":445079552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35882,"ondisk_log_size":35882,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":272632,"kb_used_data":142364,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94099208,"statfs":{"total":96636764160,"available":96357588992,"internally_reserved":0,"allocated":145780736,"data_stored":145614463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133361359},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":264876,"kb_used_data":143452,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94106964,"statfs":{"total":96636764160,"available":96365531136,"internally_reserved":0,"allocated":146894848,"data_stored":146724038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,5,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54048,"kb_used_data":4304,"kb_used_omap":4,"kb_used_meta":49723,"kb_avail":94317792,"statfs":{"total":96636764160,"available":96581419008,"internally_reserved":0,"allocated":4407296,"data_stored":4236486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":50917071},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":273328,"kb_used_data":143136,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94098512,"statfs":{"total":96636764160,"available":96356876288,"internally_reserved":0,"allocated":146571264,"data_stored":146400454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":133295822},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":147431424,"data_stored":147431424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4857856,"data_stored":4857856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":147046400,"data_stored":147046400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":145743872,"data_stored":145743872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:32.784 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:41:32.784 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2222", 'reported_seq': 2261, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:27.885279+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:41:27.885279+0000', 'last_peered': '2024-09-16T04:41:27.885279+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:41:27.885279+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2222, 'log_dups_size': 0, 'ondisk_log_size': 2222, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18202624, 'num_objects': 2222, 'num_object_clones': 0, 'num_object_copies': 8888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2222, 'num_write_kb': 17776, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:32.784 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:41:32.784 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2251", 'reported_seq': 2312, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:28.834774+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:41:28.834774+0000', 'last_peered': '2024-09-16T04:41:28.834774+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:41:28.834774+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2251, 'log_dups_size': 0, 'ondisk_log_size': 2251, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18404620, 'num_objects': 2247, 'num_object_clones': 0, 'num_object_copies': 8988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2247, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2249, 'num_write_kb': 17988, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 36}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:32.785 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:41:32.785 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2258", 'reported_seq': 2327, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:28.614577+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:41:28.614577+0000', 'last_peered': '2024-09-16T04:41:28.614577+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:41:28.614577+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2258, 'log_dups_size': 0, 'ondisk_log_size': 2258, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18471558, 'num_objects': 2255, 'num_object_clones': 0, 'num_object_copies': 9020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2257, 'num_write_kb': 18046, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 28}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:32.785 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:41:32.785 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2266", 'reported_seq': 2327, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:28.791392+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:41:28.791392+0000', 'last_peered': '2024-09-16T04:41:28.791392+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:41:28.791392+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2266, 'log_dups_size': 0, 'ondisk_log_size': 2266, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18527500, 'num_objects': 2262, 'num_object_clones': 0, 'num_object_copies': 9048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2264, 'num_write_kb': 18101, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 33}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:32.785 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:41:32.785 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2293", 'reported_seq': 2342, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:27.926319+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:41:27.926319+0000', 'last_peered': '2024-09-16T04:41:27.926319+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:41:27.926319+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2293, 'log_dups_size': 0, 'ondisk_log_size': 2293, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18756876, 'num_objects': 2290, 'num_object_clones': 0, 'num_object_copies': 9160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2291, 'num_write_kb': 18325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 7}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:32.785 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:41:32.785 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2210", 'reported_seq': 2249, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:27.822165+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:41:27.822165+0000', 'last_peered': '2024-09-16T04:41:27.822165+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:41:27.822165+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2210, 'log_dups_size': 0, 'ondisk_log_size': 2210, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18104320, 'num_objects': 2210, 'num_object_clones': 0, 'num_object_copies': 8840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2210, 'num_write_kb': 17680, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:32.785 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:41:32.785 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2268", 'reported_seq': 2328, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:28.744367+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:41:28.744367+0000', 'last_peered': '2024-09-16T04:41:28.744367+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:41:28.744367+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2268, 'log_dups_size': 0, 'ondisk_log_size': 2268, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18527500, 'num_objects': 2262, 'num_object_clones': 0, 'num_object_copies': 9048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2266, 'num_write_kb': 18108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 36}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:32.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:41:32.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2274", 'reported_seq': 2366, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:28.805003+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:41:28.805003+0000', 'last_peered': '2024-09-16T04:41:28.805003+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:41:28.805003+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2274, 'log_dups_size': 0, 'ondisk_log_size': 2274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18584844, 'num_objects': 2269, 'num_object_clones': 0, 'num_object_copies': 9076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2272, 'num_write_kb': 18164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 131072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 47}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:32.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:41:32.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2353", 'reported_seq': 2405, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:27.770135+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:41:27.770135+0000', 'last_peered': '2024-09-16T04:41:27.770135+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:41:27.770135+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2353, 'log_dups_size': 0, 'ondisk_log_size': 2353, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19204632, 'num_objects': 2345, 'num_object_clones': 0, 'num_object_copies': 9380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2350, 'num_write_kb': 18777, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 5}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:32.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:41:32.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2227", 'reported_seq': 2297, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:28.609308+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:41:28.609308+0000', 'last_peered': '2024-09-16T04:41:28.609308+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:41:28.609308+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2227, 'log_dups_size': 0, 'ondisk_log_size': 2227, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18224396, 'num_objects': 2225, 'num_object_clones': 0, 'num_object_copies': 8900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2225, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2225, 'num_write_kb': 17798, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 25}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:32.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:41:32.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2198", 'reported_seq': 2251, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:27.873131+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:41:27.873131+0000', 'last_peered': '2024-09-16T04:41:27.873131+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:41:27.873131+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2198, 'log_dups_size': 0, 'ondisk_log_size': 2198, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17960850, 'num_objects': 2193, 'num_object_clones': 0, 'num_object_copies': 8772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2195, 'num_write_kb': 17548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:32.786 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:41:32.786 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2133", 'reported_seq': 2168, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:27.877880+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:41:27.877880+0000', 'last_peered': '2024-09-16T04:41:27.877880+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:41:27.877880+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2133, 'log_dups_size': 0, 'ondisk_log_size': 2133, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17463942, 'num_objects': 2132, 'num_object_clones': 0, 'num_object_copies': 8528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2132, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2132, 'num_write_kb': 17055, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:32.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:41:32.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2256", 'reported_seq': 2325, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:28.511847+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:41:28.511847+0000', 'last_peered': '2024-09-16T04:41:28.511847+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:41:28.511847+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2256, 'log_dups_size': 0, 'ondisk_log_size': 2256, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18471558, 'num_objects': 2255, 'num_object_clones': 0, 'num_object_copies': 9020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2255, 'num_write_kb': 18039, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 30}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:32.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:41:32.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2245", 'reported_seq': 2292, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:27.902819+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:41:27.902819+0000', 'last_peered': '2024-09-16T04:41:27.902819+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:41:27.902819+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2245, 'log_dups_size': 0, 'ondisk_log_size': 2245, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18355468, 'num_objects': 2241, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2243, 'num_write_kb': 17940, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 7}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:32.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:41:32.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2156", 'reported_seq': 2230, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:28.556082+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:41:28.556082+0000', 'last_peered': '2024-09-16T04:41:28.556082+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:41:28.556082+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2156, 'log_dups_size': 0, 'ondisk_log_size': 2156, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17634572, 'num_objects': 2153, 'num_object_clones': 0, 'num_object_copies': 8612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2154, 'num_write_kb': 17229, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 21}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:32.787 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:41:32.787 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2272", 'reported_seq': 2333, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:28.780049+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:41:28.780049+0000', 'last_peered': '2024-09-16T04:41:28.780049+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:41:28.780049+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2272, 'log_dups_size': 0, 'ondisk_log_size': 2272, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18568460, 'num_objects': 2267, 'num_object_clones': 0, 'num_object_copies': 9068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2267, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2270, 'num_write_kb': 18148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 28}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:32.788 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:41:33.252 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 39 20 37743 37723 7.55472 3.02344 0.0618563 0.0206657 2024-09-16T04:41:33.561 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:33.561 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35537/143312 objects degraded (24.797%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":105,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":35828,"data_bytes":293463720,"bytes_used":885641216,"bytes_avail":385661415424,"bytes_total":386547056640,"degraded_objects":35537,"degraded_total":143312,"degraded_ratio":0.24796946522273081,"recovering_objects_per_sec":2,"recovering_bytes_per_sec":20520,"recovering_keys_per_sec":0,"num_objects_recovered":16,"num_bytes_recovered":131072,"num_keys_recovered":0,"write_bytes_sec":3517941,"read_op_per_sec":0,"write_op_per_sec":429},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-09-16T04:41:00.553854+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:41:33.593 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:41:33.594 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:34.182 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:34.183 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:34.206 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-09-16T04:41:32.565247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":293463720,"num_objects":35828,"num_object_clones":0,"num_object_copies":143312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35855,"num_write_kb":286722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35882,"ondisk_log_size":35882,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":864884,"kb_used_data":433256,"kb_used_omap":14,"kb_used_meta":431537,"kb_avail":376622476,"statfs":{"total":386547056640,"available":385661415424,"internally_reserved":0,"allocated":443654144,"data_stored":442975441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":441894449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,7,8,11,2,0,5,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22470656,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2452,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2743,"num_write_kb":21944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.387444"},"pg_stats":[{"pgid":"1.f","version":"31'2222","reported_seq":2261,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.885279+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:27.885279+0000","last_peered":"2024-09-16T04:41:27.885279+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:27.885279+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2222,"log_dups_size":0,"ondisk_log_size":2222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18202624,"num_objects":2222,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2222,"num_write_kb":17776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2251","reported_seq":2312,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.834774+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:28.834774+0000","last_peered":"2024-09-16T04:41:28.834774+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:28.834774+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2251,"log_dups_size":0,"ondisk_log_size":2251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18404620,"num_objects":2247,"num_object_clones":0,"num_object_copies":8988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2249,"num_write_kb":17988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":36}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2258","reported_seq":2327,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.614577+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:28.614577+0000","last_peered":"2024-09-16T04:41:28.614577+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:28.614577+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2258,"log_dups_size":0,"ondisk_log_size":2258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18471558,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2257,"num_write_kb":18046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":28}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2266","reported_seq":2327,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.791392+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:28.791392+0000","last_peered":"2024-09-16T04:41:28.791392+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:28.791392+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2266,"log_dups_size":0,"ondisk_log_size":2266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18527500,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2264,"num_write_kb":18101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":33}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2293","reported_seq":2342,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.926319+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:27.926319+0000","last_peered":"2024-09-16T04:41:27.926319+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:27.926319+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2293,"log_dups_size":0,"ondisk_log_size":2293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18756876,"num_objects":2290,"num_object_clones":0,"num_object_copies":9160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2291,"num_write_kb":18325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2210","reported_seq":2249,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.822165+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:27.822165+0000","last_peered":"2024-09-16T04:41:27.822165+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:27.822165+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2210,"log_dups_size":0,"ondisk_log_size":2210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18104320,"num_objects":2210,"num_object_clones":0,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2210,"num_write_kb":17680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2268","reported_seq":2328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.744367+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:28.744367+0000","last_peered":"2024-09-16T04:41:28.744367+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:28.744367+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2268,"log_dups_size":0,"ondisk_log_size":2268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18527500,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2266,"num_write_kb":18108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":36}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2274","reported_seq":2366,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.805003+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:28.805003+0000","last_peered":"2024-09-16T04:41:28.805003+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:28.805003+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2274,"log_dups_size":0,"ondisk_log_size":2274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18584844,"num_objects":2269,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2272,"num_write_kb":18164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":47}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2353","reported_seq":2405,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.770135+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:27.770135+0000","last_peered":"2024-09-16T04:41:27.770135+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:27.770135+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2353,"log_dups_size":0,"ondisk_log_size":2353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19204632,"num_objects":2345,"num_object_clones":0,"num_object_copies":9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2350,"num_write_kb":18777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2227","reported_seq":2297,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.609308+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:28.609308+0000","last_peered":"2024-09-16T04:41:28.609308+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:28.609308+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2227,"log_dups_size":0,"ondisk_log_size":2227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18224396,"num_objects":2225,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2225,"num_write_kb":17798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":25}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2198","reported_seq":2251,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.873131+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:27.873131+0000","last_peered":"2024-09-16T04:41:27.873131+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:27.873131+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2198,"log_dups_size":0,"ondisk_log_size":2198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17960850,"num_objects":2193,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2195,"num_write_kb":17548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2133","reported_seq":2168,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.877880+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:27.877880+0000","last_peered":"2024-09-16T04:41:27.877880+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:27.877880+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2133,"log_dups_size":0,"ondisk_log_size":2133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17463942,"num_objects":2132,"num_object_clones":0,"num_object_copies":8528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":17055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2256","reported_seq":2325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.511847+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:28.511847+0000","last_peered":"2024-09-16T04:41:28.511847+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:28.511847+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2256,"log_dups_size":0,"ondisk_log_size":2256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18471558,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2255,"num_write_kb":18039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":30}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2245","reported_seq":2292,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:27.902819+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:27.902819+0000","last_peered":"2024-09-16T04:41:27.902819+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:27.902819+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2245,"log_dups_size":0,"ondisk_log_size":2245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355468,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2243,"num_write_kb":17940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2156","reported_seq":2230,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.556082+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:28.556082+0000","last_peered":"2024-09-16T04:41:28.556082+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:28.556082+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2156,"log_dups_size":0,"ondisk_log_size":2156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17634572,"num_objects":2153,"num_object_clones":0,"num_object_copies":8612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2154,"num_write_kb":17229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2272","reported_seq":2333,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:28.780049+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:28.780049+0000","last_peered":"2024-09-16T04:41:28.780049+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:28.780049+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2272,"log_dups_size":0,"ondisk_log_size":2272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18568460,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2270,"num_write_kb":18148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":293463720,"num_objects":35828,"num_object_clones":0,"num_object_copies":143312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35855,"num_write_kb":286722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":445079552,"data_stored":445079552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35882,"ondisk_log_size":35882,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":272632,"kb_used_data":142364,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94099208,"statfs":{"total":96636764160,"available":96357588992,"internally_reserved":0,"allocated":145780736,"data_stored":145614463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133361359},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":264876,"kb_used_data":143452,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94106964,"statfs":{"total":96636764160,"available":96365531136,"internally_reserved":0,"allocated":146894848,"data_stored":146724038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,5,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54048,"kb_used_data":4304,"kb_used_omap":4,"kb_used_meta":49723,"kb_avail":94317792,"statfs":{"total":96636764160,"available":96581419008,"internally_reserved":0,"allocated":4407296,"data_stored":4236486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":50917071},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":273328,"kb_used_data":143136,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94098512,"statfs":{"total":96636764160,"available":96356876288,"internally_reserved":0,"allocated":146571264,"data_stored":146400454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":133295822},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":147431424,"data_stored":147431424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4857856,"data_stored":4857856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":147046400,"data_stored":147046400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":145743872,"data_stored":145743872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:34.254 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:2024-09-16T04:41:34.253427+0000 min lat: 0.00269411 max lat: 1.73107 avg lat: 0.0209855 2024-09-16T04:41:34.254 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T04:41:34.254 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 40 20 38122 38102 7.43966 2.96094 0.0415782 0.0209855 2024-09-16T04:41:35.254 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 41 20 38556 38536 7.34091 3.39062 0.0555951 0.0212665 2024-09-16T04:41:36.255 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 42 20 38948 38928 7.23892 3.0625 0.0348671 0.0215666 2024-09-16T04:41:37.207 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:37.255 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 43 20 39374 39354 7.14799 3.32812 0.0465804 0.0218369 2024-09-16T04:41:37.811 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:37.812 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:37.841 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-16T04:41:36.566177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":309716648,"num_objects":37812,"num_object_clones":0,"num_object_copies":151248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37839,"num_write_kb":302594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37866,"ondisk_log_size":37866,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":942088,"kb_used_data":464648,"kb_used_omap":14,"kb_used_meta":477361,"kb_avail":376545272,"statfs":{"total":386547056640,"available":385582358528,"internally_reserved":0,"allocated":475799552,"data_stored":475116312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":488818225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,6,9,9,7,15,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38723584,"num_objects":4727,"num_object_clones":0,"num_object_copies":18908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4727,"num_write_kb":37816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005767"},"pg_stats":[{"pgid":"1.f","version":"31'2343","reported_seq":2382,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:32.751403+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:32.751403+0000","last_peered":"2024-09-16T04:41:32.751403+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:32.751403+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2343,"log_dups_size":0,"ondisk_log_size":2343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19193856,"num_objects":2343,"num_object_clones":0,"num_object_copies":9372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2343,"num_write_kb":18744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2370","reported_seq":2431,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.853553+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:33.853553+0000","last_peered":"2024-09-16T04:41:33.853553+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:33.853553+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2370,"log_dups_size":0,"ondisk_log_size":2370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379468,"num_objects":2366,"num_object_clones":0,"num_object_copies":9464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2368,"num_write_kb":18940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2380","reported_seq":2450,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.571198+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:33.571198+0000","last_peered":"2024-09-16T04:41:33.571198+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:33.571198+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2380,"log_dups_size":0,"ondisk_log_size":2380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19470982,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2379,"num_write_kb":19022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":150}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2395","reported_seq":2456,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.807234+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:33.807234+0000","last_peered":"2024-09-16T04:41:33.807234+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:33.807234+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2395,"log_dups_size":0,"ondisk_log_size":2395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19584268,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2393,"num_write_kb":19133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2420","reported_seq":2471,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:32.825428+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:32.825428+0000","last_peered":"2024-09-16T04:41:32.825428+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:32.825428+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2420,"log_dups_size":0,"ondisk_log_size":2420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19797260,"num_objects":2417,"num_object_clones":0,"num_object_copies":9668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2418,"num_write_kb":19341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":134}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2329","reported_seq":2368,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:32.852042+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:32.852042+0000","last_peered":"2024-09-16T04:41:32.852042+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:32.852042+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2329,"log_dups_size":0,"ondisk_log_size":2329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19079168,"num_objects":2329,"num_object_clones":0,"num_object_copies":9316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2329,"num_write_kb":18632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":121}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2419","reported_seq":2479,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.707357+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:33.707357+0000","last_peered":"2024-09-16T04:41:33.707357+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:33.707357+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2419,"log_dups_size":0,"ondisk_log_size":2419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19764492,"num_objects":2413,"num_object_clones":0,"num_object_copies":9652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2417,"num_write_kb":19316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":187}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2402","reported_seq":2618,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.844364+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:33.844364+0000","last_peered":"2024-09-16T04:41:33.844364+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:33.844364+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2402,"log_dups_size":0,"ondisk_log_size":2402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19633420,"num_objects":2397,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2400,"num_write_kb":19188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":237}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2476","reported_seq":2528,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:32.894233+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:32.894233+0000","last_peered":"2024-09-16T04:41:32.894233+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:32.894233+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2476,"log_dups_size":0,"ondisk_log_size":2476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20212248,"num_objects":2468,"num_object_clones":0,"num_object_copies":9872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2473,"num_write_kb":19761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2361","reported_seq":2434,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.642965+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:33.642965+0000","last_peered":"2024-09-16T04:41:33.642965+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:33.642965+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2361,"log_dups_size":0,"ondisk_log_size":2361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19322124,"num_objects":2359,"num_object_clones":0,"num_object_copies":9436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2359,"num_write_kb":18870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":159}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2316","reported_seq":2369,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:32.792576+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:32.792576+0000","last_peered":"2024-09-16T04:41:32.792576+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:32.792576+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18927506,"num_objects":2311,"num_object_clones":0,"num_object_copies":9244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2313,"num_write_kb":18492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2259","reported_seq":2294,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:32.900236+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:32.900236+0000","last_peered":"2024-09-16T04:41:32.900236+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:32.900236+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2259,"log_dups_size":0,"ondisk_log_size":2259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18496134,"num_objects":2258,"num_object_clones":0,"num_object_copies":9032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2258,"num_write_kb":18063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":130}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2383","reported_seq":2452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.664273+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:33.664273+0000","last_peered":"2024-09-16T04:41:33.664273+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:33.664273+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2383,"log_dups_size":0,"ondisk_log_size":2383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19511942,"num_objects":2382,"num_object_clones":0,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2382,"num_write_kb":19055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":157}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2373","reported_seq":2421,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:32.916702+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:32.916702+0000","last_peered":"2024-09-16T04:41:32.916702+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:32.916702+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2373,"log_dups_size":0,"ondisk_log_size":2373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19404044,"num_objects":2369,"num_object_clones":0,"num_object_copies":9476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2371,"num_write_kb":18964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":135}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2256","reported_seq":2330,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.533792+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:33.533792+0000","last_peered":"2024-09-16T04:41:33.533792+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:33.533792+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2256,"log_dups_size":0,"ondisk_log_size":2256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18453772,"num_objects":2253,"num_object_clones":0,"num_object_copies":9012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2254,"num_write_kb":18029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":121}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2384","reported_seq":2444,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.753893+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:33.753893+0000","last_peered":"2024-09-16T04:41:33.753893+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:33.753893+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2384,"log_dups_size":0,"ondisk_log_size":2384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19485964,"num_objects":2379,"num_object_clones":0,"num_object_copies":9516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2382,"num_write_kb":19044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":309716648,"num_objects":37812,"num_object_clones":0,"num_object_copies":151248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37839,"num_write_kb":302594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":478097408,"data_stored":478097408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37866,"ondisk_log_size":37866,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":285408,"kb_used_data":149584,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94086432,"statfs":{"total":96636764160,"available":96344506368,"internally_reserved":0,"allocated":153174016,"data_stored":153003206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":139062991},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299176,"kb_used_data":151384,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94072664,"statfs":{"total":96636764160,"available":96330407936,"internally_reserved":0,"allocated":155017216,"data_stored":154846406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66328,"kb_used_data":12424,"kb_used_omap":4,"kb_used_meta":53883,"kb_avail":94305512,"statfs":{"total":96636764160,"available":96568844288,"internally_reserved":0,"allocated":12722176,"data_stored":12551366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":55176911},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,2,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291176,"kb_used_data":151256,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94080664,"statfs":{"total":96636764160,"available":96338599936,"internally_reserved":0,"allocated":154886144,"data_stored":154715334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":143257294},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,3,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":155549696,"data_stored":155549696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13201408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155230208,"data_stored":155230208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":154116096,"data_stored":154116096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:37.843 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:38.256 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 44 20 39753 39733 7.05264 2.96094 0.0507444 0.0221398 2024-09-16T04:41:38.423 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:38.423 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:38.448 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-16T04:41:36.566177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":309716648,"num_objects":37812,"num_object_clones":0,"num_object_copies":151248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37839,"num_write_kb":302594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37866,"ondisk_log_size":37866,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":942088,"kb_used_data":464648,"kb_used_omap":14,"kb_used_meta":477361,"kb_avail":376545272,"statfs":{"total":386547056640,"available":385582358528,"internally_reserved":0,"allocated":475799552,"data_stored":475116312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":488818225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,6,9,9,7,15,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38723584,"num_objects":4727,"num_object_clones":0,"num_object_copies":18908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4727,"num_write_kb":37816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005767"},"pg_stats":[{"pgid":"1.f","version":"31'2343","reported_seq":2382,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:32.751403+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:32.751403+0000","last_peered":"2024-09-16T04:41:32.751403+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:32.751403+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2343,"log_dups_size":0,"ondisk_log_size":2343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19193856,"num_objects":2343,"num_object_clones":0,"num_object_copies":9372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2343,"num_write_kb":18744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2370","reported_seq":2431,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.853553+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:33.853553+0000","last_peered":"2024-09-16T04:41:33.853553+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:33.853553+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2370,"log_dups_size":0,"ondisk_log_size":2370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379468,"num_objects":2366,"num_object_clones":0,"num_object_copies":9464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2368,"num_write_kb":18940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2380","reported_seq":2450,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.571198+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:33.571198+0000","last_peered":"2024-09-16T04:41:33.571198+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:33.571198+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2380,"log_dups_size":0,"ondisk_log_size":2380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19470982,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2379,"num_write_kb":19022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":150}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2395","reported_seq":2456,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.807234+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:33.807234+0000","last_peered":"2024-09-16T04:41:33.807234+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:33.807234+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2395,"log_dups_size":0,"ondisk_log_size":2395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19584268,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2393,"num_write_kb":19133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2420","reported_seq":2471,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:32.825428+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:32.825428+0000","last_peered":"2024-09-16T04:41:32.825428+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:32.825428+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2420,"log_dups_size":0,"ondisk_log_size":2420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19797260,"num_objects":2417,"num_object_clones":0,"num_object_copies":9668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2418,"num_write_kb":19341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":134}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2329","reported_seq":2368,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:32.852042+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:32.852042+0000","last_peered":"2024-09-16T04:41:32.852042+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:32.852042+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2329,"log_dups_size":0,"ondisk_log_size":2329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19079168,"num_objects":2329,"num_object_clones":0,"num_object_copies":9316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2329,"num_write_kb":18632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":121}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2419","reported_seq":2479,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.707357+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:33.707357+0000","last_peered":"2024-09-16T04:41:33.707357+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:33.707357+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2419,"log_dups_size":0,"ondisk_log_size":2419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19764492,"num_objects":2413,"num_object_clones":0,"num_object_copies":9652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2417,"num_write_kb":19316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":187}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2402","reported_seq":2618,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.844364+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:33.844364+0000","last_peered":"2024-09-16T04:41:33.844364+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:33.844364+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2402,"log_dups_size":0,"ondisk_log_size":2402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19633420,"num_objects":2397,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2400,"num_write_kb":19188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":237}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2476","reported_seq":2528,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:32.894233+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:32.894233+0000","last_peered":"2024-09-16T04:41:32.894233+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:32.894233+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2476,"log_dups_size":0,"ondisk_log_size":2476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20212248,"num_objects":2468,"num_object_clones":0,"num_object_copies":9872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2473,"num_write_kb":19761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2361","reported_seq":2434,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.642965+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:33.642965+0000","last_peered":"2024-09-16T04:41:33.642965+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:33.642965+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2361,"log_dups_size":0,"ondisk_log_size":2361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19322124,"num_objects":2359,"num_object_clones":0,"num_object_copies":9436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2359,"num_write_kb":18870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":159}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2316","reported_seq":2369,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:32.792576+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:32.792576+0000","last_peered":"2024-09-16T04:41:32.792576+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:32.792576+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18927506,"num_objects":2311,"num_object_clones":0,"num_object_copies":9244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2313,"num_write_kb":18492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2259","reported_seq":2294,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:32.900236+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:32.900236+0000","last_peered":"2024-09-16T04:41:32.900236+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:32.900236+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2259,"log_dups_size":0,"ondisk_log_size":2259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18496134,"num_objects":2258,"num_object_clones":0,"num_object_copies":9032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2258,"num_write_kb":18063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":130}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2383","reported_seq":2452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.664273+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:33.664273+0000","last_peered":"2024-09-16T04:41:33.664273+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:33.664273+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2383,"log_dups_size":0,"ondisk_log_size":2383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19511942,"num_objects":2382,"num_object_clones":0,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2382,"num_write_kb":19055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":157}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2373","reported_seq":2421,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:32.916702+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:32.916702+0000","last_peered":"2024-09-16T04:41:32.916702+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:32.916702+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2373,"log_dups_size":0,"ondisk_log_size":2373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19404044,"num_objects":2369,"num_object_clones":0,"num_object_copies":9476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2371,"num_write_kb":18964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":135}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2256","reported_seq":2330,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.533792+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:33.533792+0000","last_peered":"2024-09-16T04:41:33.533792+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:33.533792+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2256,"log_dups_size":0,"ondisk_log_size":2256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18453772,"num_objects":2253,"num_object_clones":0,"num_object_copies":9012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2254,"num_write_kb":18029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":121}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2384","reported_seq":2444,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.753893+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:33.753893+0000","last_peered":"2024-09-16T04:41:33.753893+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:33.753893+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2384,"log_dups_size":0,"ondisk_log_size":2384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19485964,"num_objects":2379,"num_object_clones":0,"num_object_copies":9516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2382,"num_write_kb":19044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":309716648,"num_objects":37812,"num_object_clones":0,"num_object_copies":151248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37839,"num_write_kb":302594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":478097408,"data_stored":478097408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37866,"ondisk_log_size":37866,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":285408,"kb_used_data":149584,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94086432,"statfs":{"total":96636764160,"available":96344506368,"internally_reserved":0,"allocated":153174016,"data_stored":153003206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":139062991},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299176,"kb_used_data":151384,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94072664,"statfs":{"total":96636764160,"available":96330407936,"internally_reserved":0,"allocated":155017216,"data_stored":154846406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66328,"kb_used_data":12424,"kb_used_omap":4,"kb_used_meta":53883,"kb_avail":94305512,"statfs":{"total":96636764160,"available":96568844288,"internally_reserved":0,"allocated":12722176,"data_stored":12551366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":55176911},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,2,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291176,"kb_used_data":151256,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94080664,"statfs":{"total":96636764160,"available":96338599936,"internally_reserved":0,"allocated":154886144,"data_stored":154715334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":143257294},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,3,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":155549696,"data_stored":155549696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13201408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155230208,"data_stored":155230208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":154116096,"data_stored":154116096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:38.449 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:41:38.449 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2343", 'reported_seq': 2382, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:32.751403+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:41:32.751403+0000', 'last_peered': '2024-09-16T04:41:32.751403+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:41:32.751403+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2343, 'log_dups_size': 0, 'ondisk_log_size': 2343, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19193856, 'num_objects': 2343, 'num_object_clones': 0, 'num_object_copies': 9372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2343, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2343, 'num_write_kb': 18744, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 124}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:38.450 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:41:38.450 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2370", 'reported_seq': 2431, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:33.853553+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:41:33.853553+0000', 'last_peered': '2024-09-16T04:41:33.853553+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:41:33.853553+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2370, 'log_dups_size': 0, 'ondisk_log_size': 2370, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19379468, 'num_objects': 2366, 'num_object_clones': 0, 'num_object_copies': 9464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2368, 'num_write_kb': 18940, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 155}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:38.450 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:41:38.450 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2380", 'reported_seq': 2450, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:33.571198+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:41:33.571198+0000', 'last_peered': '2024-09-16T04:41:33.571198+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:41:33.571198+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2380, 'log_dups_size': 0, 'ondisk_log_size': 2380, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19470982, 'num_objects': 2377, 'num_object_clones': 0, 'num_object_copies': 9508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2379, 'num_write_kb': 19022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 150}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:38.450 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:41:38.450 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2395", 'reported_seq': 2456, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:33.807234+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:41:33.807234+0000', 'last_peered': '2024-09-16T04:41:33.807234+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:41:33.807234+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2395, 'log_dups_size': 0, 'ondisk_log_size': 2395, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19584268, 'num_objects': 2391, 'num_object_clones': 0, 'num_object_copies': 9564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2393, 'num_write_kb': 19133, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 162}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:38.450 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:41:38.451 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2420", 'reported_seq': 2471, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:32.825428+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:41:32.825428+0000', 'last_peered': '2024-09-16T04:41:32.825428+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:41:32.825428+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2420, 'log_dups_size': 0, 'ondisk_log_size': 2420, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19797260, 'num_objects': 2417, 'num_object_clones': 0, 'num_object_copies': 9668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2417, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2418, 'num_write_kb': 19341, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 134}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:38.451 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:41:38.451 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2329", 'reported_seq': 2368, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:32.852042+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:41:32.852042+0000', 'last_peered': '2024-09-16T04:41:32.852042+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:41:32.852042+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2329, 'log_dups_size': 0, 'ondisk_log_size': 2329, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19079168, 'num_objects': 2329, 'num_object_clones': 0, 'num_object_copies': 9316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2329, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2329, 'num_write_kb': 18632, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 121}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:38.451 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:41:38.451 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2419", 'reported_seq': 2479, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:33.707357+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:41:33.707357+0000', 'last_peered': '2024-09-16T04:41:33.707357+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:41:33.707357+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2419, 'log_dups_size': 0, 'ondisk_log_size': 2419, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19764492, 'num_objects': 2413, 'num_object_clones': 0, 'num_object_copies': 9652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2413, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2417, 'num_write_kb': 19316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 187}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:38.451 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:41:38.451 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2402", 'reported_seq': 2618, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:33.844364+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:41:33.844364+0000', 'last_peered': '2024-09-16T04:41:33.844364+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:41:33.844364+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2402, 'log_dups_size': 0, 'ondisk_log_size': 2402, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19633420, 'num_objects': 2397, 'num_object_clones': 0, 'num_object_copies': 9588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2400, 'num_write_kb': 19188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 80, 'num_bytes_recovered': 638976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:38.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:41:38.452 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2476", 'reported_seq': 2528, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:32.894233+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:41:32.894233+0000', 'last_peered': '2024-09-16T04:41:32.894233+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:41:32.894233+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2476, 'log_dups_size': 0, 'ondisk_log_size': 2476, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20212248, 'num_objects': 2468, 'num_object_clones': 0, 'num_object_copies': 9872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2468, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2473, 'num_write_kb': 19761, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 128}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:38.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:41:38.452 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2361", 'reported_seq': 2434, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:33.642965+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:41:33.642965+0000', 'last_peered': '2024-09-16T04:41:33.642965+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:41:33.642965+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2361, 'log_dups_size': 0, 'ondisk_log_size': 2361, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19322124, 'num_objects': 2359, 'num_object_clones': 0, 'num_object_copies': 9436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2359, 'num_write_kb': 18870, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 159}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:38.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:41:38.452 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2316", 'reported_seq': 2369, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:32.792576+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:41:32.792576+0000', 'last_peered': '2024-09-16T04:41:32.792576+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:41:32.792576+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2316, 'log_dups_size': 0, 'ondisk_log_size': 2316, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18927506, 'num_objects': 2311, 'num_object_clones': 0, 'num_object_copies': 9244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2311, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2313, 'num_write_kb': 18492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 122}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:38.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:41:38.452 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2259", 'reported_seq': 2294, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:32.900236+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:41:32.900236+0000', 'last_peered': '2024-09-16T04:41:32.900236+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:41:32.900236+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2259, 'log_dups_size': 0, 'ondisk_log_size': 2259, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18496134, 'num_objects': 2258, 'num_object_clones': 0, 'num_object_copies': 9032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2258, 'num_write_kb': 18063, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 130}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:38.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:41:38.452 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2383", 'reported_seq': 2452, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:33.664273+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:41:33.664273+0000', 'last_peered': '2024-09-16T04:41:33.664273+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:41:33.664273+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2383, 'log_dups_size': 0, 'ondisk_log_size': 2383, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19511942, 'num_objects': 2382, 'num_object_clones': 0, 'num_object_copies': 9528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2382, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2382, 'num_write_kb': 19055, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 157}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:38.453 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:41:38.453 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2373", 'reported_seq': 2421, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:32.916702+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:41:32.916702+0000', 'last_peered': '2024-09-16T04:41:32.916702+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:41:32.916702+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2373, 'log_dups_size': 0, 'ondisk_log_size': 2373, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19404044, 'num_objects': 2369, 'num_object_clones': 0, 'num_object_copies': 9476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2371, 'num_write_kb': 18964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 135}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:38.453 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:41:38.453 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2256", 'reported_seq': 2330, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:33.533792+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:41:33.533792+0000', 'last_peered': '2024-09-16T04:41:33.533792+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:41:33.533792+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2256, 'log_dups_size': 0, 'ondisk_log_size': 2256, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18453772, 'num_objects': 2253, 'num_object_clones': 0, 'num_object_copies': 9012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2253, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2254, 'num_write_kb': 18029, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:38.453 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:41:38.453 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2384", 'reported_seq': 2444, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:33.753893+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:41:33.753893+0000', 'last_peered': '2024-09-16T04:41:33.753893+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:41:33.753893+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2384, 'log_dups_size': 0, 'ondisk_log_size': 2384, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19485964, 'num_objects': 2379, 'num_object_clones': 0, 'num_object_copies': 9516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2379, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2382, 'num_write_kb': 19044, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 140}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:38.454 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:41:39.163 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:39.164 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35475/151248 objects degraded (23.455%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":111,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":37812,"data_bytes":309716648,"bytes_used":964698112,"bytes_avail":385582358528,"bytes_total":386547056640,"degraded_objects":35475,"degraded_total":151248,"degraded_ratio":0.23454855601396382,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":63860,"recovering_keys_per_sec":0,"num_objects_recovered":78,"num_bytes_recovered":638976,"num_keys_recovered":0,"write_bytes_sec":3870126,"read_op_per_sec":0,"write_op_per_sec":472},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T04:41:34.565807+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:41:39.187 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:41:39.187 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:39.256 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 45 20 40133 40113 6.96189 2.96875 0.0329427 0.0224275 2024-09-16T04:41:39.773 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:39.774 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:39.804 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-16T04:41:38.566602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":316909224,"num_objects":38690,"num_object_clones":0,"num_object_copies":154760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38717,"num_write_kb":309618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38744,"ondisk_log_size":38744,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":954296,"kb_used_data":472696,"kb_used_omap":14,"kb_used_meta":481521,"kb_avail":376533064,"statfs":{"total":386547056640,"available":385569857536,"internally_reserved":0,"allocated":484040704,"data_stored":483357464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":493078065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,14,15,8,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45916160,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.086676"},"pg_stats":[{"pgid":"1.f","version":"31'2476","reported_seq":2514,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.051802+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:38.051802+0000","last_peered":"2024-09-16T04:41:38.051802+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:38.051802+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2476,"log_dups_size":0,"ondisk_log_size":2476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20283392,"num_objects":2476,"num_object_clones":0,"num_object_copies":9904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2476,"num_write_kb":19808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2370","reported_seq":2431,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.853553+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:33.853553+0000","last_peered":"2024-09-16T04:41:33.853553+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:33.853553+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2370,"log_dups_size":0,"ondisk_log_size":2370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379468,"num_objects":2366,"num_object_clones":0,"num_object_copies":9464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2368,"num_write_kb":18940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2380","reported_seq":2450,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.571198+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:33.571198+0000","last_peered":"2024-09-16T04:41:33.571198+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:33.571198+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2380,"log_dups_size":0,"ondisk_log_size":2380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19470982,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2379,"num_write_kb":19022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":150}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2395","reported_seq":2456,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.807234+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:33.807234+0000","last_peered":"2024-09-16T04:41:33.807234+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:33.807234+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2395,"log_dups_size":0,"ondisk_log_size":2395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19584268,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2393,"num_write_kb":19133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2540","reported_seq":2591,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:37.946632+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:37.946632+0000","last_peered":"2024-09-16T04:41:37.946632+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:37.946632+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2540,"log_dups_size":0,"ondisk_log_size":2540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20780300,"num_objects":2537,"num_object_clones":0,"num_object_copies":10148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":20301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2445","reported_seq":2483,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:37.982636+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:37.982636+0000","last_peered":"2024-09-16T04:41:37.982636+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:37.982636+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2445,"log_dups_size":0,"ondisk_log_size":2445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20029440,"num_objects":2445,"num_object_clones":0,"num_object_copies":9780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2445,"num_write_kb":19560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":237}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2419","reported_seq":2479,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.707357+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:33.707357+0000","last_peered":"2024-09-16T04:41:33.707357+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:33.707357+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2419,"log_dups_size":0,"ondisk_log_size":2419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19764492,"num_objects":2413,"num_object_clones":0,"num_object_copies":9652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2417,"num_write_kb":19316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":187}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2402","reported_seq":2618,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.844364+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:33.844364+0000","last_peered":"2024-09-16T04:41:33.844364+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:33.844364+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2402,"log_dups_size":0,"ondisk_log_size":2402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19633420,"num_objects":2397,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2400,"num_write_kb":19188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":237}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2608","reported_seq":2660,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.027969+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:38.027969+0000","last_peered":"2024-09-16T04:41:38.027969+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:38.027969+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2608,"log_dups_size":0,"ondisk_log_size":2608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21293592,"num_objects":2600,"num_object_clones":0,"num_object_copies":10400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2605,"num_write_kb":20817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2361","reported_seq":2434,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.642965+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:33.642965+0000","last_peered":"2024-09-16T04:41:33.642965+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:33.642965+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2361,"log_dups_size":0,"ondisk_log_size":2361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19322124,"num_objects":2359,"num_object_clones":0,"num_object_copies":9436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2359,"num_write_kb":18870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":159}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2455","reported_seq":2508,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.028976+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:38.028976+0000","last_peered":"2024-09-16T04:41:38.028976+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:38.028976+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2455,"log_dups_size":0,"ondisk_log_size":2455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20066194,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2371","reported_seq":2405,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:37.987803+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:37.987803+0000","last_peered":"2024-09-16T04:41:37.987803+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:37.987803+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2371,"log_dups_size":0,"ondisk_log_size":2371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19413638,"num_objects":2370,"num_object_clones":0,"num_object_copies":9480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2370,"num_write_kb":18959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":242}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2383","reported_seq":2452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.664273+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:33.664273+0000","last_peered":"2024-09-16T04:41:33.664273+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:33.664273+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2383,"log_dups_size":0,"ondisk_log_size":2383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19511942,"num_objects":2382,"num_object_clones":0,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2382,"num_write_kb":19055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":157}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2499","reported_seq":2548,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:37.935765+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:37.935765+0000","last_peered":"2024-09-16T04:41:37.935765+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:37.935765+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2499,"log_dups_size":0,"ondisk_log_size":2499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20436236,"num_objects":2495,"num_object_clones":0,"num_object_copies":9980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2497,"num_write_kb":19972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":261}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2256","reported_seq":2330,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.533792+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:33.533792+0000","last_peered":"2024-09-16T04:41:33.533792+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:33.533792+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2256,"log_dups_size":0,"ondisk_log_size":2256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18453772,"num_objects":2253,"num_object_clones":0,"num_object_copies":9012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2254,"num_write_kb":18029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":121}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2384","reported_seq":2444,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:33.753893+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:33.753893+0000","last_peered":"2024-09-16T04:41:33.753893+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:33.753893+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2384,"log_dups_size":0,"ondisk_log_size":2384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19485964,"num_objects":2379,"num_object_clones":0,"num_object_copies":9516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2382,"num_write_kb":19044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":316909224,"num_objects":38690,"num_object_clones":0,"num_object_copies":154760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38717,"num_write_kb":309618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":486469632,"data_stored":486469632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38744,"ondisk_log_size":38744,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297616,"kb_used_data":157632,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94074224,"statfs":{"total":96636764160,"available":96332005376,"internally_reserved":0,"allocated":161415168,"data_stored":161244358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143322831},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,9,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299176,"kb_used_data":151384,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94072664,"statfs":{"total":96636764160,"available":96330407936,"internally_reserved":0,"allocated":155017216,"data_stored":154846406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66328,"kb_used_data":12424,"kb_used_omap":4,"kb_used_meta":53883,"kb_avail":94305512,"statfs":{"total":96636764160,"available":96568844288,"internally_reserved":0,"allocated":12722176,"data_stored":12551366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":55176911},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,2,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291176,"kb_used_data":151256,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94080664,"statfs":{"total":96636764160,"available":96338599936,"internally_reserved":0,"allocated":154886144,"data_stored":154715334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":143257294},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,3,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":155549696,"data_stored":155549696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13201408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155230208,"data_stored":155230208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":162488320,"data_stored":162488320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:40.256 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 46 20 40472 40452 6.86814 2.64844 0.0603725 0.0227343 2024-09-16T04:41:41.258 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 47 20 40893 40873 6.792 3.28906 0.0347043 0.0229902 2024-09-16T04:41:42.256 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 48 20 41317 41297 6.71952 3.3125 0.0913313 0.0232384 2024-09-16T04:41:42.806 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:43.256 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 49 20 41695 41675 6.64266 2.95312 0.0367647 0.0235098 2024-09-16T04:41:43.425 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:43.426 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:43.447 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-16T04:41:42.567551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":326510248,"num_objects":39862,"num_object_clones":0,"num_object_copies":159448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39889,"num_write_kb":318994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1105920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39916,"ondisk_log_size":39916,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":978448,"kb_used_data":496848,"kb_used_omap":14,"kb_used_meta":481521,"kb_avail":376508912,"statfs":{"total":386547056640,"available":385545125888,"internally_reserved":0,"allocated":508772352,"data_stored":508089112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":493078065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,1,5,17,15,21,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33046528,"num_objects":4034,"num_object_clones":0,"num_object_copies":16136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4034,"num_write_kb":32272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002776"},"pg_stats":[{"pgid":"1.f","version":"31'2476","reported_seq":2514,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.051802+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:38.051802+0000","last_peered":"2024-09-16T04:41:38.051802+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:38.051802+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2476,"log_dups_size":0,"ondisk_log_size":2476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20283392,"num_objects":2476,"num_object_clones":0,"num_object_copies":9904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2476,"num_write_kb":19808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2502","reported_seq":2563,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.816972+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:38.816972+0000","last_peered":"2024-09-16T04:41:38.816972+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:38.816972+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2502,"log_dups_size":0,"ondisk_log_size":2502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20460812,"num_objects":2498,"num_object_clones":0,"num_object_copies":9992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2500,"num_write_kb":19996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2527","reported_seq":2596,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.652020+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:38.652020+0000","last_peered":"2024-09-16T04:41:38.652020+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:38.652020+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2527,"log_dups_size":0,"ondisk_log_size":2527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20675206,"num_objects":2524,"num_object_clones":0,"num_object_copies":10096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2526,"num_write_kb":20198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":297}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2522","reported_seq":2583,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.868537+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:38.868537+0000","last_peered":"2024-09-16T04:41:38.868537+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:38.868537+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2522,"log_dups_size":0,"ondisk_log_size":2522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20624652,"num_objects":2518,"num_object_clones":0,"num_object_copies":10072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2520,"num_write_kb":20149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2540","reported_seq":2591,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:37.946632+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:37.946632+0000","last_peered":"2024-09-16T04:41:37.946632+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:37.946632+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2540,"log_dups_size":0,"ondisk_log_size":2540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20780300,"num_objects":2537,"num_object_clones":0,"num_object_copies":10148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":20301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2445","reported_seq":2483,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:37.982636+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:37.982636+0000","last_peered":"2024-09-16T04:41:37.982636+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:37.982636+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2445,"log_dups_size":0,"ondisk_log_size":2445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20029440,"num_objects":2445,"num_object_clones":0,"num_object_copies":9780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2445,"num_write_kb":19560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":237}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2548","reported_seq":2608,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.835840+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:38.835840+0000","last_peered":"2024-09-16T04:41:38.835840+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:38.835840+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2548,"log_dups_size":0,"ondisk_log_size":2548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20821260,"num_objects":2542,"num_object_clones":0,"num_object_copies":10168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2546,"num_write_kb":20348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":316}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2535","reported_seq":2863,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.816358+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:38.816358+0000","last_peered":"2024-09-16T04:41:38.816358+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:38.816358+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2535,"log_dups_size":0,"ondisk_log_size":2535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20722956,"num_objects":2530,"num_object_clones":0,"num_object_copies":10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2533,"num_write_kb":20252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":1105920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":427}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2608","reported_seq":2660,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.027969+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:38.027969+0000","last_peered":"2024-09-16T04:41:38.027969+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:38.027969+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2608,"log_dups_size":0,"ondisk_log_size":2608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21293592,"num_objects":2600,"num_object_clones":0,"num_object_copies":10400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2605,"num_write_kb":20817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2487","reported_seq":2560,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.553306+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:38.553306+0000","last_peered":"2024-09-16T04:41:38.553306+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:38.553306+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2487,"log_dups_size":0,"ondisk_log_size":2487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20354316,"num_objects":2485,"num_object_clones":0,"num_object_copies":9940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2485,"num_write_kb":19878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":285}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2455","reported_seq":2508,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.028976+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:38.028976+0000","last_peered":"2024-09-16T04:41:38.028976+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:38.028976+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2455,"log_dups_size":0,"ondisk_log_size":2455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20066194,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2371","reported_seq":2405,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:37.987803+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:37.987803+0000","last_peered":"2024-09-16T04:41:37.987803+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:37.987803+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2371,"log_dups_size":0,"ondisk_log_size":2371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19413638,"num_objects":2370,"num_object_clones":0,"num_object_copies":9480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2370,"num_write_kb":18959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":242}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2499","reported_seq":2568,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.583387+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:38.583387+0000","last_peered":"2024-09-16T04:41:38.583387+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:38.583387+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2499,"log_dups_size":0,"ondisk_log_size":2499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20462214,"num_objects":2498,"num_object_clones":0,"num_object_copies":9992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2498,"num_write_kb":19983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2499","reported_seq":2548,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:37.935765+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:37.935765+0000","last_peered":"2024-09-16T04:41:37.935765+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:37.935765+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2499,"log_dups_size":0,"ondisk_log_size":2499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20436236,"num_objects":2495,"num_object_clones":0,"num_object_copies":9980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2497,"num_write_kb":19972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":261}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2396","reported_seq":2469,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.671242+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:38.671242+0000","last_peered":"2024-09-16T04:41:38.671242+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:38.671242+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2396,"log_dups_size":0,"ondisk_log_size":2396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19600652,"num_objects":2393,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2394,"num_write_kb":19149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":261}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2506","reported_seq":2567,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.812957+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:38.812957+0000","last_peered":"2024-09-16T04:41:38.812957+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:38.812957+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2506,"log_dups_size":0,"ondisk_log_size":2506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20485388,"num_objects":2501,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":20020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":326510248,"num_objects":39862,"num_object_clones":0,"num_object_copies":159448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39889,"num_write_kb":318994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1105920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":511197184,"data_stored":511197184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39916,"ondisk_log_size":39916,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297616,"kb_used_data":157632,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94074224,"statfs":{"total":96636764160,"available":96332005376,"internally_reserved":0,"allocated":161415168,"data_stored":161244358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143322831},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,9,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294776,"kb_used_data":159464,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94077064,"statfs":{"total":96636764160,"available":96334913536,"internally_reserved":0,"allocated":163291136,"data_stored":163120326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,0,1,0,2,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82768,"kb_used_data":20544,"kb_used_omap":4,"kb_used_meta":62203,"kb_avail":94289072,"statfs":{"total":96636764160,"available":96552009728,"internally_reserved":0,"allocated":21037056,"data_stored":20866246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":63696591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,9,1,11],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303288,"kb_used_data":159208,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94068552,"statfs":{"total":96636764160,"available":96326197248,"internally_reserved":0,"allocated":163028992,"data_stored":162858182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":147517134},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,3,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163852288,"data_stored":163852288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21372928,"data_stored":21372928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163483648,"data_stored":163483648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":162488320,"data_stored":162488320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:43.448 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:44.033 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:44.033 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:44.060 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-16T04:41:42.567551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":326510248,"num_objects":39862,"num_object_clones":0,"num_object_copies":159448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39889,"num_write_kb":318994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1105920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39916,"ondisk_log_size":39916,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":978448,"kb_used_data":496848,"kb_used_omap":14,"kb_used_meta":481521,"kb_avail":376508912,"statfs":{"total":386547056640,"available":385545125888,"internally_reserved":0,"allocated":508772352,"data_stored":508089112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":493078065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,1,5,17,15,21,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33046528,"num_objects":4034,"num_object_clones":0,"num_object_copies":16136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4034,"num_write_kb":32272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002776"},"pg_stats":[{"pgid":"1.f","version":"31'2476","reported_seq":2514,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.051802+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:38.051802+0000","last_peered":"2024-09-16T04:41:38.051802+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:38.051802+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2476,"log_dups_size":0,"ondisk_log_size":2476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20283392,"num_objects":2476,"num_object_clones":0,"num_object_copies":9904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2476,"num_write_kb":19808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2502","reported_seq":2563,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.816972+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:38.816972+0000","last_peered":"2024-09-16T04:41:38.816972+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:38.816972+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2502,"log_dups_size":0,"ondisk_log_size":2502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20460812,"num_objects":2498,"num_object_clones":0,"num_object_copies":9992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2500,"num_write_kb":19996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2527","reported_seq":2596,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.652020+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:38.652020+0000","last_peered":"2024-09-16T04:41:38.652020+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:38.652020+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2527,"log_dups_size":0,"ondisk_log_size":2527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20675206,"num_objects":2524,"num_object_clones":0,"num_object_copies":10096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2526,"num_write_kb":20198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":297}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2522","reported_seq":2583,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.868537+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:38.868537+0000","last_peered":"2024-09-16T04:41:38.868537+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:38.868537+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2522,"log_dups_size":0,"ondisk_log_size":2522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20624652,"num_objects":2518,"num_object_clones":0,"num_object_copies":10072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2520,"num_write_kb":20149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2540","reported_seq":2591,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:37.946632+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:37.946632+0000","last_peered":"2024-09-16T04:41:37.946632+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:37.946632+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2540,"log_dups_size":0,"ondisk_log_size":2540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20780300,"num_objects":2537,"num_object_clones":0,"num_object_copies":10148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":20301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2445","reported_seq":2483,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:37.982636+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:37.982636+0000","last_peered":"2024-09-16T04:41:37.982636+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:37.982636+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2445,"log_dups_size":0,"ondisk_log_size":2445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20029440,"num_objects":2445,"num_object_clones":0,"num_object_copies":9780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2445,"num_write_kb":19560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":237}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2548","reported_seq":2608,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.835840+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:38.835840+0000","last_peered":"2024-09-16T04:41:38.835840+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:38.835840+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2548,"log_dups_size":0,"ondisk_log_size":2548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20821260,"num_objects":2542,"num_object_clones":0,"num_object_copies":10168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2546,"num_write_kb":20348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":316}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2535","reported_seq":2863,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.816358+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:38.816358+0000","last_peered":"2024-09-16T04:41:38.816358+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:38.816358+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2535,"log_dups_size":0,"ondisk_log_size":2535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20722956,"num_objects":2530,"num_object_clones":0,"num_object_copies":10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2533,"num_write_kb":20252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":1105920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":427}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2608","reported_seq":2660,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.027969+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:38.027969+0000","last_peered":"2024-09-16T04:41:38.027969+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:38.027969+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2608,"log_dups_size":0,"ondisk_log_size":2608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21293592,"num_objects":2600,"num_object_clones":0,"num_object_copies":10400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2605,"num_write_kb":20817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2487","reported_seq":2560,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.553306+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:38.553306+0000","last_peered":"2024-09-16T04:41:38.553306+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:38.553306+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2487,"log_dups_size":0,"ondisk_log_size":2487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20354316,"num_objects":2485,"num_object_clones":0,"num_object_copies":9940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2485,"num_write_kb":19878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":285}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2455","reported_seq":2508,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.028976+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:38.028976+0000","last_peered":"2024-09-16T04:41:38.028976+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:38.028976+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2455,"log_dups_size":0,"ondisk_log_size":2455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20066194,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2371","reported_seq":2405,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:37.987803+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:37.987803+0000","last_peered":"2024-09-16T04:41:37.987803+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:37.987803+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2371,"log_dups_size":0,"ondisk_log_size":2371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19413638,"num_objects":2370,"num_object_clones":0,"num_object_copies":9480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2370,"num_write_kb":18959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":242}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2499","reported_seq":2568,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.583387+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:38.583387+0000","last_peered":"2024-09-16T04:41:38.583387+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:38.583387+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2499,"log_dups_size":0,"ondisk_log_size":2499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20462214,"num_objects":2498,"num_object_clones":0,"num_object_copies":9992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2498,"num_write_kb":19983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2499","reported_seq":2548,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:37.935765+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:37.935765+0000","last_peered":"2024-09-16T04:41:37.935765+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:37.935765+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2499,"log_dups_size":0,"ondisk_log_size":2499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20436236,"num_objects":2495,"num_object_clones":0,"num_object_copies":9980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2497,"num_write_kb":19972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":261}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2396","reported_seq":2469,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.671242+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:38.671242+0000","last_peered":"2024-09-16T04:41:38.671242+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:38.671242+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2396,"log_dups_size":0,"ondisk_log_size":2396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19600652,"num_objects":2393,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2394,"num_write_kb":19149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":261}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2506","reported_seq":2567,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:38.812957+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:38.812957+0000","last_peered":"2024-09-16T04:41:38.812957+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:38.812957+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2506,"log_dups_size":0,"ondisk_log_size":2506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20485388,"num_objects":2501,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":20020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":326510248,"num_objects":39862,"num_object_clones":0,"num_object_copies":159448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39889,"num_write_kb":318994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1105920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":511197184,"data_stored":511197184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39916,"ondisk_log_size":39916,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297616,"kb_used_data":157632,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94074224,"statfs":{"total":96636764160,"available":96332005376,"internally_reserved":0,"allocated":161415168,"data_stored":161244358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143322831},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,9,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294776,"kb_used_data":159464,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94077064,"statfs":{"total":96636764160,"available":96334913536,"internally_reserved":0,"allocated":163291136,"data_stored":163120326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,0,1,0,2,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82768,"kb_used_data":20544,"kb_used_omap":4,"kb_used_meta":62203,"kb_avail":94289072,"statfs":{"total":96636764160,"available":96552009728,"internally_reserved":0,"allocated":21037056,"data_stored":20866246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":63696591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,9,1,11],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303288,"kb_used_data":159208,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94068552,"statfs":{"total":96636764160,"available":96326197248,"internally_reserved":0,"allocated":163028992,"data_stored":162858182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":147517134},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,3,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163852288,"data_stored":163852288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21372928,"data_stored":21372928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163483648,"data_stored":163483648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":162488320,"data_stored":162488320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:44.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:41:44.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2476", 'reported_seq': 2514, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:38.051802+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:41:38.051802+0000', 'last_peered': '2024-09-16T04:41:38.051802+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:41:38.051802+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2476, 'log_dups_size': 0, 'ondisk_log_size': 2476, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20283392, 'num_objects': 2476, 'num_object_clones': 0, 'num_object_copies': 9904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2476, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2476, 'num_write_kb': 19808, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 257}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:44.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:41:44.061 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2502", 'reported_seq': 2563, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:38.816972+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:41:38.816972+0000', 'last_peered': '2024-09-16T04:41:38.816972+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:41:38.816972+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2502, 'log_dups_size': 0, 'ondisk_log_size': 2502, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20460812, 'num_objects': 2498, 'num_object_clones': 0, 'num_object_copies': 9992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2498, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2500, 'num_write_kb': 19996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:44.061 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:41:44.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2527", 'reported_seq': 2596, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:38.652020+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:41:38.652020+0000', 'last_peered': '2024-09-16T04:41:38.652020+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:41:38.652020+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2527, 'log_dups_size': 0, 'ondisk_log_size': 2527, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20675206, 'num_objects': 2524, 'num_object_clones': 0, 'num_object_copies': 10096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2526, 'num_write_kb': 20198, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 297}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:44.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:41:44.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2522", 'reported_seq': 2583, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:38.868537+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:41:38.868537+0000', 'last_peered': '2024-09-16T04:41:38.868537+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:41:38.868537+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2522, 'log_dups_size': 0, 'ondisk_log_size': 2522, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20624652, 'num_objects': 2518, 'num_object_clones': 0, 'num_object_copies': 10072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2520, 'num_write_kb': 20149, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 289}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:44.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:41:44.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2540", 'reported_seq': 2591, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:37.946632+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:41:37.946632+0000', 'last_peered': '2024-09-16T04:41:37.946632+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:41:37.946632+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2540, 'log_dups_size': 0, 'ondisk_log_size': 2540, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20780300, 'num_objects': 2537, 'num_object_clones': 0, 'num_object_copies': 10148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2537, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2538, 'num_write_kb': 20301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 254}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:44.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:41:44.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2445", 'reported_seq': 2483, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:37.982636+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:41:37.982636+0000', 'last_peered': '2024-09-16T04:41:37.982636+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:41:37.982636+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2445, 'log_dups_size': 0, 'ondisk_log_size': 2445, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20029440, 'num_objects': 2445, 'num_object_clones': 0, 'num_object_copies': 9780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2445, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2445, 'num_write_kb': 19560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 237}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:44.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:41:44.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2548", 'reported_seq': 2608, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:38.835840+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:41:38.835840+0000', 'last_peered': '2024-09-16T04:41:38.835840+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:41:38.835840+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2548, 'log_dups_size': 0, 'ondisk_log_size': 2548, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20821260, 'num_objects': 2542, 'num_object_clones': 0, 'num_object_copies': 10168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2542, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2546, 'num_write_kb': 20348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 316}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:44.062 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:41:44.062 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2535", 'reported_seq': 2863, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:38.816358+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:41:38.816358+0000', 'last_peered': '2024-09-16T04:41:38.816358+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:41:38.816358+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2535, 'log_dups_size': 0, 'ondisk_log_size': 2535, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20722956, 'num_objects': 2530, 'num_object_clones': 0, 'num_object_copies': 10120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2530, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2533, 'num_write_kb': 20252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 137, 'num_bytes_recovered': 1105920, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 427}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:44.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:41:44.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2608", 'reported_seq': 2660, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:38.027969+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:41:38.027969+0000', 'last_peered': '2024-09-16T04:41:38.027969+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:41:38.027969+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2608, 'log_dups_size': 0, 'ondisk_log_size': 2608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21293592, 'num_objects': 2600, 'num_object_clones': 0, 'num_object_copies': 10400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2600, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2605, 'num_write_kb': 20817, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 260}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:44.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:41:44.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2487", 'reported_seq': 2560, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:38.553306+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:41:38.553306+0000', 'last_peered': '2024-09-16T04:41:38.553306+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:41:38.553306+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2487, 'log_dups_size': 0, 'ondisk_log_size': 2487, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20354316, 'num_objects': 2485, 'num_object_clones': 0, 'num_object_copies': 9940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2485, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2485, 'num_write_kb': 19878, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 285}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:44.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:41:44.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2455", 'reported_seq': 2508, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:38.028976+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:41:38.028976+0000', 'last_peered': '2024-09-16T04:41:38.028976+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:41:38.028976+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2455, 'log_dups_size': 0, 'ondisk_log_size': 2455, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20066194, 'num_objects': 2450, 'num_object_clones': 0, 'num_object_copies': 9800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2450, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2452, 'num_write_kb': 19604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 261}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:44.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:41:44.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2371", 'reported_seq': 2405, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:37.987803+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:41:37.987803+0000', 'last_peered': '2024-09-16T04:41:37.987803+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:41:37.987803+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2371, 'log_dups_size': 0, 'ondisk_log_size': 2371, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19413638, 'num_objects': 2370, 'num_object_clones': 0, 'num_object_copies': 9480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2370, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2370, 'num_write_kb': 18959, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 242}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:44.063 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:41:44.063 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2499", 'reported_seq': 2568, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:38.583387+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:41:38.583387+0000', 'last_peered': '2024-09-16T04:41:38.583387+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:41:38.583387+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2499, 'log_dups_size': 0, 'ondisk_log_size': 2499, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20462214, 'num_objects': 2498, 'num_object_clones': 0, 'num_object_copies': 9992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2498, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2498, 'num_write_kb': 19983, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 273}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:44.064 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:41:44.064 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2499", 'reported_seq': 2548, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:37.935765+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:41:37.935765+0000', 'last_peered': '2024-09-16T04:41:37.935765+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:41:37.935765+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2499, 'log_dups_size': 0, 'ondisk_log_size': 2499, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20436236, 'num_objects': 2495, 'num_object_clones': 0, 'num_object_copies': 9980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2495, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2497, 'num_write_kb': 19972, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 261}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:44.064 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:41:44.064 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2396", 'reported_seq': 2469, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:38.671242+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:41:38.671242+0000', 'last_peered': '2024-09-16T04:41:38.671242+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:41:38.671242+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2396, 'log_dups_size': 0, 'ondisk_log_size': 2396, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19600652, 'num_objects': 2393, 'num_object_clones': 0, 'num_object_copies': 9572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2393, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2394, 'num_write_kb': 19149, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 261}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:44.064 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:41:44.064 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2506", 'reported_seq': 2567, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:38.812957+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:41:38.812957+0000', 'last_peered': '2024-09-16T04:41:38.812957+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:41:38.812957+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2506, 'log_dups_size': 0, 'ondisk_log_size': 2506, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20485388, 'num_objects': 2501, 'num_object_clones': 0, 'num_object_copies': 10004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2504, 'num_write_kb': 20020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:44.064 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:41:44.257 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 50 20 42060 42040 6.56685 2.85156 0.0317259 0.0237745 2024-09-16T04:41:44.754 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:44.755 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35418/159448 objects degraded (22.213%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":117,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":39862,"data_bytes":326510248,"bytes_used":1001930752,"bytes_avail":385545125888,"bytes_total":386547056640,"degraded_objects":35418,"degraded_total":159448,"degraded_ratio":0.22212884451357179,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":81218,"recovering_keys_per_sec":0,"num_objects_recovered":119,"num_bytes_recovered":974848,"num_keys_recovered":0,"write_bytes_sec":2753240,"read_op_per_sec":0,"write_op_per_sec":336},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T04:41:34.565807+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:41:44.800 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:41:44.800 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:45.258 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 51 20 42421 42401 6.49321 2.82031 0.0665558 0.0240467 2024-09-16T04:41:45.371 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:45.372 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:45.396 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-09-16T04:41:44.567985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":342394536,"num_objects":41801,"num_object_clones":0,"num_object_copies":167204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35344,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41828,"num_write_kb":334506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41855,"ondisk_log_size":41855,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1018364,"kb_used_data":520120,"kb_used_omap":14,"kb_used_meta":498161,"kb_avail":376468996,"statfs":{"total":386547056640,"available":385504251904,"internally_reserved":0,"allocated":532602880,"data_stored":531919640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":510117425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,12,10,18,20,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":48930816,"num_objects":5973,"num_object_clones":0,"num_object_copies":23892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5973,"num_write_kb":47784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":1581056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002737"},"pg_stats":[{"pgid":"1.f","version":"31'2608","reported_seq":2647,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:42.972867+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:42.972867+0000","last_peered":"2024-09-16T04:41:42.972867+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:42.972867+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2608,"log_dups_size":0,"ondisk_log_size":2608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21364736,"num_objects":2608,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2608,"num_write_kb":20864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2617","reported_seq":2677,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.872429+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:43.872429+0000","last_peered":"2024-09-16T04:41:43.872429+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:43.872429+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2617,"log_dups_size":0,"ondisk_log_size":2617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21402892,"num_objects":2613,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2615,"num_write_kb":20916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2634","reported_seq":2704,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.536594+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:43.536594+0000","last_peered":"2024-09-16T04:41:43.536594+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:43.536594+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21551750,"num_objects":2631,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":21054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2635","reported_seq":2696,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.918139+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:43.918139+0000","last_peered":"2024-09-16T04:41:43.918139+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:43.918139+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2635,"log_dups_size":0,"ondisk_log_size":2635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21550348,"num_objects":2631,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":21053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2665","reported_seq":2716,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.075517+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:43.075517+0000","last_peered":"2024-09-16T04:41:43.075517+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:43.075517+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2665,"log_dups_size":0,"ondisk_log_size":2665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21804300,"num_objects":2662,"num_object_clones":0,"num_object_copies":10648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2663,"num_write_kb":21301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":379}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2570","reported_seq":2608,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.021657+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:43.021657+0000","last_peered":"2024-09-16T04:41:43.021657+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:43.021657+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2570,"log_dups_size":0,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21053440,"num_objects":2570,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2570,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2677","reported_seq":2737,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.791473+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:43.791473+0000","last_peered":"2024-09-16T04:41:43.791473+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:43.791473+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2677,"log_dups_size":0,"ondisk_log_size":2677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21878028,"num_objects":2671,"num_object_clones":0,"num_object_copies":10684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2675,"num_write_kb":21380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":445}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2665","reported_seq":3142,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.920180+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:43.920180+0000","last_peered":"2024-09-16T04:41:43.920180+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:43.920180+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2665,"log_dups_size":0,"ondisk_log_size":2665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21787916,"num_objects":2660,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2663,"num_write_kb":21292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2745","reported_seq":2796,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.048129+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:43.048129+0000","last_peered":"2024-09-16T04:41:43.048129+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:43.048129+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2745,"log_dups_size":0,"ondisk_log_size":2745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22415896,"num_objects":2737,"num_object_clones":0,"num_object_copies":10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2742,"num_write_kb":21913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2611","reported_seq":2684,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.670992+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:43.670992+0000","last_peered":"2024-09-16T04:41:43.670992+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:43.670992+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2611,"log_dups_size":0,"ondisk_log_size":2611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21370124,"num_objects":2609,"num_object_clones":0,"num_object_copies":10436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2609,"num_write_kb":20870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":409}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2593","reported_seq":2646,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:42.984699+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:42.984699+0000","last_peered":"2024-09-16T04:41:42.984699+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:42.984699+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2593,"log_dups_size":0,"ondisk_log_size":2593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21196690,"num_objects":2588,"num_object_clones":0,"num_object_copies":10352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2590,"num_write_kb":20708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":399}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2491","reported_seq":2526,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.102154+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:43.102154+0000","last_peered":"2024-09-16T04:41:43.102154+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:43.102154+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2491,"log_dups_size":0,"ondisk_log_size":2491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20396678,"num_objects":2490,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2490,"num_write_kb":19919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":362}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2609","reported_seq":2678,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.678157+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:43.678157+0000","last_peered":"2024-09-16T04:41:43.678157+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:43.678157+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2609,"log_dups_size":0,"ondisk_log_size":2609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21363334,"num_objects":2608,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2608,"num_write_kb":20863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2606","reported_seq":2655,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.070149+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:43.070149+0000","last_peered":"2024-09-16T04:41:43.070149+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:43.070149+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2606,"log_dups_size":0,"ondisk_log_size":2606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21312780,"num_objects":2602,"num_object_clones":0,"num_object_copies":10408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2604,"num_write_kb":20828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":368}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2507","reported_seq":2581,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.628005+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:43.628005+0000","last_peered":"2024-09-16T04:41:43.628005+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:43.628005+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2507,"log_dups_size":0,"ondisk_log_size":2507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20509964,"num_objects":2504,"num_object_clones":0,"num_object_copies":10016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2505,"num_write_kb":20037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":372}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2622","reported_seq":2683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.839245+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:43.839245+0000","last_peered":"2024-09-16T04:41:43.839245+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:43.839245+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2622,"log_dups_size":0,"ondisk_log_size":2622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21435660,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2620,"num_write_kb":20948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":378}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":342394536,"num_objects":41801,"num_object_clones":0,"num_object_copies":167204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35344,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41828,"num_write_kb":334506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":535109632,"data_stored":535109632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41855,"ondisk_log_size":41855,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313628,"kb_used_data":165324,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94058212,"statfs":{"total":96636764160,"available":96315609088,"internally_reserved":0,"allocated":169291776,"data_stored":169120966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":151842511},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,8,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306700,"kb_used_data":167224,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94065140,"statfs":{"total":96636764160,"available":96322703360,"internally_reserved":0,"allocated":171237376,"data_stored":171066566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,12,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82768,"kb_used_data":20544,"kb_used_omap":4,"kb_used_meta":62203,"kb_avail":94289072,"statfs":{"total":96636764160,"available":96552009728,"internally_reserved":0,"allocated":21037056,"data_stored":20866246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":63696591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,9,1,11],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315268,"kb_used_data":167028,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94056572,"statfs":{"total":96636764160,"available":96313929728,"internally_reserved":0,"allocated":171036672,"data_stored":170865862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":151776974},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,0,1,1,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":171773952,"data_stored":171773952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21372928,"data_stored":21372928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":171421696,"data_stored":171421696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":170541056,"data_stored":170541056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:46.259 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 52 20 42837 42817 6.43084 3.25 0.0212938 0.0242859 2024-09-16T04:41:47.260 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 53 20 43242 43222 6.36922 3.16406 0.0393358 0.0245078 2024-09-16T04:41:48.259 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 54 20 43654 43634 6.31088 3.21875 0.0392879 0.0247429 2024-09-16T04:41:48.398 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:49.014 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:49.014 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:49.033 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-09-16T04:41:48.569578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":349308584,"num_objects":42645,"num_object_clones":0,"num_object_copies":170580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35344,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42672,"num_write_kb":341258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42699,"ondisk_log_size":42699,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1025900,"kb_used_data":535976,"kb_used_omap":14,"kb_used_meta":489841,"kb_avail":376461460,"statfs":{"total":386547056640,"available":385496535040,"internally_reserved":0,"allocated":548839424,"data_stored":548156184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":501597745},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,12,1,23,8,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39591936,"num_objects":4833,"num_object_clones":0,"num_object_copies":19332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4833,"num_write_kb":38664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":1073152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003401"},"pg_stats":[{"pgid":"1.f","version":"31'2734","reported_seq":2773,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.032338+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:48.032338+0000","last_peered":"2024-09-16T04:41:48.032338+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:48.032338+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2734,"log_dups_size":0,"ondisk_log_size":2734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22396928,"num_objects":2734,"num_object_clones":0,"num_object_copies":10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2734,"num_write_kb":21872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":515}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2617","reported_seq":2677,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.872429+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:43.872429+0000","last_peered":"2024-09-16T04:41:43.872429+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:43.872429+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2617,"log_dups_size":0,"ondisk_log_size":2617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21402892,"num_objects":2613,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2615,"num_write_kb":20916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2634","reported_seq":2704,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.536594+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:43.536594+0000","last_peered":"2024-09-16T04:41:43.536594+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:43.536594+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21551750,"num_objects":2631,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":21054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2635","reported_seq":2696,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.918139+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:43.918139+0000","last_peered":"2024-09-16T04:41:43.918139+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:43.918139+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2635,"log_dups_size":0,"ondisk_log_size":2635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21550348,"num_objects":2631,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":21053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2775","reported_seq":2826,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:47.837010+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:47.837010+0000","last_peered":"2024-09-16T04:41:47.837010+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:47.837010+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2775,"log_dups_size":0,"ondisk_log_size":2775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22705420,"num_objects":2772,"num_object_clones":0,"num_object_copies":11088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2773,"num_write_kb":22181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2689","reported_seq":2728,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.050059+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:48.050059+0000","last_peered":"2024-09-16T04:41:48.050059+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:48.050059+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2689,"log_dups_size":0,"ondisk_log_size":2689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22028288,"num_objects":2689,"num_object_clones":0,"num_object_copies":10756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2689,"num_write_kb":21512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":481}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2677","reported_seq":2737,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.791473+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:43.791473+0000","last_peered":"2024-09-16T04:41:43.791473+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:43.791473+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2677,"log_dups_size":0,"ondisk_log_size":2677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21878028,"num_objects":2671,"num_object_clones":0,"num_object_copies":10684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2675,"num_write_kb":21380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":445}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2665","reported_seq":3142,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.920180+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:43.920180+0000","last_peered":"2024-09-16T04:41:43.920180+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:43.920180+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2665,"log_dups_size":0,"ondisk_log_size":2665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21787916,"num_objects":2660,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2663,"num_write_kb":21292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2862","reported_seq":2913,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.085591+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:48.085591+0000","last_peered":"2024-09-16T04:41:48.085591+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:48.085591+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2862,"log_dups_size":0,"ondisk_log_size":2862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23374360,"num_objects":2854,"num_object_clones":0,"num_object_copies":11416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2611","reported_seq":2684,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.670992+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:43.670992+0000","last_peered":"2024-09-16T04:41:43.670992+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:43.670992+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2611,"log_dups_size":0,"ondisk_log_size":2611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21370124,"num_objects":2609,"num_object_clones":0,"num_object_copies":10436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2609,"num_write_kb":20870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":409}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2708","reported_seq":2761,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:47.916053+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:47.916053+0000","last_peered":"2024-09-16T04:41:47.916053+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:47.916053+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2708,"log_dups_size":0,"ondisk_log_size":2708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22138770,"num_objects":2703,"num_object_clones":0,"num_object_copies":10812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2705,"num_write_kb":21628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2634","reported_seq":2669,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.000936+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:48.000936+0000","last_peered":"2024-09-16T04:41:48.000936+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:48.000936+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21568134,"num_objects":2633,"num_object_clones":0,"num_object_copies":10532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":21063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":505}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2609","reported_seq":2678,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.678157+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:43.678157+0000","last_peered":"2024-09-16T04:41:43.678157+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:43.678157+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2609,"log_dups_size":0,"ondisk_log_size":2609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21363334,"num_objects":2608,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2608,"num_write_kb":20863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2720","reported_seq":2769,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.094784+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:48.094784+0000","last_peered":"2024-09-16T04:41:48.094784+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:48.094784+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2720,"log_dups_size":0,"ondisk_log_size":2720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22246668,"num_objects":2716,"num_object_clones":0,"num_object_copies":10864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2718,"num_write_kb":21740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":482}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2507","reported_seq":2581,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.628005+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:43.628005+0000","last_peered":"2024-09-16T04:41:43.628005+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:43.628005+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2507,"log_dups_size":0,"ondisk_log_size":2507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20509964,"num_objects":2504,"num_object_clones":0,"num_object_copies":10016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2505,"num_write_kb":20037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":372}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2622","reported_seq":2683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.839245+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:43.839245+0000","last_peered":"2024-09-16T04:41:43.839245+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:43.839245+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2622,"log_dups_size":0,"ondisk_log_size":2622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21435660,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2620,"num_write_kb":20948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":378}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":349308584,"num_objects":42645,"num_object_clones":0,"num_object_copies":170580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35344,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42672,"num_write_kb":341258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":551321600,"data_stored":551321600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42699,"ondisk_log_size":42699,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325700,"kb_used_data":173236,"kb_used_omap":4,"kb_used_meta":152443,"kb_avail":94046140,"statfs":{"total":96636764160,"available":96303247360,"internally_reserved":0,"allocated":177393664,"data_stored":177222854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":156102351},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306700,"kb_used_data":167224,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94065140,"statfs":{"total":96636764160,"available":96322703360,"internally_reserved":0,"allocated":171237376,"data_stored":171066566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,12,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051589,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78232,"kb_used_data":28488,"kb_used_omap":4,"kb_used_meta":49723,"kb_avail":94293608,"statfs":{"total":96636764160,"available":96556654592,"internally_reserved":0,"allocated":29171712,"data_stored":29000902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":50917071},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315268,"kb_used_data":167028,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94056572,"statfs":{"total":96636764160,"available":96313929728,"internally_reserved":0,"allocated":171036672,"data_stored":170865862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":151776974},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,0,1,1,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":171773952,"data_stored":171773952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29556736,"data_stored":29556736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":171421696,"data_stored":171421696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":178569216,"data_stored":178569216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:49.034 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:49.259 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 55 20 44032 44012 6.24984 2.95312 0.0471818 0.0249806 2024-09-16T04:41:49.599 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:49.599 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:49.624 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-09-16T04:41:48.569578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":349308584,"num_objects":42645,"num_object_clones":0,"num_object_copies":170580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35344,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42672,"num_write_kb":341258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42699,"ondisk_log_size":42699,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1025900,"kb_used_data":535976,"kb_used_omap":14,"kb_used_meta":489841,"kb_avail":376461460,"statfs":{"total":386547056640,"available":385496535040,"internally_reserved":0,"allocated":548839424,"data_stored":548156184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":501597745},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,12,1,23,8,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39591936,"num_objects":4833,"num_object_clones":0,"num_object_copies":19332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4833,"num_write_kb":38664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":1073152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003401"},"pg_stats":[{"pgid":"1.f","version":"31'2734","reported_seq":2773,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.032338+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:48.032338+0000","last_peered":"2024-09-16T04:41:48.032338+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:48.032338+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2734,"log_dups_size":0,"ondisk_log_size":2734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22396928,"num_objects":2734,"num_object_clones":0,"num_object_copies":10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2734,"num_write_kb":21872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":515}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2617","reported_seq":2677,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.872429+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:43.872429+0000","last_peered":"2024-09-16T04:41:43.872429+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:43.872429+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2617,"log_dups_size":0,"ondisk_log_size":2617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21402892,"num_objects":2613,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2615,"num_write_kb":20916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2634","reported_seq":2704,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.536594+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:43.536594+0000","last_peered":"2024-09-16T04:41:43.536594+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:43.536594+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21551750,"num_objects":2631,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":21054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2635","reported_seq":2696,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.918139+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:43.918139+0000","last_peered":"2024-09-16T04:41:43.918139+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:43.918139+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2635,"log_dups_size":0,"ondisk_log_size":2635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21550348,"num_objects":2631,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":21053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2775","reported_seq":2826,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:47.837010+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:47.837010+0000","last_peered":"2024-09-16T04:41:47.837010+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:47.837010+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2775,"log_dups_size":0,"ondisk_log_size":2775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22705420,"num_objects":2772,"num_object_clones":0,"num_object_copies":11088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2773,"num_write_kb":22181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2689","reported_seq":2728,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.050059+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:48.050059+0000","last_peered":"2024-09-16T04:41:48.050059+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:48.050059+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2689,"log_dups_size":0,"ondisk_log_size":2689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22028288,"num_objects":2689,"num_object_clones":0,"num_object_copies":10756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2689,"num_write_kb":21512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":481}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2677","reported_seq":2737,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.791473+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:43.791473+0000","last_peered":"2024-09-16T04:41:43.791473+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:43.791473+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2677,"log_dups_size":0,"ondisk_log_size":2677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21878028,"num_objects":2671,"num_object_clones":0,"num_object_copies":10684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2675,"num_write_kb":21380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":445}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2665","reported_seq":3142,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.920180+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:43.920180+0000","last_peered":"2024-09-16T04:41:43.920180+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:43.920180+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2665,"log_dups_size":0,"ondisk_log_size":2665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21787916,"num_objects":2660,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2663,"num_write_kb":21292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2862","reported_seq":2913,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.085591+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:48.085591+0000","last_peered":"2024-09-16T04:41:48.085591+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:48.085591+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2862,"log_dups_size":0,"ondisk_log_size":2862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23374360,"num_objects":2854,"num_object_clones":0,"num_object_copies":11416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2611","reported_seq":2684,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.670992+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:43.670992+0000","last_peered":"2024-09-16T04:41:43.670992+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:43.670992+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2611,"log_dups_size":0,"ondisk_log_size":2611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21370124,"num_objects":2609,"num_object_clones":0,"num_object_copies":10436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2609,"num_write_kb":20870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":409}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2708","reported_seq":2761,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:47.916053+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:47.916053+0000","last_peered":"2024-09-16T04:41:47.916053+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:47.916053+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2708,"log_dups_size":0,"ondisk_log_size":2708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22138770,"num_objects":2703,"num_object_clones":0,"num_object_copies":10812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2705,"num_write_kb":21628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2634","reported_seq":2669,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.000936+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:48.000936+0000","last_peered":"2024-09-16T04:41:48.000936+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:48.000936+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21568134,"num_objects":2633,"num_object_clones":0,"num_object_copies":10532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":21063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":505}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2609","reported_seq":2678,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.678157+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:43.678157+0000","last_peered":"2024-09-16T04:41:43.678157+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:43.678157+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2609,"log_dups_size":0,"ondisk_log_size":2609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21363334,"num_objects":2608,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2608,"num_write_kb":20863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2720","reported_seq":2769,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.094784+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:48.094784+0000","last_peered":"2024-09-16T04:41:48.094784+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:48.094784+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2720,"log_dups_size":0,"ondisk_log_size":2720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22246668,"num_objects":2716,"num_object_clones":0,"num_object_copies":10864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2718,"num_write_kb":21740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":482}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2507","reported_seq":2581,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.628005+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:43.628005+0000","last_peered":"2024-09-16T04:41:43.628005+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:43.628005+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2507,"log_dups_size":0,"ondisk_log_size":2507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20509964,"num_objects":2504,"num_object_clones":0,"num_object_copies":10016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2505,"num_write_kb":20037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":372}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2622","reported_seq":2683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:43.839245+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:43.839245+0000","last_peered":"2024-09-16T04:41:43.839245+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:43.839245+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2622,"log_dups_size":0,"ondisk_log_size":2622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21435660,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2620,"num_write_kb":20948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":378}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":349308584,"num_objects":42645,"num_object_clones":0,"num_object_copies":170580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35344,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42672,"num_write_kb":341258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":551321600,"data_stored":551321600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42699,"ondisk_log_size":42699,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325700,"kb_used_data":173236,"kb_used_omap":4,"kb_used_meta":152443,"kb_avail":94046140,"statfs":{"total":96636764160,"available":96303247360,"internally_reserved":0,"allocated":177393664,"data_stored":177222854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":156102351},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306700,"kb_used_data":167224,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94065140,"statfs":{"total":96636764160,"available":96322703360,"internally_reserved":0,"allocated":171237376,"data_stored":171066566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,12,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051589,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78232,"kb_used_data":28488,"kb_used_omap":4,"kb_used_meta":49723,"kb_avail":94293608,"statfs":{"total":96636764160,"available":96556654592,"internally_reserved":0,"allocated":29171712,"data_stored":29000902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":50917071},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315268,"kb_used_data":167028,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94056572,"statfs":{"total":96636764160,"available":96313929728,"internally_reserved":0,"allocated":171036672,"data_stored":170865862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":151776974},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,0,1,1,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":171773952,"data_stored":171773952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29556736,"data_stored":29556736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":171421696,"data_stored":171421696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":178569216,"data_stored":178569216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:49.625 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:41:49.625 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2734", 'reported_seq': 2773, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:48.032338+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:41:48.032338+0000', 'last_peered': '2024-09-16T04:41:48.032338+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:41:48.032338+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2734, 'log_dups_size': 0, 'ondisk_log_size': 2734, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22396928, 'num_objects': 2734, 'num_object_clones': 0, 'num_object_copies': 10936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2734, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2734, 'num_write_kb': 21872, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 515}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:49.625 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:41:49.625 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2617", 'reported_seq': 2677, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:43.872429+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:41:43.872429+0000', 'last_peered': '2024-09-16T04:41:43.872429+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:41:43.872429+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2617, 'log_dups_size': 0, 'ondisk_log_size': 2617, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21402892, 'num_objects': 2613, 'num_object_clones': 0, 'num_object_copies': 10452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2615, 'num_write_kb': 20916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 402}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:49.625 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:41:49.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2634", 'reported_seq': 2704, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:43.536594+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:41:43.536594+0000', 'last_peered': '2024-09-16T04:41:43.536594+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:41:43.536594+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2634, 'log_dups_size': 0, 'ondisk_log_size': 2634, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21551750, 'num_objects': 2631, 'num_object_clones': 0, 'num_object_copies': 10524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2631, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2633, 'num_write_kb': 21054, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 404}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:49.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:41:49.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2635", 'reported_seq': 2696, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:43.918139+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:41:43.918139+0000', 'last_peered': '2024-09-16T04:41:43.918139+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:41:43.918139+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2635, 'log_dups_size': 0, 'ondisk_log_size': 2635, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21550348, 'num_objects': 2631, 'num_object_clones': 0, 'num_object_copies': 10524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2631, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2633, 'num_write_kb': 21053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 402}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:49.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:41:49.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2775", 'reported_seq': 2826, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:47.837010+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:41:47.837010+0000', 'last_peered': '2024-09-16T04:41:47.837010+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:41:47.837010+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2775, 'log_dups_size': 0, 'ondisk_log_size': 2775, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22705420, 'num_objects': 2772, 'num_object_clones': 0, 'num_object_copies': 11088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2773, 'num_write_kb': 22181, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 489}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:49.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:41:49.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2689", 'reported_seq': 2728, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:48.050059+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:41:48.050059+0000', 'last_peered': '2024-09-16T04:41:48.050059+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:41:48.050059+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2689, 'log_dups_size': 0, 'ondisk_log_size': 2689, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22028288, 'num_objects': 2689, 'num_object_clones': 0, 'num_object_copies': 10756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2689, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2689, 'num_write_kb': 21512, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 481}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:49.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:41:49.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2677", 'reported_seq': 2737, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:43.791473+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:41:43.791473+0000', 'last_peered': '2024-09-16T04:41:43.791473+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:41:43.791473+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2677, 'log_dups_size': 0, 'ondisk_log_size': 2677, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21878028, 'num_objects': 2671, 'num_object_clones': 0, 'num_object_copies': 10684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2671, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2675, 'num_write_kb': 21380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 445}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:49.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:41:49.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2665", 'reported_seq': 3142, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:43.920180+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:41:43.920180+0000', 'last_peered': '2024-09-16T04:41:43.920180+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:41:43.920180+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2665, 'log_dups_size': 0, 'ondisk_log_size': 2665, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21787916, 'num_objects': 2660, 'num_object_clones': 0, 'num_object_copies': 10640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2029, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2660, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2663, 'num_write_kb': 21292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 211, 'num_bytes_recovered': 1712128, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 631}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:49.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:41:49.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2862", 'reported_seq': 2913, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:48.085591+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:41:48.085591+0000', 'last_peered': '2024-09-16T04:41:48.085591+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:41:48.085591+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2862, 'log_dups_size': 0, 'ondisk_log_size': 2862, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23374360, 'num_objects': 2854, 'num_object_clones': 0, 'num_object_copies': 11416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2854, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2859, 'num_write_kb': 22849, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 514}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:49.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:41:49.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2611", 'reported_seq': 2684, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:43.670992+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:41:43.670992+0000', 'last_peered': '2024-09-16T04:41:43.670992+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:41:43.670992+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2611, 'log_dups_size': 0, 'ondisk_log_size': 2611, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21370124, 'num_objects': 2609, 'num_object_clones': 0, 'num_object_copies': 10436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2609, 'num_write_kb': 20870, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 409}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:49.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:41:49.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2708", 'reported_seq': 2761, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:47.916053+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:41:47.916053+0000', 'last_peered': '2024-09-16T04:41:47.916053+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:41:47.916053+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2708, 'log_dups_size': 0, 'ondisk_log_size': 2708, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22138770, 'num_objects': 2703, 'num_object_clones': 0, 'num_object_copies': 10812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2703, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2705, 'num_write_kb': 21628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 514}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:49.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:41:49.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2634", 'reported_seq': 2669, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:48.000936+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:41:48.000936+0000', 'last_peered': '2024-09-16T04:41:48.000936+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:41:48.000936+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2634, 'log_dups_size': 0, 'ondisk_log_size': 2634, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21568134, 'num_objects': 2633, 'num_object_clones': 0, 'num_object_copies': 10532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2633, 'num_write_kb': 21063, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 505}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:49.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:41:49.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2609", 'reported_seq': 2678, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:43.678157+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:41:43.678157+0000', 'last_peered': '2024-09-16T04:41:43.678157+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:41:43.678157+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2609, 'log_dups_size': 0, 'ondisk_log_size': 2609, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21363334, 'num_objects': 2608, 'num_object_clones': 0, 'num_object_copies': 10432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2608, 'num_write_kb': 20863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 383}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:49.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:41:49.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2720", 'reported_seq': 2769, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:48.094784+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:41:48.094784+0000', 'last_peered': '2024-09-16T04:41:48.094784+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:41:48.094784+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2720, 'log_dups_size': 0, 'ondisk_log_size': 2720, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22246668, 'num_objects': 2716, 'num_object_clones': 0, 'num_object_copies': 10864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2716, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2718, 'num_write_kb': 21740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:49.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:41:49.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2507", 'reported_seq': 2581, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:43.628005+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:41:43.628005+0000', 'last_peered': '2024-09-16T04:41:43.628005+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:41:43.628005+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2507, 'log_dups_size': 0, 'ondisk_log_size': 2507, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20509964, 'num_objects': 2504, 'num_object_clones': 0, 'num_object_copies': 10016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2504, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2505, 'num_write_kb': 20037, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 372}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:49.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:41:49.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2622", 'reported_seq': 2683, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:43.839245+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:41:43.839245+0000', 'last_peered': '2024-09-16T04:41:43.839245+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:41:43.839245+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2622, 'log_dups_size': 0, 'ondisk_log_size': 2622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21435660, 'num_objects': 2617, 'num_object_clones': 0, 'num_object_copies': 10468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2617, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2620, 'num_write_kb': 20948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 378}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:49.628 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:41:50.259 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 56 20 44392 44372 6.18847 2.8125 0.0433447 0.0252323 2024-09-16T04:41:50.352 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:50.352 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35344/170580 objects degraded (20.720%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":122,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":42645,"data_bytes":349308584,"bytes_used":1050521600,"bytes_avail":385496535040,"bytes_total":386547056640,"degraded_objects":35344,"degraded_total":170580,"degraded_ratio":0.2071989682260523,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":89403,"recovering_keys_per_sec":0,"num_objects_recovered":131,"num_bytes_recovered":1073152,"num_keys_recovered":0,"write_bytes_sec":3298393,"read_op_per_sec":0,"write_op_per_sec":402},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T04:41:44.568132+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:41:50.384 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:41:50.384 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:50.946 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:50.946 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:50.975 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-16T04:41:50.570418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":358557352,"num_objects":43774,"num_object_clones":0,"num_object_copies":175096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43801,"num_write_kb":350290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":2195456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43828,"ondisk_log_size":43828,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1063508,"kb_used_data":559700,"kb_used_omap":14,"kb_used_meta":503729,"kb_avail":376423852,"statfs":{"total":386547056640,"available":385458024448,"internally_reserved":0,"allocated":573132800,"data_stored":572449560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":515819057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,14,7,14,11,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41648128,"num_objects":5084,"num_object_clones":0,"num_object_copies":20336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5084,"num_write_kb":40672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1556480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003815"},"pg_stats":[{"pgid":"1.f","version":"31'2734","reported_seq":2773,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.032338+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:48.032338+0000","last_peered":"2024-09-16T04:41:48.032338+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:48.032338+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2734,"log_dups_size":0,"ondisk_log_size":2734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22396928,"num_objects":2734,"num_object_clones":0,"num_object_copies":10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2734,"num_write_kb":21872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":515}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2734","reported_seq":2795,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.865014+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:48.865014+0000","last_peered":"2024-09-16T04:41:48.865014+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:48.865014+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2734,"log_dups_size":0,"ondisk_log_size":2734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22361356,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2732,"num_write_kb":21852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2773","reported_seq":2842,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.693132+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:48.693132+0000","last_peered":"2024-09-16T04:41:48.693132+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:48.693132+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2773,"log_dups_size":0,"ondisk_log_size":2773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22690438,"num_objects":2770,"num_object_clones":0,"num_object_copies":11080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2772,"num_write_kb":22166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":543}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2774","reported_seq":2835,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.907673+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:48.907673+0000","last_peered":"2024-09-16T04:41:48.907673+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:48.907673+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2774,"log_dups_size":0,"ondisk_log_size":2774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22689036,"num_objects":2770,"num_object_clones":0,"num_object_copies":11080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2772,"num_write_kb":22165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":541}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2775","reported_seq":2826,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:47.837010+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:47.837010+0000","last_peered":"2024-09-16T04:41:47.837010+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:47.837010+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2775,"log_dups_size":0,"ondisk_log_size":2775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22705420,"num_objects":2772,"num_object_clones":0,"num_object_copies":11088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2773,"num_write_kb":22181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2689","reported_seq":2728,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.050059+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:48.050059+0000","last_peered":"2024-09-16T04:41:48.050059+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:48.050059+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2689,"log_dups_size":0,"ondisk_log_size":2689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22028288,"num_objects":2689,"num_object_clones":0,"num_object_copies":10756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2689,"num_write_kb":21512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":481}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2788","reported_seq":2847,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.905336+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:48.905336+0000","last_peered":"2024-09-16T04:41:48.905336+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:48.905336+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2788,"log_dups_size":0,"ondisk_log_size":2788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22787340,"num_objects":2782,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":22268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":556}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2803","reported_seq":3398,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.873617+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:48.873617+0000","last_peered":"2024-09-16T04:41:48.873617+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:48.873617+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2803,"log_dups_size":0,"ondisk_log_size":2803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22918412,"num_objects":2798,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2801,"num_write_kb":22396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":2195456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2862","reported_seq":2913,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.085591+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:48.085591+0000","last_peered":"2024-09-16T04:41:48.085591+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:48.085591+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2862,"log_dups_size":0,"ondisk_log_size":2862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23374360,"num_objects":2854,"num_object_clones":0,"num_object_copies":11416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2731","reported_seq":2804,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.580397+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:48.580397+0000","last_peered":"2024-09-16T04:41:48.580397+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:48.580397+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2731,"log_dups_size":0,"ondisk_log_size":2731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22353164,"num_objects":2729,"num_object_clones":0,"num_object_copies":10916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2729,"num_write_kb":21830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":529}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2708","reported_seq":2761,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:47.916053+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:47.916053+0000","last_peered":"2024-09-16T04:41:47.916053+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:47.916053+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2708,"log_dups_size":0,"ondisk_log_size":2708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22138770,"num_objects":2703,"num_object_clones":0,"num_object_copies":10812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2705,"num_write_kb":21628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2634","reported_seq":2669,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.000936+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:48.000936+0000","last_peered":"2024-09-16T04:41:48.000936+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:48.000936+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21568134,"num_objects":2633,"num_object_clones":0,"num_object_copies":10532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":21063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":505}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2730","reported_seq":2799,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.694920+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:48.694920+0000","last_peered":"2024-09-16T04:41:48.694920+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:48.694920+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2730,"log_dups_size":0,"ondisk_log_size":2730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22354566,"num_objects":2729,"num_object_clones":0,"num_object_copies":10916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2729,"num_write_kb":21831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":504}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2720","reported_seq":2769,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.094784+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:48.094784+0000","last_peered":"2024-09-16T04:41:48.094784+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:48.094784+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2720,"log_dups_size":0,"ondisk_log_size":2720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22246668,"num_objects":2716,"num_object_clones":0,"num_object_copies":10864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2718,"num_write_kb":21740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":482}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2625","reported_seq":2699,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.663936+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:48.663936+0000","last_peered":"2024-09-16T04:41:48.663936+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:48.663936+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2625,"log_dups_size":0,"ondisk_log_size":2625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21476620,"num_objects":2622,"num_object_clones":0,"num_object_copies":10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2623,"num_write_kb":20981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":490}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2748","reported_seq":2807,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.903668+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:48.903668+0000","last_peered":"2024-09-16T04:41:48.903668+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:48.903668+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2748,"log_dups_size":0,"ondisk_log_size":2748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22467852,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":21956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":358557352,"num_objects":43774,"num_object_clones":0,"num_object_copies":175096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43801,"num_write_kb":350290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":2195456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":575926272,"data_stored":575926272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43828,"ondisk_log_size":43828,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325700,"kb_used_data":173236,"kb_used_omap":4,"kb_used_meta":152443,"kb_avail":94046140,"statfs":{"total":96636764160,"available":96303247360,"internally_reserved":0,"allocated":177393664,"data_stored":177222854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":156102351},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318804,"kb_used_data":175172,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94053036,"statfs":{"total":96636764160,"available":96310308864,"internally_reserved":0,"allocated":179376128,"data_stored":179205318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108340,"kb_used_data":36388,"kb_used_omap":4,"kb_used_meta":71931,"kb_avail":94263500,"statfs":{"total":96636764160,"available":96525824000,"internally_reserved":0,"allocated":37261312,"data_stored":37090502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":73658063},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310664,"kb_used_data":174904,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94061176,"statfs":{"total":96636764160,"available":96318644224,"internally_reserved":0,"allocated":179101696,"data_stored":178930886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":138997454},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":179884032,"data_stored":179884032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37941248,"data_stored":37941248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179531776,"data_stored":179531776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":178569216,"data_stored":178569216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:51.262 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 57 20 44764 44744 6.13076 2.90625 0.047502 0.0254748 2024-09-16T04:41:52.262 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 58 20 45171 45151 6.07967 3.17969 0.0174743 0.0256847 2024-09-16T04:41:53.262 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 59 20 45569 45549 6.02933 3.10938 0.0391771 0.0259024 2024-09-16T04:41:53.977 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:54.262 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:2024-09-16T04:41:54.262092+0000 min lat: 0.00269411 max lat: 1.73107 avg lat: 0.0261056 2024-09-16T04:41:54.262 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T04:41:54.263 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 60 20 45938 45918 5.9769 2.88281 0.0849229 0.0261056 2024-09-16T04:41:54.503 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:54.504 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:54.530 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-16T04:41:52.571250+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":358557352,"num_objects":43774,"num_object_clones":0,"num_object_copies":175096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43801,"num_write_kb":350290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":2195456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43828,"ondisk_log_size":43828,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1063508,"kb_used_data":559700,"kb_used_omap":14,"kb_used_meta":503729,"kb_avail":376423852,"statfs":{"total":386547056640,"available":385458024448,"internally_reserved":0,"allocated":573132800,"data_stored":572449560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":515819057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,14,7,14,11,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32047104,"num_objects":3912,"num_object_clones":0,"num_object_copies":15648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3912,"num_write_kb":31296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":1089536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004163"},"pg_stats":[{"pgid":"1.f","version":"31'2734","reported_seq":2773,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.032338+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:48.032338+0000","last_peered":"2024-09-16T04:41:48.032338+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:48.032338+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2734,"log_dups_size":0,"ondisk_log_size":2734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22396928,"num_objects":2734,"num_object_clones":0,"num_object_copies":10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2734,"num_write_kb":21872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":515}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2734","reported_seq":2795,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.865014+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:48.865014+0000","last_peered":"2024-09-16T04:41:48.865014+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:48.865014+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2734,"log_dups_size":0,"ondisk_log_size":2734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22361356,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2732,"num_write_kb":21852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2773","reported_seq":2842,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.693132+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:48.693132+0000","last_peered":"2024-09-16T04:41:48.693132+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:48.693132+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2773,"log_dups_size":0,"ondisk_log_size":2773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22690438,"num_objects":2770,"num_object_clones":0,"num_object_copies":11080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2772,"num_write_kb":22166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":543}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2774","reported_seq":2835,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.907673+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:48.907673+0000","last_peered":"2024-09-16T04:41:48.907673+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:48.907673+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2774,"log_dups_size":0,"ondisk_log_size":2774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22689036,"num_objects":2770,"num_object_clones":0,"num_object_copies":11080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2772,"num_write_kb":22165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":541}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2775","reported_seq":2826,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:47.837010+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:47.837010+0000","last_peered":"2024-09-16T04:41:47.837010+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:47.837010+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2775,"log_dups_size":0,"ondisk_log_size":2775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22705420,"num_objects":2772,"num_object_clones":0,"num_object_copies":11088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2773,"num_write_kb":22181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2689","reported_seq":2728,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.050059+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:48.050059+0000","last_peered":"2024-09-16T04:41:48.050059+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:48.050059+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2689,"log_dups_size":0,"ondisk_log_size":2689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22028288,"num_objects":2689,"num_object_clones":0,"num_object_copies":10756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2689,"num_write_kb":21512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":481}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2788","reported_seq":2847,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.905336+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:48.905336+0000","last_peered":"2024-09-16T04:41:48.905336+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:48.905336+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2788,"log_dups_size":0,"ondisk_log_size":2788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22787340,"num_objects":2782,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":22268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":556}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2803","reported_seq":3398,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.873617+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:48.873617+0000","last_peered":"2024-09-16T04:41:48.873617+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:48.873617+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2803,"log_dups_size":0,"ondisk_log_size":2803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22918412,"num_objects":2798,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2801,"num_write_kb":22396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":2195456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2862","reported_seq":2913,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.085591+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:48.085591+0000","last_peered":"2024-09-16T04:41:48.085591+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:48.085591+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2862,"log_dups_size":0,"ondisk_log_size":2862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23374360,"num_objects":2854,"num_object_clones":0,"num_object_copies":11416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2731","reported_seq":2804,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.580397+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:48.580397+0000","last_peered":"2024-09-16T04:41:48.580397+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:48.580397+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2731,"log_dups_size":0,"ondisk_log_size":2731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22353164,"num_objects":2729,"num_object_clones":0,"num_object_copies":10916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2729,"num_write_kb":21830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":529}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2708","reported_seq":2761,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:47.916053+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:47.916053+0000","last_peered":"2024-09-16T04:41:47.916053+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:47.916053+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2708,"log_dups_size":0,"ondisk_log_size":2708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22138770,"num_objects":2703,"num_object_clones":0,"num_object_copies":10812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2705,"num_write_kb":21628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2634","reported_seq":2669,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.000936+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:48.000936+0000","last_peered":"2024-09-16T04:41:48.000936+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:48.000936+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21568134,"num_objects":2633,"num_object_clones":0,"num_object_copies":10532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":21063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":505}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2730","reported_seq":2799,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.694920+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:48.694920+0000","last_peered":"2024-09-16T04:41:48.694920+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:48.694920+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2730,"log_dups_size":0,"ondisk_log_size":2730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22354566,"num_objects":2729,"num_object_clones":0,"num_object_copies":10916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2729,"num_write_kb":21831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":504}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2720","reported_seq":2769,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.094784+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:48.094784+0000","last_peered":"2024-09-16T04:41:48.094784+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:48.094784+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2720,"log_dups_size":0,"ondisk_log_size":2720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22246668,"num_objects":2716,"num_object_clones":0,"num_object_copies":10864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2718,"num_write_kb":21740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":482}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2625","reported_seq":2699,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.663936+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:48.663936+0000","last_peered":"2024-09-16T04:41:48.663936+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:48.663936+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2625,"log_dups_size":0,"ondisk_log_size":2625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21476620,"num_objects":2622,"num_object_clones":0,"num_object_copies":10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2623,"num_write_kb":20981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":490}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2748","reported_seq":2807,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:48.903668+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:48.903668+0000","last_peered":"2024-09-16T04:41:48.903668+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:48.903668+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2748,"log_dups_size":0,"ondisk_log_size":2748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22467852,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":21956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":358557352,"num_objects":43774,"num_object_clones":0,"num_object_copies":175096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43801,"num_write_kb":350290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":2195456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":575926272,"data_stored":575926272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43828,"ondisk_log_size":43828,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325700,"kb_used_data":173236,"kb_used_omap":4,"kb_used_meta":152443,"kb_avail":94046140,"statfs":{"total":96636764160,"available":96303247360,"internally_reserved":0,"allocated":177393664,"data_stored":177222854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":156102351},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318804,"kb_used_data":175172,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94053036,"statfs":{"total":96636764160,"available":96310308864,"internally_reserved":0,"allocated":179376128,"data_stored":179205318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108340,"kb_used_data":36388,"kb_used_omap":4,"kb_used_meta":71931,"kb_avail":94263500,"statfs":{"total":96636764160,"available":96525824000,"internally_reserved":0,"allocated":37261312,"data_stored":37090502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":73658063},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310664,"kb_used_data":174904,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94061176,"statfs":{"total":96636764160,"available":96318644224,"internally_reserved":0,"allocated":179101696,"data_stored":178930886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":138997454},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":179884032,"data_stored":179884032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37941248,"data_stored":37941248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179531776,"data_stored":179531776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":178569216,"data_stored":178569216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:54.531 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:55.135 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:55.135 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:55.162 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-16T04:41:54.572014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":374277800,"num_objects":45693,"num_object_clones":0,"num_object_copies":182772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45720,"num_write_kb":365642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":2736128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45747,"ondisk_log_size":45747,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1082340,"kb_used_data":582692,"kb_used_omap":14,"kb_used_meta":499569,"kb_avail":376405020,"statfs":{"total":386547056640,"available":385438740480,"internally_reserved":0,"allocated":596676608,"data_stored":595993368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":511559217},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,14,16,13,13,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":47767552,"num_objects":5831,"num_object_clones":0,"num_object_copies":23324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5831,"num_write_kb":46648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":1630208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004462"},"pg_stats":[{"pgid":"1.f","version":"31'2837","reported_seq":2875,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.157935+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:53.157935+0000","last_peered":"2024-09-16T04:41:53.157935+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:53.157935+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2837,"log_dups_size":0,"ondisk_log_size":2837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23240704,"num_objects":2837,"num_object_clones":0,"num_object_copies":11348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2837,"num_write_kb":22696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":618}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2865","reported_seq":2926,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.914533+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:53.914533+0000","last_peered":"2024-09-16T04:41:53.914533+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:53.914533+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2865,"log_dups_size":0,"ondisk_log_size":2865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23434508,"num_objects":2861,"num_object_clones":0,"num_object_copies":11444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2863,"num_write_kb":22900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2876","reported_seq":2946,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.651853+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:53.651853+0000","last_peered":"2024-09-16T04:41:53.651853+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:53.651853+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2876,"log_dups_size":0,"ondisk_log_size":2876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23534214,"num_objects":2873,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2875,"num_write_kb":22990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":646}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2896","reported_seq":2956,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.950098+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:53.950098+0000","last_peered":"2024-09-16T04:41:53.950098+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:53.950098+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2896,"log_dups_size":0,"ondisk_log_size":2896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23688460,"num_objects":2892,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2894,"num_write_kb":23141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2888","reported_seq":2939,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.128794+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:53.128794+0000","last_peered":"2024-09-16T04:41:53.128794+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:53.128794+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2888,"log_dups_size":0,"ondisk_log_size":2888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23631116,"num_objects":2885,"num_object_clones":0,"num_object_copies":11540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":23085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":602}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2805","reported_seq":2844,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.120738+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:53.120738+0000","last_peered":"2024-09-16T04:41:53.120738+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:53.120738+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2805,"log_dups_size":0,"ondisk_log_size":2805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22978560,"num_objects":2805,"num_object_clones":0,"num_object_copies":11220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2805,"num_write_kb":22440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2901","reported_seq":2960,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.940131+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:53.940131+0000","last_peered":"2024-09-16T04:41:53.940131+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:53.940131+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2901,"log_dups_size":0,"ondisk_log_size":2901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23713036,"num_objects":2895,"num_object_clones":0,"num_object_copies":11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2899,"num_write_kb":23172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":669}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2906","reported_seq":3633,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.942934+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:53.942934+0000","last_peered":"2024-09-16T04:41:53.942934+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:53.942934+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2906,"log_dups_size":0,"ondisk_log_size":2906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23762188,"num_objects":2901,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2904,"num_write_kb":23220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":2736128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":997}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2990","reported_seq":3041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.112789+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:53.112789+0000","last_peered":"2024-09-16T04:41:53.112789+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:53.112789+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2990,"log_dups_size":0,"ondisk_log_size":2990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24422936,"num_objects":2982,"num_object_clones":0,"num_object_copies":11928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":23873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":642}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2861","reported_seq":2934,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.606235+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:53.606235+0000","last_peered":"2024-09-16T04:41:53.606235+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:53.606235+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2861,"log_dups_size":0,"ondisk_log_size":2861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23418124,"num_objects":2859,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":659}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2856","reported_seq":2909,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.080526+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:53.080526+0000","last_peered":"2024-09-16T04:41:53.080526+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:53.080526+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2856,"log_dups_size":0,"ondisk_log_size":2856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23351186,"num_objects":2851,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2853,"num_write_kb":22812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":662}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2750","reported_seq":2785,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.125129+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:53.125129+0000","last_peered":"2024-09-16T04:41:53.125129+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:53.125129+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2750,"log_dups_size":0,"ondisk_log_size":2750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22518406,"num_objects":2749,"num_object_clones":0,"num_object_copies":10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":21991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":621}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2851","reported_seq":2920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.670849+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:53.670849+0000","last_peered":"2024-09-16T04:41:53.670849+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:53.670849+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2851,"log_dups_size":0,"ondisk_log_size":2851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23345798,"num_objects":2850,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2850,"num_write_kb":22799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":625}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2855","reported_seq":2904,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.118033+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:53.118033+0000","last_peered":"2024-09-16T04:41:53.118033+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:53.118033+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2855,"log_dups_size":0,"ondisk_log_size":2855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23352588,"num_objects":2851,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2853,"num_write_kb":22820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":617}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2746","reported_seq":2820,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.535265+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:53.535265+0000","last_peered":"2024-09-16T04:41:53.535265+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:53.535265+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2746,"log_dups_size":0,"ondisk_log_size":2746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22467852,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2744,"num_write_kb":21949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":611}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2864","reported_seq":2925,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.934441+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:53.934441+0000","last_peered":"2024-09-16T04:41:53.934441+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:53.934441+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2864,"log_dups_size":0,"ondisk_log_size":2864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23418124,"num_objects":2859,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2862,"num_write_kb":22884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":374277800,"num_objects":45693,"num_object_clones":0,"num_object_copies":182772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45720,"num_write_kb":365642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":2736128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":599683072,"data_stored":599683072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45747,"ondisk_log_size":45747,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320920,"kb_used_data":180936,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94050920,"statfs":{"total":96636764160,"available":96308142080,"internally_reserved":0,"allocated":185278464,"data_stored":185107654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143322831},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,8,7,4,1,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330536,"kb_used_data":182744,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94041304,"statfs":{"total":96636764160,"available":96298295296,"internally_reserved":0,"allocated":187129856,"data_stored":186959046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,3,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108340,"kb_used_data":36388,"kb_used_omap":4,"kb_used_meta":71931,"kb_avail":94263500,"statfs":{"total":96636764160,"available":96525824000,"internally_reserved":0,"allocated":37261312,"data_stored":37090502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":73658063},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322544,"kb_used_data":182624,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94049296,"statfs":{"total":96636764160,"available":96306479104,"internally_reserved":0,"allocated":187006976,"data_stored":186836166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":143257294},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":187813888,"data_stored":187813888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37941248,"data_stored":37941248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":187383808,"data_stored":187383808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":186544128,"data_stored":186544128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:55.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:41:55.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2837", 'reported_seq': 2875, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.157935+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:41:53.157935+0000', 'last_peered': '2024-09-16T04:41:53.157935+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:41:53.157935+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2837, 'log_dups_size': 0, 'ondisk_log_size': 2837, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23240704, 'num_objects': 2837, 'num_object_clones': 0, 'num_object_copies': 11348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2837, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2837, 'num_write_kb': 22696, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 618}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:55.164 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:41:55.164 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2865", 'reported_seq': 2926, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.914533+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:41:53.914533+0000', 'last_peered': '2024-09-16T04:41:53.914533+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:41:53.914533+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2865, 'log_dups_size': 0, 'ondisk_log_size': 2865, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23434508, 'num_objects': 2861, 'num_object_clones': 0, 'num_object_copies': 11444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2863, 'num_write_kb': 22900, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 650}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:55.164 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:41:55.164 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2876", 'reported_seq': 2946, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.651853+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:41:53.651853+0000', 'last_peered': '2024-09-16T04:41:53.651853+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:41:53.651853+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2876, 'log_dups_size': 0, 'ondisk_log_size': 2876, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23534214, 'num_objects': 2873, 'num_object_clones': 0, 'num_object_copies': 11492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2875, 'num_write_kb': 22990, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 646}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:55.164 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:41:55.164 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2896", 'reported_seq': 2956, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.950098+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:41:53.950098+0000', 'last_peered': '2024-09-16T04:41:53.950098+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:41:53.950098+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2896, 'log_dups_size': 0, 'ondisk_log_size': 2896, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23688460, 'num_objects': 2892, 'num_object_clones': 0, 'num_object_copies': 11568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2892, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2894, 'num_write_kb': 23141, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 663}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:55.164 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:41:55.164 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2888", 'reported_seq': 2939, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.128794+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:41:53.128794+0000', 'last_peered': '2024-09-16T04:41:53.128794+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:41:53.128794+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2888, 'log_dups_size': 0, 'ondisk_log_size': 2888, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23631116, 'num_objects': 2885, 'num_object_clones': 0, 'num_object_copies': 11540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2885, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2886, 'num_write_kb': 23085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 602}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:55.164 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:41:55.164 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2805", 'reported_seq': 2844, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.120738+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:41:53.120738+0000', 'last_peered': '2024-09-16T04:41:53.120738+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:41:53.120738+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2805, 'log_dups_size': 0, 'ondisk_log_size': 2805, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22978560, 'num_objects': 2805, 'num_object_clones': 0, 'num_object_copies': 11220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2805, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2805, 'num_write_kb': 22440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 597}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:55.165 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:41:55.165 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2901", 'reported_seq': 2960, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.940131+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:41:53.940131+0000', 'last_peered': '2024-09-16T04:41:53.940131+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:41:53.940131+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2901, 'log_dups_size': 0, 'ondisk_log_size': 2901, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23713036, 'num_objects': 2895, 'num_object_clones': 0, 'num_object_copies': 11580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2895, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2899, 'num_write_kb': 23172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 669}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:55.165 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:41:55.165 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2906", 'reported_seq': 3633, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.942934+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:41:53.942934+0000', 'last_peered': '2024-09-16T04:41:53.942934+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:41:53.942934+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2906, 'log_dups_size': 0, 'ondisk_log_size': 2906, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23762188, 'num_objects': 2901, 'num_object_clones': 0, 'num_object_copies': 11604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2901, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2904, 'num_write_kb': 23220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 336, 'num_bytes_recovered': 2736128, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 997}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:55.165 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:41:55.165 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2990", 'reported_seq': 3041, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.112789+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:41:53.112789+0000', 'last_peered': '2024-09-16T04:41:53.112789+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:41:53.112789+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2990, 'log_dups_size': 0, 'ondisk_log_size': 2990, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24422936, 'num_objects': 2982, 'num_object_clones': 0, 'num_object_copies': 11928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2987, 'num_write_kb': 23873, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 642}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:55.165 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:41:55.165 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2861", 'reported_seq': 2934, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.606235+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:41:53.606235+0000', 'last_peered': '2024-09-16T04:41:53.606235+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:41:53.606235+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2861, 'log_dups_size': 0, 'ondisk_log_size': 2861, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23418124, 'num_objects': 2859, 'num_object_clones': 0, 'num_object_copies': 11436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2859, 'num_write_kb': 22870, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 659}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:55.165 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:41:55.166 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2856", 'reported_seq': 2909, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.080526+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:41:53.080526+0000', 'last_peered': '2024-09-16T04:41:53.080526+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:41:53.080526+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2856, 'log_dups_size': 0, 'ondisk_log_size': 2856, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23351186, 'num_objects': 2851, 'num_object_clones': 0, 'num_object_copies': 11404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2853, 'num_write_kb': 22812, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 662}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:55.166 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:41:55.166 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2750", 'reported_seq': 2785, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.125129+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:41:53.125129+0000', 'last_peered': '2024-09-16T04:41:53.125129+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:41:53.125129+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2750, 'log_dups_size': 0, 'ondisk_log_size': 2750, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22518406, 'num_objects': 2749, 'num_object_clones': 0, 'num_object_copies': 10996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2749, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2749, 'num_write_kb': 21991, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 621}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:55.166 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:41:55.166 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2851", 'reported_seq': 2920, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.670849+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:41:53.670849+0000', 'last_peered': '2024-09-16T04:41:53.670849+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:41:53.670849+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2851, 'log_dups_size': 0, 'ondisk_log_size': 2851, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23345798, 'num_objects': 2850, 'num_object_clones': 0, 'num_object_copies': 11400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2850, 'num_write_kb': 22799, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 625}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:55.166 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:41:55.166 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2855", 'reported_seq': 2904, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.118033+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:41:53.118033+0000', 'last_peered': '2024-09-16T04:41:53.118033+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:41:53.118033+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2855, 'log_dups_size': 0, 'ondisk_log_size': 2855, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23352588, 'num_objects': 2851, 'num_object_clones': 0, 'num_object_copies': 11404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2853, 'num_write_kb': 22820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 617}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:41:55.166 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:41:55.166 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2746", 'reported_seq': 2820, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.535265+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:41:53.535265+0000', 'last_peered': '2024-09-16T04:41:53.535265+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:41:53.535265+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2746, 'log_dups_size': 0, 'ondisk_log_size': 2746, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22467852, 'num_objects': 2743, 'num_object_clones': 0, 'num_object_copies': 10972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2744, 'num_write_kb': 21949, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 611}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:41:55.166 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:41:55.167 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2864", 'reported_seq': 2925, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:53.934441+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:41:53.934441+0000', 'last_peered': '2024-09-16T04:41:53.934441+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:41:53.934441+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2864, 'log_dups_size': 0, 'ondisk_log_size': 2864, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23418124, 'num_objects': 2859, 'num_object_clones': 0, 'num_object_copies': 11436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2862, 'num_write_kb': 22884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 620}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:41:55.167 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:41:55.262 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 61 20 46302 46282 5.92554 2.84375 0.0502671 0.0263482 2024-09-16T04:41:55.875 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:55.875 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35219/182772 objects degraded (19.269%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":128,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":45693,"data_bytes":374277800,"bytes_used":1108316160,"bytes_avail":385438740480,"bytes_total":386547056640,"degraded_objects":35219,"degraded_total":182772,"degraded_ratio":0.19269362922110608,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":135800,"recovering_keys_per_sec":0,"num_objects_recovered":199,"num_bytes_recovered":1630208,"num_keys_recovered":0,"write_bytes_sec":3979149,"read_op_per_sec":0,"write_op_per_sec":485},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T04:41:44.568132+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:41:55.899 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:41:55.899 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:56.262 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 62 20 46659 46639 5.87496 2.78906 0.0353868 0.026573 2024-09-16T04:41:56.493 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:41:56.494 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:41:56.516 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-16T04:41:54.572014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":374277800,"num_objects":45693,"num_object_clones":0,"num_object_copies":182772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45720,"num_write_kb":365642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":2736128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45747,"ondisk_log_size":45747,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1082340,"kb_used_data":582692,"kb_used_omap":14,"kb_used_meta":499569,"kb_avail":376405020,"statfs":{"total":386547056640,"available":385438740480,"internally_reserved":0,"allocated":596676608,"data_stored":595993368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":511559217},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,14,16,13,13,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":47767552,"num_objects":5831,"num_object_clones":0,"num_object_copies":23324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5831,"num_write_kb":46648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":1630208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004462"},"pg_stats":[{"pgid":"1.f","version":"31'2837","reported_seq":2875,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.157935+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:53.157935+0000","last_peered":"2024-09-16T04:41:53.157935+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:53.157935+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2837,"log_dups_size":0,"ondisk_log_size":2837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23240704,"num_objects":2837,"num_object_clones":0,"num_object_copies":11348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2837,"num_write_kb":22696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":618}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2865","reported_seq":2926,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.914533+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:53.914533+0000","last_peered":"2024-09-16T04:41:53.914533+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:53.914533+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2865,"log_dups_size":0,"ondisk_log_size":2865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23434508,"num_objects":2861,"num_object_clones":0,"num_object_copies":11444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2863,"num_write_kb":22900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2876","reported_seq":2946,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.651853+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:53.651853+0000","last_peered":"2024-09-16T04:41:53.651853+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:53.651853+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2876,"log_dups_size":0,"ondisk_log_size":2876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23534214,"num_objects":2873,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2875,"num_write_kb":22990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":646}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2896","reported_seq":2956,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.950098+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:53.950098+0000","last_peered":"2024-09-16T04:41:53.950098+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:53.950098+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2896,"log_dups_size":0,"ondisk_log_size":2896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23688460,"num_objects":2892,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2894,"num_write_kb":23141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2888","reported_seq":2939,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.128794+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:53.128794+0000","last_peered":"2024-09-16T04:41:53.128794+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:53.128794+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2888,"log_dups_size":0,"ondisk_log_size":2888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23631116,"num_objects":2885,"num_object_clones":0,"num_object_copies":11540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":23085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":602}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2805","reported_seq":2844,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.120738+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:53.120738+0000","last_peered":"2024-09-16T04:41:53.120738+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:53.120738+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2805,"log_dups_size":0,"ondisk_log_size":2805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22978560,"num_objects":2805,"num_object_clones":0,"num_object_copies":11220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2805,"num_write_kb":22440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2901","reported_seq":2960,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.940131+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:53.940131+0000","last_peered":"2024-09-16T04:41:53.940131+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:53.940131+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2901,"log_dups_size":0,"ondisk_log_size":2901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23713036,"num_objects":2895,"num_object_clones":0,"num_object_copies":11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2899,"num_write_kb":23172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":669}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2906","reported_seq":3633,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.942934+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:53.942934+0000","last_peered":"2024-09-16T04:41:53.942934+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:53.942934+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2906,"log_dups_size":0,"ondisk_log_size":2906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23762188,"num_objects":2901,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2904,"num_write_kb":23220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":2736128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":997}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2990","reported_seq":3041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.112789+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:53.112789+0000","last_peered":"2024-09-16T04:41:53.112789+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:53.112789+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2990,"log_dups_size":0,"ondisk_log_size":2990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24422936,"num_objects":2982,"num_object_clones":0,"num_object_copies":11928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":23873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":642}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2861","reported_seq":2934,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.606235+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:53.606235+0000","last_peered":"2024-09-16T04:41:53.606235+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:53.606235+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2861,"log_dups_size":0,"ondisk_log_size":2861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23418124,"num_objects":2859,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":659}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2856","reported_seq":2909,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.080526+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:53.080526+0000","last_peered":"2024-09-16T04:41:53.080526+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:53.080526+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2856,"log_dups_size":0,"ondisk_log_size":2856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23351186,"num_objects":2851,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2853,"num_write_kb":22812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":662}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2750","reported_seq":2785,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.125129+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:53.125129+0000","last_peered":"2024-09-16T04:41:53.125129+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:53.125129+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2750,"log_dups_size":0,"ondisk_log_size":2750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22518406,"num_objects":2749,"num_object_clones":0,"num_object_copies":10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":21991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":621}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2851","reported_seq":2920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.670849+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:53.670849+0000","last_peered":"2024-09-16T04:41:53.670849+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:53.670849+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2851,"log_dups_size":0,"ondisk_log_size":2851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23345798,"num_objects":2850,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2850,"num_write_kb":22799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":625}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2855","reported_seq":2904,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.118033+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:53.118033+0000","last_peered":"2024-09-16T04:41:53.118033+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:53.118033+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2855,"log_dups_size":0,"ondisk_log_size":2855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23352588,"num_objects":2851,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2853,"num_write_kb":22820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":617}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2746","reported_seq":2820,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.535265+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:53.535265+0000","last_peered":"2024-09-16T04:41:53.535265+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:53.535265+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2746,"log_dups_size":0,"ondisk_log_size":2746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22467852,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2744,"num_write_kb":21949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":611}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2864","reported_seq":2925,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.934441+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:53.934441+0000","last_peered":"2024-09-16T04:41:53.934441+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:53.934441+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2864,"log_dups_size":0,"ondisk_log_size":2864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23418124,"num_objects":2859,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2862,"num_write_kb":22884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":374277800,"num_objects":45693,"num_object_clones":0,"num_object_copies":182772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45720,"num_write_kb":365642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":2736128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":599683072,"data_stored":599683072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45747,"ondisk_log_size":45747,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320920,"kb_used_data":180936,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94050920,"statfs":{"total":96636764160,"available":96308142080,"internally_reserved":0,"allocated":185278464,"data_stored":185107654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143322831},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,8,7,4,1,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330536,"kb_used_data":182744,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94041304,"statfs":{"total":96636764160,"available":96298295296,"internally_reserved":0,"allocated":187129856,"data_stored":186959046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,3,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108340,"kb_used_data":36388,"kb_used_omap":4,"kb_used_meta":71931,"kb_avail":94263500,"statfs":{"total":96636764160,"available":96525824000,"internally_reserved":0,"allocated":37261312,"data_stored":37090502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":73658063},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322544,"kb_used_data":182624,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94049296,"statfs":{"total":96636764160,"available":96306479104,"internally_reserved":0,"allocated":187006976,"data_stored":186836166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":143257294},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":187813888,"data_stored":187813888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37941248,"data_stored":37941248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":187383808,"data_stored":187383808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":186544128,"data_stored":186544128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:57.262 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 63 20 47045 47025 5.82958 3.01562 0.0518877 0.0267863 2024-09-16T04:41:58.263 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 64 20 47450 47430 5.78794 3.16406 0.0453502 0.0269789 2024-09-16T04:41:59.263 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 65 20 47843 47823 5.74614 3.07031 0.0483585 0.0271768 2024-09-16T04:41:59.518 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:00.075 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:00.075 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:00.109 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-16T04:41:58.573630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":381003432,"num_objects":46514,"num_object_clones":0,"num_object_copies":186056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46541,"num_write_kb":372210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":2736128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46568,"ondisk_log_size":46568,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1142128,"kb_used_data":598064,"kb_used_omap":14,"kb_used_meta":543985,"kb_avail":376345232,"statfs":{"total":386547056640,"available":385377517568,"internally_reserved":0,"allocated":612417536,"data_stored":611734296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":557041201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,10,8,10,13,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38608896,"num_objects":4713,"num_object_clones":0,"num_object_copies":18852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4713,"num_write_kb":37704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004802"},"pg_stats":[{"pgid":"1.f","version":"31'2962","reported_seq":3001,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.146345+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:58.146345+0000","last_peered":"2024-09-16T04:41:58.146345+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:58.146345+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24264704,"num_objects":2962,"num_object_clones":0,"num_object_copies":11848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2962,"num_write_kb":23696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2865","reported_seq":2926,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.914533+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:53.914533+0000","last_peered":"2024-09-16T04:41:53.914533+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:53.914533+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2865,"log_dups_size":0,"ondisk_log_size":2865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23434508,"num_objects":2861,"num_object_clones":0,"num_object_copies":11444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2863,"num_write_kb":22900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2876","reported_seq":2946,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.651853+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:53.651853+0000","last_peered":"2024-09-16T04:41:53.651853+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:53.651853+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2876,"log_dups_size":0,"ondisk_log_size":2876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23534214,"num_objects":2873,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2875,"num_write_kb":22990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":646}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2896","reported_seq":2956,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.950098+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:53.950098+0000","last_peered":"2024-09-16T04:41:53.950098+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:53.950098+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2896,"log_dups_size":0,"ondisk_log_size":2896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23688460,"num_objects":2892,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2894,"num_write_kb":23141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3022","reported_seq":3073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.170892+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:58.170892+0000","last_peered":"2024-09-16T04:41:58.170892+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:58.170892+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3022,"log_dups_size":0,"ondisk_log_size":3022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24728844,"num_objects":3019,"num_object_clones":0,"num_object_copies":12076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":24157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2922","reported_seq":2961,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.124593+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:58.124593+0000","last_peered":"2024-09-16T04:41:58.124593+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:58.124593+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2922,"log_dups_size":0,"ondisk_log_size":2922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23937024,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2922,"num_write_kb":23376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":714}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2901","reported_seq":2960,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.940131+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:53.940131+0000","last_peered":"2024-09-16T04:41:53.940131+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:53.940131+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2901,"log_dups_size":0,"ondisk_log_size":2901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23713036,"num_objects":2895,"num_object_clones":0,"num_object_copies":11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2899,"num_write_kb":23172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":669}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2906","reported_seq":3633,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.942934+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:53.942934+0000","last_peered":"2024-09-16T04:41:53.942934+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:53.942934+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2906,"log_dups_size":0,"ondisk_log_size":2906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23762188,"num_objects":2901,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2904,"num_write_kb":23220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":2736128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":997}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3078","reported_seq":3130,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.201123+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:58.201123+0000","last_peered":"2024-09-16T04:41:58.201123+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:58.201123+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3078,"log_dups_size":0,"ondisk_log_size":3078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25143832,"num_objects":3070,"num_object_clones":0,"num_object_copies":12280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3075,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":730}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2861","reported_seq":2934,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.606235+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:53.606235+0000","last_peered":"2024-09-16T04:41:53.606235+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:53.606235+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2861,"log_dups_size":0,"ondisk_log_size":2861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23418124,"num_objects":2859,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":659}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2980","reported_seq":3033,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.167626+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:58.167626+0000","last_peered":"2024-09-16T04:41:58.167626+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:58.167626+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2980,"log_dups_size":0,"ondisk_log_size":2980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24366994,"num_objects":2975,"num_object_clones":0,"num_object_copies":11900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2977,"num_write_kb":23804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2865","reported_seq":2900,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.159603+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:58.159603+0000","last_peered":"2024-09-16T04:41:58.159603+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:58.159603+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2865,"log_dups_size":0,"ondisk_log_size":2865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23460486,"num_objects":2864,"num_object_clones":0,"num_object_copies":11456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2864,"num_write_kb":22911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":736}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2851","reported_seq":2920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.670849+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:53.670849+0000","last_peered":"2024-09-16T04:41:53.670849+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:53.670849+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2851,"log_dups_size":0,"ondisk_log_size":2851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23345798,"num_objects":2850,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2850,"num_write_kb":22799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":625}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2973","reported_seq":3022,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.119923+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:58.119923+0000","last_peered":"2024-09-16T04:41:58.119923+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:58.119923+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24319244,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":23764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":735}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2746","reported_seq":2820,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.535265+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:53.535265+0000","last_peered":"2024-09-16T04:41:53.535265+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:53.535265+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2746,"log_dups_size":0,"ondisk_log_size":2746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22467852,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2744,"num_write_kb":21949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":611}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2864","reported_seq":2925,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:53.934441+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:53.934441+0000","last_peered":"2024-09-16T04:41:53.934441+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:53.934441+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2864,"log_dups_size":0,"ondisk_log_size":2864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23418124,"num_objects":2859,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2862,"num_write_kb":22884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":381003432,"num_objects":46514,"num_object_clones":0,"num_object_copies":186056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46541,"num_write_kb":372210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":2736128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":615645184,"data_stored":615645184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46568,"ondisk_log_size":46568,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350652,"kb_used_data":188460,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94021188,"statfs":{"total":96636764160,"available":96277696512,"internally_reserved":0,"allocated":192983040,"data_stored":192812230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":166063823},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3,7,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330536,"kb_used_data":182744,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94041304,"statfs":{"total":96636764160,"available":96298295296,"internally_reserved":0,"allocated":187129856,"data_stored":186959046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,3,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051591,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138396,"kb_used_data":44236,"kb_used_omap":4,"kb_used_meta":94139,"kb_avail":94233444,"statfs":{"total":96636764160,"available":96495046656,"internally_reserved":0,"allocated":45297664,"data_stored":45126854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":96399055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322544,"kb_used_data":182624,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94049296,"statfs":{"total":96636764160,"available":96306479104,"internally_reserved":0,"allocated":187006976,"data_stored":186836166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":143257294},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":187813888,"data_stored":187813888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46174208,"data_stored":46174208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":187383808,"data_stored":187383808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":194273280,"data_stored":194273280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:00.110 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:00.263 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 66 20 48207 48187 5.70217 2.84375 0.0440458 0.0273842 2024-09-16T04:42:00.741 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:00.742 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:00.764 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-16T04:42:00.577772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":389752488,"num_objects":47582,"num_object_clones":0,"num_object_copies":190328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47609,"num_write_kb":380754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":424,"num_bytes_recovered":3252224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47636,"ondisk_log_size":47636,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1193344,"kb_used_data":613184,"kb_used_omap":14,"kb_used_meta":580081,"kb_avail":376294016,"statfs":{"total":386547056640,"available":385325072384,"internally_reserved":0,"allocated":627900416,"data_stored":627217176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":594003505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,8,12,13,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40443904,"num_objects":4937,"num_object_clones":0,"num_object_copies":19748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4937,"num_write_kb":39496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008193"},"pg_stats":[{"pgid":"1.f","version":"31'2962","reported_seq":3001,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.146345+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:58.146345+0000","last_peered":"2024-09-16T04:41:58.146345+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:58.146345+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24264704,"num_objects":2962,"num_object_clones":0,"num_object_copies":11848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2962,"num_write_kb":23696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2978","reported_seq":3039,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.756473+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:58.756473+0000","last_peered":"2024-09-16T04:41:58.756473+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:58.756473+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24360204,"num_objects":2974,"num_object_clones":0,"num_object_copies":11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3003","reported_seq":3073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.746437+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:58.746437+0000","last_peered":"2024-09-16T04:41:58.746437+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:58.746437+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3003,"log_dups_size":0,"ondisk_log_size":3003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24574598,"num_objects":3000,"num_object_clones":0,"num_object_copies":12000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3002,"num_write_kb":24006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":773}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3006","reported_seq":3066,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.875287+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:58.875287+0000","last_peered":"2024-09-16T04:41:58.875287+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:58.875287+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3006,"log_dups_size":0,"ondisk_log_size":3006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24589580,"num_objects":3002,"num_object_clones":0,"num_object_copies":12008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3004,"num_write_kb":24021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3022","reported_seq":3073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.170892+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:58.170892+0000","last_peered":"2024-09-16T04:41:58.170892+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:58.170892+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3022,"log_dups_size":0,"ondisk_log_size":3022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24728844,"num_objects":3019,"num_object_clones":0,"num_object_copies":12076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":24157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2922","reported_seq":2961,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.124593+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:58.124593+0000","last_peered":"2024-09-16T04:41:58.124593+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:58.124593+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2922,"log_dups_size":0,"ondisk_log_size":2922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23937024,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2922,"num_write_kb":23376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":714}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3016","reported_seq":3076,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.921906+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:58.921906+0000","last_peered":"2024-09-16T04:41:58.921906+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:58.921906+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3016,"log_dups_size":0,"ondisk_log_size":3016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24655116,"num_objects":3010,"num_object_clones":0,"num_object_copies":12040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3014,"num_write_kb":24092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":784}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3025","reported_seq":3879,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.961503+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:58.961503+0000","last_peered":"2024-09-16T04:41:58.961503+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:58.961503+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3025,"log_dups_size":0,"ondisk_log_size":3025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24737036,"num_objects":3020,"num_object_clones":0,"num_object_copies":12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1841,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3023,"num_write_kb":24172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":399,"num_bytes_recovered":3252224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3078","reported_seq":3130,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.201123+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:58.201123+0000","last_peered":"2024-09-16T04:41:58.201123+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:58.201123+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3078,"log_dups_size":0,"ondisk_log_size":3078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25143832,"num_objects":3070,"num_object_clones":0,"num_object_copies":12280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3075,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":730}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2987","reported_seq":3060,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.692748+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:58.692748+0000","last_peered":"2024-09-16T04:41:58.692748+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:58.692748+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2987,"log_dups_size":0,"ondisk_log_size":2987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24450316,"num_objects":2985,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2985,"num_write_kb":23878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":785}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2980","reported_seq":3033,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.167626+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:58.167626+0000","last_peered":"2024-09-16T04:41:58.167626+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:58.167626+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2980,"log_dups_size":0,"ondisk_log_size":2980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24366994,"num_objects":2975,"num_object_clones":0,"num_object_copies":11900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2977,"num_write_kb":23804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2865","reported_seq":2900,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.159603+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:58.159603+0000","last_peered":"2024-09-16T04:41:58.159603+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:58.159603+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2865,"log_dups_size":0,"ondisk_log_size":2865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23460486,"num_objects":2864,"num_object_clones":0,"num_object_copies":11456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2864,"num_write_kb":22911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":736}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2973","reported_seq":3042,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.613718+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:58.613718+0000","last_peered":"2024-09-16T04:41:58.613718+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:58.613718+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24345222,"num_objects":2972,"num_object_clones":0,"num_object_copies":11888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2972,"num_write_kb":23775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":747}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2973","reported_seq":3022,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.119923+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:58.119923+0000","last_peered":"2024-09-16T04:41:58.119923+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:58.119923+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24319244,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":23764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":735}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2862","reported_seq":2936,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.559003+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:58.559003+0000","last_peered":"2024-09-16T04:41:58.559003+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:58.559003+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2862,"log_dups_size":0,"ondisk_log_size":2862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23418124,"num_objects":2859,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2860,"num_write_kb":22877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":727}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2984","reported_seq":3045,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.877658+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:58.877658+0000","last_peered":"2024-09-16T04:41:58.877658+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:58.877658+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2984,"log_dups_size":0,"ondisk_log_size":2984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24401164,"num_objects":2979,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2982,"num_write_kb":23844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":740}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":389752488,"num_objects":47582,"num_object_clones":0,"num_object_copies":190328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47609,"num_write_kb":380754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":424,"num_bytes_recovered":3252224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":631095296,"data_stored":631095296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47636,"ondisk_log_size":47636,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350652,"kb_used_data":188460,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94021188,"statfs":{"total":96636764160,"available":96277696512,"internally_reserved":0,"allocated":192983040,"data_stored":192812230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":166063823},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3,7,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347688,"kb_used_data":190168,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":94024152,"statfs":{"total":96636764160,"available":96280731648,"internally_reserved":0,"allocated":194732032,"data_stored":194561222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051591,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138396,"kb_used_data":44236,"kb_used_omap":4,"kb_used_meta":94139,"kb_avail":94233444,"statfs":{"total":96636764160,"available":96495046656,"internally_reserved":0,"allocated":45297664,"data_stored":45126854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":96399055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356608,"kb_used_data":190320,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94015232,"statfs":{"total":96636764160,"available":96271597568,"internally_reserved":0,"allocated":194887680,"data_stored":194716870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":170258126},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":195497984,"data_stored":195497984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46174208,"data_stored":46174208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195149824,"data_stored":195149824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":194273280,"data_stored":194273280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:00.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:42:00.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2962", 'reported_seq': 3001, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.146345+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:41:58.146345+0000', 'last_peered': '2024-09-16T04:41:58.146345+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:41:58.146345+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2962, 'log_dups_size': 0, 'ondisk_log_size': 2962, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24264704, 'num_objects': 2962, 'num_object_clones': 0, 'num_object_copies': 11848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2962, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2962, 'num_write_kb': 23696, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 743}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:00.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:42:00.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2978", 'reported_seq': 3039, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.756473+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:41:58.756473+0000', 'last_peered': '2024-09-16T04:41:58.756473+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:41:58.756473+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2978, 'log_dups_size': 0, 'ondisk_log_size': 2978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24360204, 'num_objects': 2974, 'num_object_clones': 0, 'num_object_copies': 11896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2974, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2976, 'num_write_kb': 23804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 763}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:00.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:42:00.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3003", 'reported_seq': 3073, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.746437+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:41:58.746437+0000', 'last_peered': '2024-09-16T04:41:58.746437+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:41:58.746437+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3003, 'log_dups_size': 0, 'ondisk_log_size': 3003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24574598, 'num_objects': 3000, 'num_object_clones': 0, 'num_object_copies': 12000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3000, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3002, 'num_write_kb': 24006, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 773}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:00.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:42:00.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3006", 'reported_seq': 3066, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.875287+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:41:58.875287+0000', 'last_peered': '2024-09-16T04:41:58.875287+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:41:58.875287+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3006, 'log_dups_size': 0, 'ondisk_log_size': 3006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24589580, 'num_objects': 3002, 'num_object_clones': 0, 'num_object_copies': 12008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3004, 'num_write_kb': 24021, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 773}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:00.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:42:00.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3022", 'reported_seq': 3073, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.170892+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:41:58.170892+0000', 'last_peered': '2024-09-16T04:41:58.170892+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:41:58.170892+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3022, 'log_dups_size': 0, 'ondisk_log_size': 3022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24728844, 'num_objects': 3019, 'num_object_clones': 0, 'num_object_copies': 12076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3019, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3020, 'num_write_kb': 24157, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 736}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:42:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2922", 'reported_seq': 2961, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.124593+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:41:58.124593+0000', 'last_peered': '2024-09-16T04:41:58.124593+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:41:58.124593+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2922, 'log_dups_size': 0, 'ondisk_log_size': 2922, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23937024, 'num_objects': 2922, 'num_object_clones': 0, 'num_object_copies': 11688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2922, 'num_write_kb': 23376, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 714}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:42:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3016", 'reported_seq': 3076, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.921906+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:41:58.921906+0000', 'last_peered': '2024-09-16T04:41:58.921906+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:41:58.921906+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3016, 'log_dups_size': 0, 'ondisk_log_size': 3016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24655116, 'num_objects': 3010, 'num_object_clones': 0, 'num_object_copies': 12040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3014, 'num_write_kb': 24092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 784}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:42:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3025", 'reported_seq': 3879, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.961503+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:41:58.961503+0000', 'last_peered': '2024-09-16T04:41:58.961503+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:41:58.961503+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3025, 'log_dups_size': 0, 'ondisk_log_size': 3025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24737036, 'num_objects': 3020, 'num_object_clones': 0, 'num_object_copies': 12080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1841, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3023, 'num_write_kb': 24172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 399, 'num_bytes_recovered': 3252224, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1179}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:42:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3078", 'reported_seq': 3130, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.201123+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:41:58.201123+0000', 'last_peered': '2024-09-16T04:41:58.201123+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:41:58.201123+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3078, 'log_dups_size': 0, 'ondisk_log_size': 3078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25143832, 'num_objects': 3070, 'num_object_clones': 0, 'num_object_copies': 12280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3070, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3075, 'num_write_kb': 24577, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 730}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:42:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2987", 'reported_seq': 3060, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.692748+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:41:58.692748+0000', 'last_peered': '2024-09-16T04:41:58.692748+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:41:58.692748+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2987, 'log_dups_size': 0, 'ondisk_log_size': 2987, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24450316, 'num_objects': 2985, 'num_object_clones': 0, 'num_object_copies': 11940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2985, 'num_write_kb': 23878, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 785}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:42:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2980", 'reported_seq': 3033, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.167626+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:41:58.167626+0000', 'last_peered': '2024-09-16T04:41:58.167626+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:41:58.167626+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2980, 'log_dups_size': 0, 'ondisk_log_size': 2980, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24366994, 'num_objects': 2975, 'num_object_clones': 0, 'num_object_copies': 11900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2977, 'num_write_kb': 23804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 786}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:42:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2865", 'reported_seq': 2900, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.159603+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:41:58.159603+0000', 'last_peered': '2024-09-16T04:41:58.159603+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:41:58.159603+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2865, 'log_dups_size': 0, 'ondisk_log_size': 2865, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23460486, 'num_objects': 2864, 'num_object_clones': 0, 'num_object_copies': 11456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2864, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2864, 'num_write_kb': 22911, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 736}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:42:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2973", 'reported_seq': 3042, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.613718+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:41:58.613718+0000', 'last_peered': '2024-09-16T04:41:58.613718+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:41:58.613718+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2973, 'log_dups_size': 0, 'ondisk_log_size': 2973, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24345222, 'num_objects': 2972, 'num_object_clones': 0, 'num_object_copies': 11888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2972, 'num_write_kb': 23775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 747}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:42:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2973", 'reported_seq': 3022, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.119923+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:41:58.119923+0000', 'last_peered': '2024-09-16T04:41:58.119923+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:41:58.119923+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2973, 'log_dups_size': 0, 'ondisk_log_size': 2973, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24319244, 'num_objects': 2969, 'num_object_clones': 0, 'num_object_copies': 11876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2969, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2971, 'num_write_kb': 23764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 735}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:42:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2862", 'reported_seq': 2936, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.559003+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:41:58.559003+0000', 'last_peered': '2024-09-16T04:41:58.559003+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:41:58.559003+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2862, 'log_dups_size': 0, 'ondisk_log_size': 2862, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23418124, 'num_objects': 2859, 'num_object_clones': 0, 'num_object_copies': 11436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2860, 'num_write_kb': 22877, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 727}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:00.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:42:00.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2984", 'reported_seq': 3045, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:41:58.877658+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:41:58.877658+0000', 'last_peered': '2024-09-16T04:41:58.877658+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:41:58.877658+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2984, 'log_dups_size': 0, 'ondisk_log_size': 2984, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24401164, 'num_objects': 2979, 'num_object_clones': 0, 'num_object_copies': 11916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2982, 'num_write_kb': 23844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 740}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:00.769 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:42:01.263 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 67 20 48560 48540 5.65823 2.75781 0.0625995 0.0275939 2024-09-16T04:42:01.537 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:01.538 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35156/190328 objects degraded (18.471%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":133,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47582,"data_bytes":389752488,"bytes_used":1221984256,"bytes_avail":385325072384,"bytes_total":386547056640,"degraded_objects":35156,"degraded_total":190328,"degraded_ratio":0.18471270648564583,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":128253,"recovering_keys_per_sec":0,"num_objects_recovered":188,"num_bytes_recovered":1540096,"num_keys_recovered":0,"write_bytes_sec":3368025,"read_op_per_sec":0,"write_op_per_sec":411},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T04:41:44.568132+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:42:01.562 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:42:01.562 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:02.122 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:02.122 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:02.155 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-16T04:42:00.577772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":389752488,"num_objects":47582,"num_object_clones":0,"num_object_copies":190328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47609,"num_write_kb":380754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":424,"num_bytes_recovered":3252224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47636,"ondisk_log_size":47636,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1193344,"kb_used_data":613184,"kb_used_omap":14,"kb_used_meta":580081,"kb_avail":376294016,"statfs":{"total":386547056640,"available":385325072384,"internally_reserved":0,"allocated":627900416,"data_stored":627217176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":594003505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,8,12,13,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40443904,"num_objects":4937,"num_object_clones":0,"num_object_copies":19748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4937,"num_write_kb":39496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008193"},"pg_stats":[{"pgid":"1.f","version":"31'2962","reported_seq":3001,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.146345+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:41:58.146345+0000","last_peered":"2024-09-16T04:41:58.146345+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:41:58.146345+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24264704,"num_objects":2962,"num_object_clones":0,"num_object_copies":11848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2962,"num_write_kb":23696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2978","reported_seq":3039,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.756473+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:41:58.756473+0000","last_peered":"2024-09-16T04:41:58.756473+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:41:58.756473+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24360204,"num_objects":2974,"num_object_clones":0,"num_object_copies":11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3003","reported_seq":3073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.746437+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:41:58.746437+0000","last_peered":"2024-09-16T04:41:58.746437+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:41:58.746437+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3003,"log_dups_size":0,"ondisk_log_size":3003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24574598,"num_objects":3000,"num_object_clones":0,"num_object_copies":12000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3002,"num_write_kb":24006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":773}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3006","reported_seq":3066,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.875287+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:41:58.875287+0000","last_peered":"2024-09-16T04:41:58.875287+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:41:58.875287+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3006,"log_dups_size":0,"ondisk_log_size":3006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24589580,"num_objects":3002,"num_object_clones":0,"num_object_copies":12008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3004,"num_write_kb":24021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3022","reported_seq":3073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.170892+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:41:58.170892+0000","last_peered":"2024-09-16T04:41:58.170892+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:41:58.170892+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3022,"log_dups_size":0,"ondisk_log_size":3022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24728844,"num_objects":3019,"num_object_clones":0,"num_object_copies":12076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":24157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2922","reported_seq":2961,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.124593+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:41:58.124593+0000","last_peered":"2024-09-16T04:41:58.124593+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:41:58.124593+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2922,"log_dups_size":0,"ondisk_log_size":2922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23937024,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2922,"num_write_kb":23376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":714}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3016","reported_seq":3076,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.921906+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:41:58.921906+0000","last_peered":"2024-09-16T04:41:58.921906+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:41:58.921906+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3016,"log_dups_size":0,"ondisk_log_size":3016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24655116,"num_objects":3010,"num_object_clones":0,"num_object_copies":12040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3014,"num_write_kb":24092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":784}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3025","reported_seq":3879,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.961503+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:41:58.961503+0000","last_peered":"2024-09-16T04:41:58.961503+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:41:58.961503+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3025,"log_dups_size":0,"ondisk_log_size":3025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24737036,"num_objects":3020,"num_object_clones":0,"num_object_copies":12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1841,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3023,"num_write_kb":24172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":399,"num_bytes_recovered":3252224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3078","reported_seq":3130,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.201123+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:41:58.201123+0000","last_peered":"2024-09-16T04:41:58.201123+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:41:58.201123+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3078,"log_dups_size":0,"ondisk_log_size":3078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25143832,"num_objects":3070,"num_object_clones":0,"num_object_copies":12280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3075,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":730}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2987","reported_seq":3060,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.692748+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:41:58.692748+0000","last_peered":"2024-09-16T04:41:58.692748+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:41:58.692748+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2987,"log_dups_size":0,"ondisk_log_size":2987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24450316,"num_objects":2985,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2985,"num_write_kb":23878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":785}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2980","reported_seq":3033,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.167626+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:41:58.167626+0000","last_peered":"2024-09-16T04:41:58.167626+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:41:58.167626+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2980,"log_dups_size":0,"ondisk_log_size":2980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24366994,"num_objects":2975,"num_object_clones":0,"num_object_copies":11900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2977,"num_write_kb":23804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2865","reported_seq":2900,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.159603+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:41:58.159603+0000","last_peered":"2024-09-16T04:41:58.159603+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:41:58.159603+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2865,"log_dups_size":0,"ondisk_log_size":2865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23460486,"num_objects":2864,"num_object_clones":0,"num_object_copies":11456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2864,"num_write_kb":22911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":736}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2973","reported_seq":3042,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.613718+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:41:58.613718+0000","last_peered":"2024-09-16T04:41:58.613718+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:41:58.613718+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24345222,"num_objects":2972,"num_object_clones":0,"num_object_copies":11888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2972,"num_write_kb":23775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":747}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2973","reported_seq":3022,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.119923+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:41:58.119923+0000","last_peered":"2024-09-16T04:41:58.119923+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:41:58.119923+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24319244,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":23764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":735}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2862","reported_seq":2936,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.559003+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:41:58.559003+0000","last_peered":"2024-09-16T04:41:58.559003+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:41:58.559003+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2862,"log_dups_size":0,"ondisk_log_size":2862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23418124,"num_objects":2859,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2860,"num_write_kb":22877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":727}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2984","reported_seq":3045,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:41:58.877658+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:41:58.877658+0000","last_peered":"2024-09-16T04:41:58.877658+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:41:58.877658+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2984,"log_dups_size":0,"ondisk_log_size":2984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24401164,"num_objects":2979,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2982,"num_write_kb":23844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":740}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":389752488,"num_objects":47582,"num_object_clones":0,"num_object_copies":190328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47609,"num_write_kb":380754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":424,"num_bytes_recovered":3252224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":631095296,"data_stored":631095296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47636,"ondisk_log_size":47636,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350652,"kb_used_data":188460,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94021188,"statfs":{"total":96636764160,"available":96277696512,"internally_reserved":0,"allocated":192983040,"data_stored":192812230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":166063823},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3,7,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347688,"kb_used_data":190168,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":94024152,"statfs":{"total":96636764160,"available":96280731648,"internally_reserved":0,"allocated":194732032,"data_stored":194561222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051591,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138396,"kb_used_data":44236,"kb_used_omap":4,"kb_used_meta":94139,"kb_avail":94233444,"statfs":{"total":96636764160,"available":96495046656,"internally_reserved":0,"allocated":45297664,"data_stored":45126854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":96399055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356608,"kb_used_data":190320,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94015232,"statfs":{"total":96636764160,"available":96271597568,"internally_reserved":0,"allocated":194887680,"data_stored":194716870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":170258126},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":195497984,"data_stored":195497984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46174208,"data_stored":46174208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195149824,"data_stored":195149824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":194273280,"data_stored":194273280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:02.265 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 68 20 48913 48893 5.61544 2.75781 0.0407395 0.027807 2024-09-16T04:42:03.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 69 20 49313 49293 5.5791 3.125 0.0336534 0.0279887 2024-09-16T04:42:04.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 70 20 49705 49685 5.54315 3.0625 0.0557449 0.0281717 2024-09-16T04:42:05.157 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:05.270 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 71 20 50096 50076 5.50811 3.05469 0.0819407 0.0283513 2024-09-16T04:42:05.744 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:05.744 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:05.778 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-09-16T04:42:04.578665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":405292712,"num_objects":49479,"num_object_clones":0,"num_object_copies":197916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49506,"num_write_kb":395930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":3760128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49533,"ondisk_log_size":49533,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1262440,"kb_used_data":644776,"kb_used_omap":14,"kb_used_meta":617585,"kb_avail":376224920,"statfs":{"total":386547056640,"available":385254318080,"internally_reserved":0,"allocated":660250624,"data_stored":659567384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":632407601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,8,16,16,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":46735360,"num_objects":5705,"num_object_clones":0,"num_object_copies":22820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5705,"num_write_kb":45640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":1564672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007415"},"pg_stats":[{"pgid":"1.f","version":"31'3083","reported_seq":3122,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.151283+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:03.151283+0000","last_peered":"2024-09-16T04:42:03.151283+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:03.151283+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25255936,"num_objects":3083,"num_object_clones":0,"num_object_copies":12332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":24664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":864}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3094","reported_seq":3155,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.958875+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:03.958875+0000","last_peered":"2024-09-16T04:42:03.958875+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:03.958875+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3094,"log_dups_size":0,"ondisk_log_size":3094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25310476,"num_objects":3090,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":24732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3116","reported_seq":3186,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.702074+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:03.702074+0000","last_peered":"2024-09-16T04:42:03.702074+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:03.702074+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3116,"log_dups_size":0,"ondisk_log_size":3116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25500294,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3115,"num_write_kb":24910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":886}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3130","reported_seq":3190,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.946603+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:03.946603+0000","last_peered":"2024-09-16T04:42:03.946603+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:03.946603+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3130,"log_dups_size":0,"ondisk_log_size":3130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25605388,"num_objects":3126,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3128,"num_write_kb":25013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":897}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3133","reported_seq":3184,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.191452+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:03.191452+0000","last_peered":"2024-09-16T04:42:03.191452+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:03.191452+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3133,"log_dups_size":0,"ondisk_log_size":3133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25638156,"num_objects":3130,"num_object_clones":0,"num_object_copies":12520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3131,"num_write_kb":25045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":847}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3022","reported_seq":3061,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.200839+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:03.200839+0000","last_peered":"2024-09-16T04:42:03.200839+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:03.200839+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3022,"log_dups_size":0,"ondisk_log_size":3022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24756224,"num_objects":3022,"num_object_clones":0,"num_object_copies":12088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3022,"num_write_kb":24176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":814}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3138","reported_seq":3198,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.987089+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:03.987089+0000","last_peered":"2024-09-16T04:42:03.987089+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:03.987089+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3138,"log_dups_size":0,"ondisk_log_size":3138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25654540,"num_objects":3132,"num_object_clones":0,"num_object_copies":12528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3136,"num_write_kb":25068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":906}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3151","reported_seq":4128,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.984840+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:03.984840+0000","last_peered":"2024-09-16T04:42:03.984840+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:03.984840+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3151,"log_dups_size":0,"ondisk_log_size":3151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25769228,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3149,"num_write_kb":25180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":3760128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3196","reported_seq":3248,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.126193+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:03.126193+0000","last_peered":"2024-09-16T04:42:03.126193+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:03.126193+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3196,"log_dups_size":0,"ondisk_log_size":3196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26110488,"num_objects":3188,"num_object_clones":0,"num_object_copies":12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":848}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3103","reported_seq":3176,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.708724+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:03.708724+0000","last_peered":"2024-09-16T04:42:03.708724+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:03.708724+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3103,"log_dups_size":0,"ondisk_log_size":3103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25400588,"num_objects":3101,"num_object_clones":0,"num_object_copies":12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3101,"num_write_kb":24806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":901}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3101","reported_seq":3148,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.149167+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:03.149167+0000","last_peered":"2024-09-16T04:42:03.149167+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:03.149167+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3101,"log_dups_size":0,"ondisk_log_size":3101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25358226,"num_objects":3096,"num_object_clones":0,"num_object_copies":12384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3098,"num_write_kb":24772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2998","reported_seq":3033,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.144849+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:03.144849+0000","last_peered":"2024-09-16T04:42:03.144849+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:03.144849+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2998,"log_dups_size":0,"ondisk_log_size":2998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24550022,"num_objects":2997,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2997,"num_write_kb":23975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":869}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3080","reported_seq":3149,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.762152+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:03.762152+0000","last_peered":"2024-09-16T04:42:03.762152+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:03.762152+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3080,"log_dups_size":0,"ondisk_log_size":3080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25221766,"num_objects":3079,"num_object_clones":0,"num_object_copies":12316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3079,"num_write_kb":24631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3098","reported_seq":3147,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.143911+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:03.143911+0000","last_peered":"2024-09-16T04:42:03.143911+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:03.143911+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3098,"log_dups_size":0,"ondisk_log_size":3098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25343244,"num_objects":3094,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3096,"num_write_kb":24764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":860}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2991","reported_seq":3065,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.687173+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:03.687173+0000","last_peered":"2024-09-16T04:42:03.687173+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:03.687173+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2991,"log_dups_size":0,"ondisk_log_size":2991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24474892,"num_objects":2988,"num_object_clones":0,"num_object_copies":11952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2989,"num_write_kb":23909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":856}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3099","reported_seq":3159,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.975310+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:03.975310+0000","last_peered":"2024-09-16T04:42:03.975310+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:03.975310+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3099,"log_dups_size":0,"ondisk_log_size":3099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25343244,"num_objects":3094,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3097,"num_write_kb":24764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":405292712,"num_objects":49479,"num_object_clones":0,"num_object_copies":197916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49506,"num_write_kb":395930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":3760128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":662204416,"data_stored":662204416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49533,"ondisk_log_size":49533,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381808,"kb_used_data":197408,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93990032,"statfs":{"total":96636764160,"available":96245792768,"internally_reserved":0,"allocated":202145792,"data_stored":201974982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188804815},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377364,"kb_used_data":197636,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93994476,"statfs":{"total":96636764160,"available":96250343424,"internally_reserved":0,"allocated":202379264,"data_stored":202208454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051592,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":133836,"kb_used_data":52156,"kb_used_omap":4,"kb_used_meta":81659,"kb_avail":94238004,"statfs":{"total":96636764160,"available":96499716096,"internally_reserved":0,"allocated":53407744,"data_stored":53236934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":83619535},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,9,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369432,"kb_used_data":197576,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":94002408,"statfs":{"total":96636764160,"available":96258465792,"internally_reserved":0,"allocated":202317824,"data_stored":202147014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":175959758},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,2,2,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":203165696,"data_stored":203165696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54325248,"data_stored":54325248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":202792960,"data_stored":202792960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":201920512,"data_stored":201920512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:05.779 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:06.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 72 20 50443 50423 5.46927 2.71094 0.0491122 0.0285537 2024-09-16T04:42:06.421 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:06.421 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:06.442 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-09-16T04:42:04.578665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":405292712,"num_objects":49479,"num_object_clones":0,"num_object_copies":197916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49506,"num_write_kb":395930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":3760128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49533,"ondisk_log_size":49533,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1262440,"kb_used_data":644776,"kb_used_omap":14,"kb_used_meta":617585,"kb_avail":376224920,"statfs":{"total":386547056640,"available":385254318080,"internally_reserved":0,"allocated":660250624,"data_stored":659567384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":632407601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,8,16,16,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":46735360,"num_objects":5705,"num_object_clones":0,"num_object_copies":22820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5705,"num_write_kb":45640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":1564672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007415"},"pg_stats":[{"pgid":"1.f","version":"31'3083","reported_seq":3122,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.151283+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:03.151283+0000","last_peered":"2024-09-16T04:42:03.151283+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:03.151283+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25255936,"num_objects":3083,"num_object_clones":0,"num_object_copies":12332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":24664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":864}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3094","reported_seq":3155,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.958875+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:03.958875+0000","last_peered":"2024-09-16T04:42:03.958875+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:03.958875+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3094,"log_dups_size":0,"ondisk_log_size":3094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25310476,"num_objects":3090,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":24732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3116","reported_seq":3186,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.702074+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:03.702074+0000","last_peered":"2024-09-16T04:42:03.702074+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:03.702074+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3116,"log_dups_size":0,"ondisk_log_size":3116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25500294,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3115,"num_write_kb":24910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":886}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3130","reported_seq":3190,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.946603+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:03.946603+0000","last_peered":"2024-09-16T04:42:03.946603+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:03.946603+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3130,"log_dups_size":0,"ondisk_log_size":3130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25605388,"num_objects":3126,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3128,"num_write_kb":25013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":897}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3133","reported_seq":3184,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.191452+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:03.191452+0000","last_peered":"2024-09-16T04:42:03.191452+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:03.191452+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3133,"log_dups_size":0,"ondisk_log_size":3133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25638156,"num_objects":3130,"num_object_clones":0,"num_object_copies":12520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3131,"num_write_kb":25045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":847}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3022","reported_seq":3061,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.200839+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:03.200839+0000","last_peered":"2024-09-16T04:42:03.200839+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:03.200839+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3022,"log_dups_size":0,"ondisk_log_size":3022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24756224,"num_objects":3022,"num_object_clones":0,"num_object_copies":12088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3022,"num_write_kb":24176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":814}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3138","reported_seq":3198,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.987089+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:03.987089+0000","last_peered":"2024-09-16T04:42:03.987089+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:03.987089+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3138,"log_dups_size":0,"ondisk_log_size":3138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25654540,"num_objects":3132,"num_object_clones":0,"num_object_copies":12528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3136,"num_write_kb":25068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":906}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3151","reported_seq":4128,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.984840+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:03.984840+0000","last_peered":"2024-09-16T04:42:03.984840+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:03.984840+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3151,"log_dups_size":0,"ondisk_log_size":3151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25769228,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3149,"num_write_kb":25180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":3760128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3196","reported_seq":3248,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.126193+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:03.126193+0000","last_peered":"2024-09-16T04:42:03.126193+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:03.126193+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3196,"log_dups_size":0,"ondisk_log_size":3196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26110488,"num_objects":3188,"num_object_clones":0,"num_object_copies":12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":848}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3103","reported_seq":3176,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.708724+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:03.708724+0000","last_peered":"2024-09-16T04:42:03.708724+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:03.708724+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3103,"log_dups_size":0,"ondisk_log_size":3103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25400588,"num_objects":3101,"num_object_clones":0,"num_object_copies":12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3101,"num_write_kb":24806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":901}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3101","reported_seq":3148,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.149167+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:03.149167+0000","last_peered":"2024-09-16T04:42:03.149167+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:03.149167+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3101,"log_dups_size":0,"ondisk_log_size":3101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25358226,"num_objects":3096,"num_object_clones":0,"num_object_copies":12384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3098,"num_write_kb":24772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2998","reported_seq":3033,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.144849+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:03.144849+0000","last_peered":"2024-09-16T04:42:03.144849+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:03.144849+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2998,"log_dups_size":0,"ondisk_log_size":2998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24550022,"num_objects":2997,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2997,"num_write_kb":23975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":869}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3080","reported_seq":3149,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.762152+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:03.762152+0000","last_peered":"2024-09-16T04:42:03.762152+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:03.762152+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3080,"log_dups_size":0,"ondisk_log_size":3080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25221766,"num_objects":3079,"num_object_clones":0,"num_object_copies":12316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3079,"num_write_kb":24631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3098","reported_seq":3147,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.143911+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:03.143911+0000","last_peered":"2024-09-16T04:42:03.143911+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:03.143911+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3098,"log_dups_size":0,"ondisk_log_size":3098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25343244,"num_objects":3094,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3096,"num_write_kb":24764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":860}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2991","reported_seq":3065,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.687173+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:03.687173+0000","last_peered":"2024-09-16T04:42:03.687173+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:03.687173+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2991,"log_dups_size":0,"ondisk_log_size":2991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24474892,"num_objects":2988,"num_object_clones":0,"num_object_copies":11952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2989,"num_write_kb":23909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":856}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3099","reported_seq":3159,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.975310+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:03.975310+0000","last_peered":"2024-09-16T04:42:03.975310+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:03.975310+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3099,"log_dups_size":0,"ondisk_log_size":3099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25343244,"num_objects":3094,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3097,"num_write_kb":24764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":405292712,"num_objects":49479,"num_object_clones":0,"num_object_copies":197916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49506,"num_write_kb":395930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":3760128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":662204416,"data_stored":662204416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49533,"ondisk_log_size":49533,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381808,"kb_used_data":197408,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93990032,"statfs":{"total":96636764160,"available":96245792768,"internally_reserved":0,"allocated":202145792,"data_stored":201974982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188804815},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377364,"kb_used_data":197636,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93994476,"statfs":{"total":96636764160,"available":96250343424,"internally_reserved":0,"allocated":202379264,"data_stored":202208454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051592,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":133836,"kb_used_data":52156,"kb_used_omap":4,"kb_used_meta":81659,"kb_avail":94238004,"statfs":{"total":96636764160,"available":96499716096,"internally_reserved":0,"allocated":53407744,"data_stored":53236934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":83619535},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,9,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369432,"kb_used_data":197576,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":94002408,"statfs":{"total":96636764160,"available":96258465792,"internally_reserved":0,"allocated":202317824,"data_stored":202147014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":175959758},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,2,2,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":203165696,"data_stored":203165696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54325248,"data_stored":54325248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":202792960,"data_stored":202792960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":201920512,"data_stored":201920512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:06.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:42:06.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3083", 'reported_seq': 3122, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.151283+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:42:03.151283+0000', 'last_peered': '2024-09-16T04:42:03.151283+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:42:03.151283+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3083, 'log_dups_size': 0, 'ondisk_log_size': 3083, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25255936, 'num_objects': 3083, 'num_object_clones': 0, 'num_object_copies': 12332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3083, 'num_write_kb': 24664, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 864}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:06.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:42:06.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3094", 'reported_seq': 3155, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.958875+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:42:03.958875+0000', 'last_peered': '2024-09-16T04:42:03.958875+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:42:03.958875+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3094, 'log_dups_size': 0, 'ondisk_log_size': 3094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25310476, 'num_objects': 3090, 'num_object_clones': 0, 'num_object_copies': 12360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3090, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3092, 'num_write_kb': 24732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:06.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:42:06.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3116", 'reported_seq': 3186, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.702074+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:42:03.702074+0000', 'last_peered': '2024-09-16T04:42:03.702074+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:42:03.702074+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3116, 'log_dups_size': 0, 'ondisk_log_size': 3116, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25500294, 'num_objects': 3113, 'num_object_clones': 0, 'num_object_copies': 12452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3115, 'num_write_kb': 24910, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 886}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:06.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:42:06.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3130", 'reported_seq': 3190, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.946603+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:42:03.946603+0000', 'last_peered': '2024-09-16T04:42:03.946603+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:42:03.946603+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3130, 'log_dups_size': 0, 'ondisk_log_size': 3130, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25605388, 'num_objects': 3126, 'num_object_clones': 0, 'num_object_copies': 12504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3128, 'num_write_kb': 25013, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:06.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:42:06.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3133", 'reported_seq': 3184, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.191452+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:42:03.191452+0000', 'last_peered': '2024-09-16T04:42:03.191452+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:42:03.191452+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3133, 'log_dups_size': 0, 'ondisk_log_size': 3133, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25638156, 'num_objects': 3130, 'num_object_clones': 0, 'num_object_copies': 12520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3130, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3131, 'num_write_kb': 25045, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 847}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:06.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:42:06.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3022", 'reported_seq': 3061, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.200839+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:42:03.200839+0000', 'last_peered': '2024-09-16T04:42:03.200839+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:42:03.200839+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3022, 'log_dups_size': 0, 'ondisk_log_size': 3022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24756224, 'num_objects': 3022, 'num_object_clones': 0, 'num_object_copies': 12088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3022, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3022, 'num_write_kb': 24176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 814}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:06.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:42:06.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3138", 'reported_seq': 3198, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.987089+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:42:03.987089+0000', 'last_peered': '2024-09-16T04:42:03.987089+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:42:03.987089+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3138, 'log_dups_size': 0, 'ondisk_log_size': 3138, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25654540, 'num_objects': 3132, 'num_object_clones': 0, 'num_object_copies': 12528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3132, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3136, 'num_write_kb': 25068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 906}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:06.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:42:06.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3151", 'reported_seq': 4128, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.984840+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:42:03.984840+0000', 'last_peered': '2024-09-16T04:42:03.984840+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:42:03.984840+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3151, 'log_dups_size': 0, 'ondisk_log_size': 3151, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25769228, 'num_objects': 3146, 'num_object_clones': 0, 'num_object_copies': 12584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1779, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3149, 'num_write_kb': 25180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 461, 'num_bytes_recovered': 3760128, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1367}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:06.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:42:06.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3196", 'reported_seq': 3248, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.126193+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:42:03.126193+0000', 'last_peered': '2024-09-16T04:42:03.126193+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:42:03.126193+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3196, 'log_dups_size': 0, 'ondisk_log_size': 3196, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26110488, 'num_objects': 3188, 'num_object_clones': 0, 'num_object_copies': 12752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3193, 'num_write_kb': 25521, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 848}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:06.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:42:06.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3103", 'reported_seq': 3176, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.708724+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:42:03.708724+0000', 'last_peered': '2024-09-16T04:42:03.708724+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:42:03.708724+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3103, 'log_dups_size': 0, 'ondisk_log_size': 3103, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25400588, 'num_objects': 3101, 'num_object_clones': 0, 'num_object_copies': 12404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3101, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3101, 'num_write_kb': 24806, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 901}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:06.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:42:06.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3101", 'reported_seq': 3148, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.149167+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:42:03.149167+0000', 'last_peered': '2024-09-16T04:42:03.149167+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:42:03.149167+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3101, 'log_dups_size': 0, 'ondisk_log_size': 3101, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25358226, 'num_objects': 3096, 'num_object_clones': 0, 'num_object_copies': 12384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3098, 'num_write_kb': 24772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 907}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:06.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:42:06.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2998", 'reported_seq': 3033, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.144849+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:42:03.144849+0000', 'last_peered': '2024-09-16T04:42:03.144849+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:42:03.144849+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2998, 'log_dups_size': 0, 'ondisk_log_size': 2998, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24550022, 'num_objects': 2997, 'num_object_clones': 0, 'num_object_copies': 11988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2997, 'num_write_kb': 23975, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 869}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:06.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:42:06.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3080", 'reported_seq': 3149, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.762152+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:42:03.762152+0000', 'last_peered': '2024-09-16T04:42:03.762152+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:42:03.762152+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3080, 'log_dups_size': 0, 'ondisk_log_size': 3080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25221766, 'num_objects': 3079, 'num_object_clones': 0, 'num_object_copies': 12316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3079, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3079, 'num_write_kb': 24631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 854}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:06.447 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:42:06.447 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3098", 'reported_seq': 3147, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.143911+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:42:03.143911+0000', 'last_peered': '2024-09-16T04:42:03.143911+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:42:03.143911+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3098, 'log_dups_size': 0, 'ondisk_log_size': 3098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25343244, 'num_objects': 3094, 'num_object_clones': 0, 'num_object_copies': 12376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3094, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3096, 'num_write_kb': 24764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 860}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:06.447 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:42:06.447 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2991", 'reported_seq': 3065, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.687173+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:42:03.687173+0000', 'last_peered': '2024-09-16T04:42:03.687173+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:42:03.687173+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 2991, 'log_dups_size': 0, 'ondisk_log_size': 2991, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24474892, 'num_objects': 2988, 'num_object_clones': 0, 'num_object_copies': 11952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2988, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2989, 'num_write_kb': 23909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 856}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:06.447 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:42:06.447 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3099", 'reported_seq': 3159, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:03.975310+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:42:03.975310+0000', 'last_peered': '2024-09-16T04:42:03.975310+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:42:03.975310+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3099, 'log_dups_size': 0, 'ondisk_log_size': 3099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25343244, 'num_objects': 3094, 'num_object_clones': 0, 'num_object_copies': 12376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3094, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3097, 'num_write_kb': 24764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 855}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:06.447 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:42:07.204 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:07.204 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35094/197916 objects degraded (17.732%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":139,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":49479,"data_bytes":405292712,"bytes_used":1305178112,"bytes_avail":385241878528,"bytes_total":386547056640,"degraded_objects":35094,"degraded_total":197916,"degraded_ratio":0.1773176499120839,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":85282,"recovering_keys_per_sec":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"write_bytes_sec":2583038,"read_op_per_sec":0,"write_op_per_sec":315},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T04:41:44.568132+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:42:07.232 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:42:07.233 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:07.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 73 20 50789 50769 5.43138 2.70312 0.0595402 0.0287502 2024-09-16T04:42:07.803 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:07.804 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:07.832 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-09-16T04:42:06.579155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":405292712,"num_objects":49479,"num_object_clones":0,"num_object_copies":197916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49506,"num_write_kb":395930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":3760128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49533,"ondisk_log_size":49533,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1274588,"kb_used_data":652764,"kb_used_omap":14,"kb_used_meta":621745,"kb_avail":376212772,"statfs":{"total":386547056640,"available":385241878528,"internally_reserved":0,"allocated":668430336,"data_stored":667747096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":636667441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,9,9,10,19,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31014912,"num_objects":3786,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3786,"num_write_kb":30288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007141"},"pg_stats":[{"pgid":"1.f","version":"31'3083","reported_seq":3122,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.151283+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:03.151283+0000","last_peered":"2024-09-16T04:42:03.151283+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:03.151283+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25255936,"num_objects":3083,"num_object_clones":0,"num_object_copies":12332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":24664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":864}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3094","reported_seq":3155,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.958875+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:03.958875+0000","last_peered":"2024-09-16T04:42:03.958875+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:03.958875+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3094,"log_dups_size":0,"ondisk_log_size":3094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25310476,"num_objects":3090,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":24732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3116","reported_seq":3186,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.702074+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:03.702074+0000","last_peered":"2024-09-16T04:42:03.702074+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:03.702074+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3116,"log_dups_size":0,"ondisk_log_size":3116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25500294,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3115,"num_write_kb":24910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":886}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3130","reported_seq":3190,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.946603+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:03.946603+0000","last_peered":"2024-09-16T04:42:03.946603+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:03.946603+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3130,"log_dups_size":0,"ondisk_log_size":3130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25605388,"num_objects":3126,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3128,"num_write_kb":25013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":897}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3133","reported_seq":3184,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.191452+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:03.191452+0000","last_peered":"2024-09-16T04:42:03.191452+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:03.191452+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3133,"log_dups_size":0,"ondisk_log_size":3133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25638156,"num_objects":3130,"num_object_clones":0,"num_object_copies":12520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3131,"num_write_kb":25045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":847}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3022","reported_seq":3061,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.200839+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:03.200839+0000","last_peered":"2024-09-16T04:42:03.200839+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:03.200839+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3022,"log_dups_size":0,"ondisk_log_size":3022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24756224,"num_objects":3022,"num_object_clones":0,"num_object_copies":12088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3022,"num_write_kb":24176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":814}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3138","reported_seq":3198,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.987089+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:03.987089+0000","last_peered":"2024-09-16T04:42:03.987089+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:03.987089+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3138,"log_dups_size":0,"ondisk_log_size":3138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25654540,"num_objects":3132,"num_object_clones":0,"num_object_copies":12528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3136,"num_write_kb":25068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":906}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3151","reported_seq":4128,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.984840+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:03.984840+0000","last_peered":"2024-09-16T04:42:03.984840+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:03.984840+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3151,"log_dups_size":0,"ondisk_log_size":3151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25769228,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3149,"num_write_kb":25180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":3760128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3196","reported_seq":3248,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.126193+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:03.126193+0000","last_peered":"2024-09-16T04:42:03.126193+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:03.126193+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3196,"log_dups_size":0,"ondisk_log_size":3196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26110488,"num_objects":3188,"num_object_clones":0,"num_object_copies":12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":848}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3103","reported_seq":3176,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.708724+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:03.708724+0000","last_peered":"2024-09-16T04:42:03.708724+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:03.708724+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3103,"log_dups_size":0,"ondisk_log_size":3103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25400588,"num_objects":3101,"num_object_clones":0,"num_object_copies":12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3101,"num_write_kb":24806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":901}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3101","reported_seq":3148,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.149167+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:03.149167+0000","last_peered":"2024-09-16T04:42:03.149167+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:03.149167+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3101,"log_dups_size":0,"ondisk_log_size":3101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25358226,"num_objects":3096,"num_object_clones":0,"num_object_copies":12384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3098,"num_write_kb":24772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2998","reported_seq":3033,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.144849+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:03.144849+0000","last_peered":"2024-09-16T04:42:03.144849+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:03.144849+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2998,"log_dups_size":0,"ondisk_log_size":2998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24550022,"num_objects":2997,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2997,"num_write_kb":23975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":869}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3080","reported_seq":3149,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.762152+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:03.762152+0000","last_peered":"2024-09-16T04:42:03.762152+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:03.762152+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3080,"log_dups_size":0,"ondisk_log_size":3080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25221766,"num_objects":3079,"num_object_clones":0,"num_object_copies":12316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3079,"num_write_kb":24631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3098","reported_seq":3147,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.143911+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:03.143911+0000","last_peered":"2024-09-16T04:42:03.143911+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:03.143911+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3098,"log_dups_size":0,"ondisk_log_size":3098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25343244,"num_objects":3094,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3096,"num_write_kb":24764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":860}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2991","reported_seq":3065,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.687173+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:03.687173+0000","last_peered":"2024-09-16T04:42:03.687173+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:03.687173+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":2991,"log_dups_size":0,"ondisk_log_size":2991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24474892,"num_objects":2988,"num_object_clones":0,"num_object_copies":11952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2989,"num_write_kb":23909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":856}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3099","reported_seq":3159,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:03.975310+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:03.975310+0000","last_peered":"2024-09-16T04:42:03.975310+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:03.975310+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3099,"log_dups_size":0,"ondisk_log_size":3099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25343244,"num_objects":3094,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3097,"num_write_kb":24764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":405292712,"num_objects":49479,"num_object_clones":0,"num_object_copies":197916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49506,"num_write_kb":395930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":3760128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":670195712,"data_stored":670195712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49533,"ondisk_log_size":49533,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381808,"kb_used_data":197408,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93990032,"statfs":{"total":96636764160,"available":96245792768,"internally_reserved":0,"allocated":202145792,"data_stored":201974982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188804815},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377364,"kb_used_data":197636,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93994476,"statfs":{"total":96636764160,"available":96250343424,"internally_reserved":0,"allocated":202379264,"data_stored":202208454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051593,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":145984,"kb_used_data":60144,"kb_used_omap":4,"kb_used_meta":85819,"kb_avail":94225856,"statfs":{"total":96636764160,"available":96487276544,"internally_reserved":0,"allocated":61587456,"data_stored":61416646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":87879375},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,3,3,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369432,"kb_used_data":197576,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":94002408,"statfs":{"total":96636764160,"available":96258465792,"internally_reserved":0,"allocated":202317824,"data_stored":202147014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":175959758},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,2,2,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":203165696,"data_stored":203165696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62316544,"data_stored":62316544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":202792960,"data_stored":202792960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":201920512,"data_stored":201920512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:08.268 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 74 20 51149 51129 5.396 2.8125 0.0452171 0.028941 2024-09-16T04:42:09.269 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 75 20 51523 51503 5.36301 2.92188 0.0423966 0.029117 2024-09-16T04:42:10.269 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 76 20 51909 51889 5.33213 3.01562 0.061149 0.0292881 2024-09-16T04:42:10.834 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:11.269 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 77 20 52245 52225 5.29698 2.625 0.0132833 0.0294424 2024-09-16T04:42:11.315 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:11.316 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:11.328 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-09-16T04:42:10.580656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":420251304,"num_objects":51305,"num_object_clones":0,"num_object_copies":205220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51332,"num_write_kb":410538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":549,"num_bytes_recovered":4276224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51359,"ondisk_log_size":51359,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1296580,"kb_used_data":674756,"kb_used_omap":14,"kb_used_meta":621745,"kb_avail":376190780,"statfs":{"total":386547056640,"available":385219358720,"internally_reserved":0,"allocated":690950144,"data_stored":690266904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":636667441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,3,14,13,15,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39247872,"num_objects":4791,"num_object_clones":0,"num_object_copies":19164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4791,"num_write_kb":38328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007025"},"pg_stats":[{"pgid":"1.f","version":"31'3203","reported_seq":3242,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.183941+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:08.183941+0000","last_peered":"2024-09-16T04:42:08.183941+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:08.183941+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3203,"log_dups_size":0,"ondisk_log_size":3203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26238976,"num_objects":3203,"num_object_clones":0,"num_object_copies":12812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3203,"num_write_kb":25624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3208","reported_seq":3269,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.868139+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:08.868139+0000","last_peered":"2024-09-16T04:42:08.868139+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:08.868139+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3208,"log_dups_size":0,"ondisk_log_size":3208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26244364,"num_objects":3204,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3206,"num_write_kb":25644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3225","reported_seq":3295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.707659+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:08.707659+0000","last_peered":"2024-09-16T04:42:08.707659+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:08.707659+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3225,"log_dups_size":0,"ondisk_log_size":3225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26393222,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3224,"num_write_kb":25782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":995}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3247","reported_seq":3308,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.992505+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:08.992505+0000","last_peered":"2024-09-16T04:42:08.992505+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:08.992505+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3247,"log_dups_size":0,"ondisk_log_size":3247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26563852,"num_objects":3243,"num_object_clones":0,"num_object_copies":12972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3245,"num_write_kb":25949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1014}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3243","reported_seq":3291,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.230783+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:08.230783+0000","last_peered":"2024-09-16T04:42:08.230783+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:08.230783+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26539276,"num_objects":3240,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3241,"num_write_kb":25925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":957}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3125","reported_seq":3164,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.061706+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:08.061706+0000","last_peered":"2024-09-16T04:42:08.061706+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:08.061706+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3125,"log_dups_size":0,"ondisk_log_size":3125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25600000,"num_objects":3125,"num_object_clones":0,"num_object_copies":12500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3125,"num_write_kb":25000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3264","reported_seq":3323,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.975079+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:08.975079+0000","last_peered":"2024-09-16T04:42:08.975079+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:08.975079+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3264,"log_dups_size":0,"ondisk_log_size":3264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26686732,"num_objects":3258,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3262,"num_write_kb":26076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1032}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3249","reported_seq":4353,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:09.000325+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:09.000325+0000","last_peered":"2024-09-16T04:42:09.000325+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:09.000325+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3249,"log_dups_size":0,"ondisk_log_size":3249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26572044,"num_objects":3244,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3247,"num_write_kb":25964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":524,"num_bytes_recovered":4276224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1528}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3329","reported_seq":3381,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.236085+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:08.236085+0000","last_peered":"2024-09-16T04:42:08.236085+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:08.236085+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3329,"log_dups_size":0,"ondisk_log_size":3329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27200024,"num_objects":3321,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3326,"num_write_kb":26585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":981}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3215","reported_seq":3288,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.779911+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:08.779911+0000","last_peered":"2024-09-16T04:42:08.779911+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:08.779911+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3215,"log_dups_size":0,"ondisk_log_size":3215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26318092,"num_objects":3213,"num_object_clones":0,"num_object_copies":12852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3213,"num_write_kb":25702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1013}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3231","reported_seq":3283,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.263799+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:08.263799+0000","last_peered":"2024-09-16T04:42:08.263799+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:08.263799+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3231,"log_dups_size":0,"ondisk_log_size":3231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26423186,"num_objects":3226,"num_object_clones":0,"num_object_copies":12904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3228,"num_write_kb":25812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1037}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3111","reported_seq":3146,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.177089+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:08.177089+0000","last_peered":"2024-09-16T04:42:08.177089+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:08.177089+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3111,"log_dups_size":0,"ondisk_log_size":3111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25475718,"num_objects":3110,"num_object_clones":0,"num_object_copies":12440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3110,"num_write_kb":24879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":982}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3187","reported_seq":3256,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.717061+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:08.717061+0000","last_peered":"2024-09-16T04:42:08.717061+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:08.717061+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3187,"log_dups_size":0,"ondisk_log_size":3187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26098310,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3186,"num_write_kb":25487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":961}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3226","reported_seq":3275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.212479+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:08.212479+0000","last_peered":"2024-09-16T04:42:08.212479+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:08.212479+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3226,"log_dups_size":0,"ondisk_log_size":3226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26391820,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3224,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":988}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3101","reported_seq":3175,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.723628+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:08.723628+0000","last_peered":"2024-09-16T04:42:08.723628+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:08.723628+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3101,"log_dups_size":0,"ondisk_log_size":3101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25376012,"num_objects":3098,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3099,"num_write_kb":24789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":966}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3195","reported_seq":3256,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.937069+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:08.937069+0000","last_peered":"2024-09-16T04:42:08.937069+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:08.937069+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3195,"log_dups_size":0,"ondisk_log_size":3195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26129676,"num_objects":3190,"num_object_clones":0,"num_object_copies":12760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":951}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":420251304,"num_objects":51305,"num_object_clones":0,"num_object_copies":205220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51332,"num_write_kb":410538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":549,"num_bytes_recovered":4276224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":692805632,"data_stored":692805632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51359,"ondisk_log_size":51359,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380716,"kb_used_data":204636,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93991124,"statfs":{"total":96636764160,"available":96246910976,"internally_reserved":0,"allocated":209547264,"data_stored":209376454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180285135},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,5,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388852,"kb_used_data":204964,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93982988,"statfs":{"total":96636764160,"available":96238579712,"internally_reserved":0,"allocated":209883136,"data_stored":209712326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051593,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":145984,"kb_used_data":60144,"kb_used_omap":4,"kb_used_meta":85819,"kb_avail":94225856,"statfs":{"total":96636764160,"available":96487276544,"internally_reserved":0,"allocated":61587456,"data_stored":61416646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":87879375},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,3,3,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381028,"kb_used_data":205012,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93990812,"statfs":{"total":96636764160,"available":96246591488,"internally_reserved":0,"allocated":209932288,"data_stored":209761478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":180219598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":210632704,"data_stored":210632704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62316544,"data_stored":62316544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":210333696,"data_stored":210333696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209522688,"data_stored":209522688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:11.330 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:11.673 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:11.673 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:11.685 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-09-16T04:42:10.580656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":420251304,"num_objects":51305,"num_object_clones":0,"num_object_copies":205220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51332,"num_write_kb":410538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":549,"num_bytes_recovered":4276224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51359,"ondisk_log_size":51359,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1296580,"kb_used_data":674756,"kb_used_omap":14,"kb_used_meta":621745,"kb_avail":376190780,"statfs":{"total":386547056640,"available":385219358720,"internally_reserved":0,"allocated":690950144,"data_stored":690266904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":636667441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,3,14,13,15,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39247872,"num_objects":4791,"num_object_clones":0,"num_object_copies":19164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4791,"num_write_kb":38328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007025"},"pg_stats":[{"pgid":"1.f","version":"31'3203","reported_seq":3242,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.183941+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:08.183941+0000","last_peered":"2024-09-16T04:42:08.183941+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:08.183941+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3203,"log_dups_size":0,"ondisk_log_size":3203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26238976,"num_objects":3203,"num_object_clones":0,"num_object_copies":12812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3203,"num_write_kb":25624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3208","reported_seq":3269,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.868139+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:08.868139+0000","last_peered":"2024-09-16T04:42:08.868139+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:08.868139+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3208,"log_dups_size":0,"ondisk_log_size":3208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26244364,"num_objects":3204,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3206,"num_write_kb":25644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3225","reported_seq":3295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.707659+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:08.707659+0000","last_peered":"2024-09-16T04:42:08.707659+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:08.707659+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3225,"log_dups_size":0,"ondisk_log_size":3225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26393222,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3224,"num_write_kb":25782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":995}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3247","reported_seq":3308,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.992505+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:08.992505+0000","last_peered":"2024-09-16T04:42:08.992505+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:08.992505+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3247,"log_dups_size":0,"ondisk_log_size":3247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26563852,"num_objects":3243,"num_object_clones":0,"num_object_copies":12972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3245,"num_write_kb":25949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1014}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3243","reported_seq":3291,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.230783+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:08.230783+0000","last_peered":"2024-09-16T04:42:08.230783+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:08.230783+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26539276,"num_objects":3240,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3241,"num_write_kb":25925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":957}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3125","reported_seq":3164,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.061706+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:08.061706+0000","last_peered":"2024-09-16T04:42:08.061706+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:08.061706+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3125,"log_dups_size":0,"ondisk_log_size":3125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25600000,"num_objects":3125,"num_object_clones":0,"num_object_copies":12500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3125,"num_write_kb":25000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3264","reported_seq":3323,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.975079+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:08.975079+0000","last_peered":"2024-09-16T04:42:08.975079+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:08.975079+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3264,"log_dups_size":0,"ondisk_log_size":3264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26686732,"num_objects":3258,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3262,"num_write_kb":26076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1032}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3249","reported_seq":4353,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:09.000325+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:09.000325+0000","last_peered":"2024-09-16T04:42:09.000325+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:09.000325+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3249,"log_dups_size":0,"ondisk_log_size":3249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26572044,"num_objects":3244,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3247,"num_write_kb":25964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":524,"num_bytes_recovered":4276224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1528}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3329","reported_seq":3381,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.236085+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:08.236085+0000","last_peered":"2024-09-16T04:42:08.236085+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:08.236085+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3329,"log_dups_size":0,"ondisk_log_size":3329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27200024,"num_objects":3321,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3326,"num_write_kb":26585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":981}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3215","reported_seq":3288,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.779911+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:08.779911+0000","last_peered":"2024-09-16T04:42:08.779911+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:08.779911+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3215,"log_dups_size":0,"ondisk_log_size":3215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26318092,"num_objects":3213,"num_object_clones":0,"num_object_copies":12852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3213,"num_write_kb":25702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1013}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3231","reported_seq":3283,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.263799+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:08.263799+0000","last_peered":"2024-09-16T04:42:08.263799+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:08.263799+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3231,"log_dups_size":0,"ondisk_log_size":3231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26423186,"num_objects":3226,"num_object_clones":0,"num_object_copies":12904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3228,"num_write_kb":25812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1037}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3111","reported_seq":3146,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.177089+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:08.177089+0000","last_peered":"2024-09-16T04:42:08.177089+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:08.177089+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3111,"log_dups_size":0,"ondisk_log_size":3111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25475718,"num_objects":3110,"num_object_clones":0,"num_object_copies":12440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3110,"num_write_kb":24879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":982}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3187","reported_seq":3256,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.717061+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:08.717061+0000","last_peered":"2024-09-16T04:42:08.717061+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:08.717061+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3187,"log_dups_size":0,"ondisk_log_size":3187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26098310,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3186,"num_write_kb":25487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":961}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3226","reported_seq":3275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.212479+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:08.212479+0000","last_peered":"2024-09-16T04:42:08.212479+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:08.212479+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3226,"log_dups_size":0,"ondisk_log_size":3226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26391820,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3224,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":988}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3101","reported_seq":3175,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.723628+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:08.723628+0000","last_peered":"2024-09-16T04:42:08.723628+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:08.723628+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3101,"log_dups_size":0,"ondisk_log_size":3101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25376012,"num_objects":3098,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3099,"num_write_kb":24789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":966}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3195","reported_seq":3256,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.937069+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:08.937069+0000","last_peered":"2024-09-16T04:42:08.937069+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:08.937069+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3195,"log_dups_size":0,"ondisk_log_size":3195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26129676,"num_objects":3190,"num_object_clones":0,"num_object_copies":12760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":951}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":420251304,"num_objects":51305,"num_object_clones":0,"num_object_copies":205220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51332,"num_write_kb":410538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":549,"num_bytes_recovered":4276224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":692805632,"data_stored":692805632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51359,"ondisk_log_size":51359,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380716,"kb_used_data":204636,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93991124,"statfs":{"total":96636764160,"available":96246910976,"internally_reserved":0,"allocated":209547264,"data_stored":209376454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180285135},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,5,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388852,"kb_used_data":204964,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93982988,"statfs":{"total":96636764160,"available":96238579712,"internally_reserved":0,"allocated":209883136,"data_stored":209712326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051593,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":145984,"kb_used_data":60144,"kb_used_omap":4,"kb_used_meta":85819,"kb_avail":94225856,"statfs":{"total":96636764160,"available":96487276544,"internally_reserved":0,"allocated":61587456,"data_stored":61416646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":87879375},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,3,3,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381028,"kb_used_data":205012,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93990812,"statfs":{"total":96636764160,"available":96246591488,"internally_reserved":0,"allocated":209932288,"data_stored":209761478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":180219598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":210632704,"data_stored":210632704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62316544,"data_stored":62316544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":210333696,"data_stored":210333696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209522688,"data_stored":209522688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:11.686 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:42:11.686 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3203", 'reported_seq': 3242, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:08.183941+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:42:08.183941+0000', 'last_peered': '2024-09-16T04:42:08.183941+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:42:08.183941+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3203, 'log_dups_size': 0, 'ondisk_log_size': 3203, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26238976, 'num_objects': 3203, 'num_object_clones': 0, 'num_object_copies': 12812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3203, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3203, 'num_write_kb': 25624, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 984}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:11.687 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:42:11.687 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3208", 'reported_seq': 3269, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:08.868139+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:42:08.868139+0000', 'last_peered': '2024-09-16T04:42:08.868139+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:42:08.868139+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3208, 'log_dups_size': 0, 'ondisk_log_size': 3208, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26244364, 'num_objects': 3204, 'num_object_clones': 0, 'num_object_copies': 12816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3206, 'num_write_kb': 25644, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 993}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:11.687 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:42:11.687 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3225", 'reported_seq': 3295, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:08.707659+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:42:08.707659+0000', 'last_peered': '2024-09-16T04:42:08.707659+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:42:08.707659+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3225, 'log_dups_size': 0, 'ondisk_log_size': 3225, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26393222, 'num_objects': 3222, 'num_object_clones': 0, 'num_object_copies': 12888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3224, 'num_write_kb': 25782, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 995}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:11.687 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:42:11.687 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3247", 'reported_seq': 3308, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:08.992505+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:42:08.992505+0000', 'last_peered': '2024-09-16T04:42:08.992505+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:42:08.992505+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3247, 'log_dups_size': 0, 'ondisk_log_size': 3247, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26563852, 'num_objects': 3243, 'num_object_clones': 0, 'num_object_copies': 12972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3243, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3245, 'num_write_kb': 25949, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1014}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:11.687 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:42:11.687 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3243", 'reported_seq': 3291, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:08.230783+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:42:08.230783+0000', 'last_peered': '2024-09-16T04:42:08.230783+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:42:08.230783+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3243, 'log_dups_size': 0, 'ondisk_log_size': 3243, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26539276, 'num_objects': 3240, 'num_object_clones': 0, 'num_object_copies': 12960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3240, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3241, 'num_write_kb': 25925, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 957}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:11.687 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:42:11.687 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3125", 'reported_seq': 3164, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:08.061706+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:42:08.061706+0000', 'last_peered': '2024-09-16T04:42:08.061706+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:42:08.061706+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3125, 'log_dups_size': 0, 'ondisk_log_size': 3125, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25600000, 'num_objects': 3125, 'num_object_clones': 0, 'num_object_copies': 12500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3125, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3125, 'num_write_kb': 25000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 917}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:11.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:42:11.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3264", 'reported_seq': 3323, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:08.975079+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:42:08.975079+0000', 'last_peered': '2024-09-16T04:42:08.975079+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:42:08.975079+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3264, 'log_dups_size': 0, 'ondisk_log_size': 3264, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26686732, 'num_objects': 3258, 'num_object_clones': 0, 'num_object_copies': 13032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3262, 'num_write_kb': 26076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1032}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:11.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:42:11.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3249", 'reported_seq': 4353, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:09.000325+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:42:09.000325+0000', 'last_peered': '2024-09-16T04:42:09.000325+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:42:09.000325+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3249, 'log_dups_size': 0, 'ondisk_log_size': 3249, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26572044, 'num_objects': 3244, 'num_object_clones': 0, 'num_object_copies': 12976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1716, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3244, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3247, 'num_write_kb': 25964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 524, 'num_bytes_recovered': 4276224, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1528}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:11.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:42:11.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3329", 'reported_seq': 3381, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:08.236085+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:42:08.236085+0000', 'last_peered': '2024-09-16T04:42:08.236085+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:42:08.236085+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3329, 'log_dups_size': 0, 'ondisk_log_size': 3329, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27200024, 'num_objects': 3321, 'num_object_clones': 0, 'num_object_copies': 13284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3326, 'num_write_kb': 26585, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 981}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:11.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:42:11.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3215", 'reported_seq': 3288, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:08.779911+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:42:08.779911+0000', 'last_peered': '2024-09-16T04:42:08.779911+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:42:08.779911+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3215, 'log_dups_size': 0, 'ondisk_log_size': 3215, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26318092, 'num_objects': 3213, 'num_object_clones': 0, 'num_object_copies': 12852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3213, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3213, 'num_write_kb': 25702, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1013}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:11.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:42:11.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3231", 'reported_seq': 3283, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:08.263799+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:42:08.263799+0000', 'last_peered': '2024-09-16T04:42:08.263799+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:42:08.263799+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3231, 'log_dups_size': 0, 'ondisk_log_size': 3231, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26423186, 'num_objects': 3226, 'num_object_clones': 0, 'num_object_copies': 12904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3226, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3228, 'num_write_kb': 25812, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1037}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:11.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:42:11.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3111", 'reported_seq': 3146, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:08.177089+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:42:08.177089+0000', 'last_peered': '2024-09-16T04:42:08.177089+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:42:08.177089+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3111, 'log_dups_size': 0, 'ondisk_log_size': 3111, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25475718, 'num_objects': 3110, 'num_object_clones': 0, 'num_object_copies': 12440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3110, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3110, 'num_write_kb': 24879, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 982}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:11.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:42:11.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3187", 'reported_seq': 3256, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:08.717061+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:42:08.717061+0000', 'last_peered': '2024-09-16T04:42:08.717061+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:42:08.717061+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3187, 'log_dups_size': 0, 'ondisk_log_size': 3187, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26098310, 'num_objects': 3186, 'num_object_clones': 0, 'num_object_copies': 12744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3186, 'num_write_kb': 25487, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 961}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:11.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:42:11.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3226", 'reported_seq': 3275, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:08.212479+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:42:08.212479+0000', 'last_peered': '2024-09-16T04:42:08.212479+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:42:08.212479+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3226, 'log_dups_size': 0, 'ondisk_log_size': 3226, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26391820, 'num_objects': 3222, 'num_object_clones': 0, 'num_object_copies': 12888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3224, 'num_write_kb': 25788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 988}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:11.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:42:11.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3101", 'reported_seq': 3175, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:08.723628+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:42:08.723628+0000', 'last_peered': '2024-09-16T04:42:08.723628+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:42:08.723628+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3101, 'log_dups_size': 0, 'ondisk_log_size': 3101, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25376012, 'num_objects': 3098, 'num_object_clones': 0, 'num_object_copies': 12392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3098, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3099, 'num_write_kb': 24789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 966}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:11.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:42:11.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3195", 'reported_seq': 3256, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:08.937069+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:42:08.937069+0000', 'last_peered': '2024-09-16T04:42:08.937069+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:42:08.937069+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3195, 'log_dups_size': 0, 'ondisk_log_size': 3195, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26129676, 'num_objects': 3190, 'num_object_clones': 0, 'num_object_copies': 12760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3190, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3193, 'num_write_kb': 25532, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 951}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:11.690 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:42:12.269 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 78 20 52422 52402 5.24681 1.38281 0.0583834 0.0297661 2024-09-16T04:42:12.365 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:12.365 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35031/205220 objects degraded (17.070%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":144,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":51305,"data_bytes":420251304,"bytes_used":1327697920,"bytes_avail":385219358720,"bytes_total":386547056640,"degraded_objects":35031,"degraded_total":205220,"degraded_ratio":0.17069973686775172,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":128266,"recovering_keys_per_sec":0,"num_objects_recovered":188,"num_bytes_recovered":1540096,"num_keys_recovered":0,"write_bytes_sec":3268742,"read_op_per_sec":0,"write_op_per_sec":399},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T04:41:44.568132+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:42:12.395 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:42:12.395 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:12.986 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:12.987 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:13.011 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-09-16T04:42:12.581275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":420251304,"num_objects":51305,"num_object_clones":0,"num_object_copies":205220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51332,"num_write_kb":410538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":549,"num_bytes_recovered":4276224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51359,"ondisk_log_size":51359,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1308388,"kb_used_data":682404,"kb_used_omap":14,"kb_used_meta":625905,"kb_avail":376178972,"statfs":{"total":386547056640,"available":385207267328,"internally_reserved":0,"allocated":698781696,"data_stored":698098456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":640927281},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,17,12,7,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30498816,"num_objects":3723,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3723,"num_write_kb":29784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003503"},"pg_stats":[{"pgid":"1.f","version":"31'3203","reported_seq":3242,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.183941+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:08.183941+0000","last_peered":"2024-09-16T04:42:08.183941+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:08.183941+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3203,"log_dups_size":0,"ondisk_log_size":3203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26238976,"num_objects":3203,"num_object_clones":0,"num_object_copies":12812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3203,"num_write_kb":25624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3208","reported_seq":3269,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.868139+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:08.868139+0000","last_peered":"2024-09-16T04:42:08.868139+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:08.868139+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3208,"log_dups_size":0,"ondisk_log_size":3208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26244364,"num_objects":3204,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3206,"num_write_kb":25644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3225","reported_seq":3295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.707659+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:08.707659+0000","last_peered":"2024-09-16T04:42:08.707659+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:08.707659+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3225,"log_dups_size":0,"ondisk_log_size":3225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26393222,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3224,"num_write_kb":25782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":995}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3247","reported_seq":3308,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.992505+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:08.992505+0000","last_peered":"2024-09-16T04:42:08.992505+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:08.992505+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3247,"log_dups_size":0,"ondisk_log_size":3247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26563852,"num_objects":3243,"num_object_clones":0,"num_object_copies":12972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3245,"num_write_kb":25949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1014}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3243","reported_seq":3291,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.230783+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:08.230783+0000","last_peered":"2024-09-16T04:42:08.230783+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:08.230783+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26539276,"num_objects":3240,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3241,"num_write_kb":25925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":957}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3125","reported_seq":3164,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.061706+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:08.061706+0000","last_peered":"2024-09-16T04:42:08.061706+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:08.061706+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3125,"log_dups_size":0,"ondisk_log_size":3125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25600000,"num_objects":3125,"num_object_clones":0,"num_object_copies":12500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3125,"num_write_kb":25000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3264","reported_seq":3323,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.975079+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:08.975079+0000","last_peered":"2024-09-16T04:42:08.975079+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:08.975079+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3264,"log_dups_size":0,"ondisk_log_size":3264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26686732,"num_objects":3258,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3262,"num_write_kb":26076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1032}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3249","reported_seq":4353,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:09.000325+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:09.000325+0000","last_peered":"2024-09-16T04:42:09.000325+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:09.000325+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3249,"log_dups_size":0,"ondisk_log_size":3249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26572044,"num_objects":3244,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3247,"num_write_kb":25964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":524,"num_bytes_recovered":4276224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1528}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3329","reported_seq":3381,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.236085+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:08.236085+0000","last_peered":"2024-09-16T04:42:08.236085+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:08.236085+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3329,"log_dups_size":0,"ondisk_log_size":3329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27200024,"num_objects":3321,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3326,"num_write_kb":26585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":981}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3215","reported_seq":3288,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.779911+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:08.779911+0000","last_peered":"2024-09-16T04:42:08.779911+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:08.779911+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3215,"log_dups_size":0,"ondisk_log_size":3215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26318092,"num_objects":3213,"num_object_clones":0,"num_object_copies":12852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3213,"num_write_kb":25702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1013}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3231","reported_seq":3283,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.263799+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:08.263799+0000","last_peered":"2024-09-16T04:42:08.263799+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:08.263799+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3231,"log_dups_size":0,"ondisk_log_size":3231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26423186,"num_objects":3226,"num_object_clones":0,"num_object_copies":12904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3228,"num_write_kb":25812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1037}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3111","reported_seq":3146,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.177089+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:08.177089+0000","last_peered":"2024-09-16T04:42:08.177089+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:08.177089+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3111,"log_dups_size":0,"ondisk_log_size":3111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25475718,"num_objects":3110,"num_object_clones":0,"num_object_copies":12440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3110,"num_write_kb":24879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":982}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3187","reported_seq":3256,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.717061+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:08.717061+0000","last_peered":"2024-09-16T04:42:08.717061+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:08.717061+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3187,"log_dups_size":0,"ondisk_log_size":3187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26098310,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3186,"num_write_kb":25487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":961}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3226","reported_seq":3275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.212479+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:08.212479+0000","last_peered":"2024-09-16T04:42:08.212479+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:08.212479+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3226,"log_dups_size":0,"ondisk_log_size":3226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26391820,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3224,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":988}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3101","reported_seq":3175,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.723628+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:08.723628+0000","last_peered":"2024-09-16T04:42:08.723628+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:08.723628+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3101,"log_dups_size":0,"ondisk_log_size":3101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25376012,"num_objects":3098,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3099,"num_write_kb":24789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":966}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3195","reported_seq":3256,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:08.937069+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:08.937069+0000","last_peered":"2024-09-16T04:42:08.937069+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:08.937069+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3195,"log_dups_size":0,"ondisk_log_size":3195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26129676,"num_objects":3190,"num_object_clones":0,"num_object_copies":12760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":951}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":420251304,"num_objects":51305,"num_object_clones":0,"num_object_copies":205220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51332,"num_write_kb":410538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":549,"num_bytes_recovered":4276224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":700628992,"data_stored":700628992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51359,"ondisk_log_size":51359,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380716,"kb_used_data":204636,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93991124,"statfs":{"total":96636764160,"available":96246910976,"internally_reserved":0,"allocated":209547264,"data_stored":209376454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180285135},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,5,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388852,"kb_used_data":204964,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93982988,"statfs":{"total":96636764160,"available":96238579712,"internally_reserved":0,"allocated":209883136,"data_stored":209712326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051594,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157792,"kb_used_data":67792,"kb_used_omap":4,"kb_used_meta":89979,"kb_avail":94214048,"statfs":{"total":96636764160,"available":96475185152,"internally_reserved":0,"allocated":69419008,"data_stored":69248198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":92139215},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381028,"kb_used_data":205012,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93990812,"statfs":{"total":96636764160,"available":96246591488,"internally_reserved":0,"allocated":209932288,"data_stored":209761478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":180219598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":210632704,"data_stored":210632704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70139904,"data_stored":70139904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":210333696,"data_stored":210333696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209522688,"data_stored":209522688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:13.269 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 79 20 52766 52746 5.21441 2.6875 0.0247265 0.029948 2024-09-16T04:42:14.269 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:2024-09-16T04:42:14.268732+0000 min lat: 0.00269411 max lat: 1.73107 avg lat: 0.0301116 2024-09-16T04:42:14.269 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T04:42:14.269 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 80 20 53151 53131 5.18684 3.00781 0.0286126 0.0301116 2024-09-16T04:42:15.269 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 81 20 53529 53509 5.15924 2.95312 0.0564188 0.0302687 2024-09-16T04:42:16.013 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:16.269 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 82 20 53883 53863 5.13006 2.76562 0.0145551 0.0304267 2024-09-16T04:42:16.527 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:16.527 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:16.559 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-16T04:42:14.581720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":433497768,"num_objects":52922,"num_object_clones":0,"num_object_copies":211688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52949,"num_write_kb":423474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":687,"num_bytes_recovered":5406720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52976,"ondisk_log_size":52976,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1323512,"kb_used_data":701684,"kb_used_omap":14,"kb_used_meta":621745,"kb_avail":376163848,"statfs":{"total":386547056640,"available":385191780352,"internally_reserved":0,"allocated":718524416,"data_stored":717841176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":636667441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,8,14,19,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":43745280,"num_objects":5340,"num_object_clones":0,"num_object_copies":21360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5340,"num_write_kb":42720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":2154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003460"},"pg_stats":[{"pgid":"1.f","version":"31'3281","reported_seq":3319,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.147462+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:13.147462+0000","last_peered":"2024-09-16T04:42:13.147462+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:13.147462+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3281,"log_dups_size":0,"ondisk_log_size":3281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26877952,"num_objects":3281,"num_object_clones":0,"num_object_copies":13124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3281,"num_write_kb":26248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3302","reported_seq":3363,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.999818+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:13.999818+0000","last_peered":"2024-09-16T04:42:13.999818+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:13.999818+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3302,"log_dups_size":0,"ondisk_log_size":3302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27014412,"num_objects":3298,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3300,"num_write_kb":26396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1087}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3307","reported_seq":3377,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.671143+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:13.671143+0000","last_peered":"2024-09-16T04:42:13.671143+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:13.671143+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3307,"log_dups_size":0,"ondisk_log_size":3307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27064966,"num_objects":3304,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3306,"num_write_kb":26438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1077}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3357","reported_seq":3418,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.991981+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:13.991981+0000","last_peered":"2024-09-16T04:42:13.991981+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:13.991981+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3357,"log_dups_size":0,"ondisk_log_size":3357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27464972,"num_objects":3353,"num_object_clones":0,"num_object_copies":13412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3355,"num_write_kb":26829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3355","reported_seq":3406,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.218547+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:13.218547+0000","last_peered":"2024-09-16T04:42:13.218547+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:13.218547+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3355,"log_dups_size":0,"ondisk_log_size":3355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27456780,"num_objects":3352,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3353,"num_write_kb":26821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1069}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3237","reported_seq":3276,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.257688+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:13.257688+0000","last_peered":"2024-09-16T04:42:13.257688+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:13.257688+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3237,"log_dups_size":0,"ondisk_log_size":3237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26517504,"num_objects":3237,"num_object_clones":0,"num_object_copies":12948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3237,"num_write_kb":25896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3367","reported_seq":3427,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:14.018964+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:14.018964+0000","last_peered":"2024-09-16T04:42:14.018964+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:14.018964+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3367,"log_dups_size":0,"ondisk_log_size":3367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27530508,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3365,"num_write_kb":26900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1135}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3345","reported_seq":4725,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.989234+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:13.989234+0000","last_peered":"2024-09-16T04:42:13.989234+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:13.989234+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3345,"log_dups_size":0,"ondisk_log_size":3345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27358476,"num_objects":3340,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3343,"num_write_kb":26732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":662,"num_bytes_recovered":5406720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1762}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3432","reported_seq":3484,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.243412+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:13.243412+0000","last_peered":"2024-09-16T04:42:13.243412+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:13.243412+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3432,"log_dups_size":0,"ondisk_log_size":3432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28043800,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3429,"num_write_kb":27409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1084}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3306","reported_seq":3379,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.706259+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:13.706259+0000","last_peered":"2024-09-16T04:42:13.706259+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:13.706259+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3306,"log_dups_size":0,"ondisk_log_size":3306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27063564,"num_objects":3304,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3304,"num_write_kb":26430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1104}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3348","reported_seq":3401,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.219191+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:13.219191+0000","last_peered":"2024-09-16T04:42:13.219191+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:13.219191+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3348,"log_dups_size":0,"ondisk_log_size":3348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27381650,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3345,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1154}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3194","reported_seq":3229,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.261413+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:13.261413+0000","last_peered":"2024-09-16T04:42:13.261413+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:13.261413+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3194,"log_dups_size":0,"ondisk_log_size":3194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26155654,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1065}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3296","reported_seq":3365,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.792096+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:13.792096+0000","last_peered":"2024-09-16T04:42:13.792096+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:13.792096+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3296,"log_dups_size":0,"ondisk_log_size":3296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26991238,"num_objects":3295,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":26359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1070}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3327","reported_seq":3376,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.261453+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:13.261453+0000","last_peered":"2024-09-16T04:42:13.261453+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:13.261453+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3327,"log_dups_size":0,"ondisk_log_size":3327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27219212,"num_objects":3323,"num_object_clones":0,"num_object_copies":13292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3325,"num_write_kb":26596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3205","reported_seq":3278,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.781802+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:13.781802+0000","last_peered":"2024-09-16T04:42:13.781802+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:13.781802+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3205,"log_dups_size":0,"ondisk_log_size":3205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26227980,"num_objects":3202,"num_object_clones":0,"num_object_copies":12808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3203,"num_write_kb":25621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1070}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3317","reported_seq":3378,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.984466+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:13.984466+0000","last_peered":"2024-09-16T04:42:13.984466+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:13.984466+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3317,"log_dups_size":0,"ondisk_log_size":3317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27129100,"num_objects":3312,"num_object_clones":0,"num_object_copies":13248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3315,"num_write_kb":26508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1073}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":433497768,"num_objects":52922,"num_object_clones":0,"num_object_copies":211688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52949,"num_write_kb":423474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":687,"num_bytes_recovered":5406720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":720650240,"data_stored":720650240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52976,"ondisk_log_size":52976,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391444,"kb_used_data":211204,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93980396,"statfs":{"total":96636764160,"available":96235925504,"internally_reserved":0,"allocated":216272896,"data_stored":216102086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184544975},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,12,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":382816,"kb_used_data":211408,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93989024,"statfs":{"total":96636764160,"available":96244760576,"internally_reserved":0,"allocated":216481792,"data_stored":216310982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,2,5,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051594,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157792,"kb_used_data":67792,"kb_used_omap":4,"kb_used_meta":89979,"kb_avail":94214048,"statfs":{"total":96636764160,"available":96475185152,"internally_reserved":0,"allocated":69419008,"data_stored":69248198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":92139215},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391460,"kb_used_data":211280,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93980380,"statfs":{"total":96636764160,"available":96235909120,"internally_reserved":0,"allocated":216350720,"data_stored":216179910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":184479438},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217346048,"data_stored":217346048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70139904,"data_stored":70139904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":216997888,"data_stored":216997888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":216166400,"data_stored":216166400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:16.560 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:17.125 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:17.125 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:17.146 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-09-16T04:42:16.582461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":433497768,"num_objects":52922,"num_object_clones":0,"num_object_copies":211688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52949,"num_write_kb":423474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":687,"num_bytes_recovered":5406720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52976,"ondisk_log_size":52976,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1334716,"kb_used_data":708728,"kb_used_omap":14,"kb_used_meta":625905,"kb_avail":376152644,"statfs":{"total":386547056640,"available":385180307456,"internally_reserved":0,"allocated":725737472,"data_stored":725054232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":640927281},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,7,9,20,9,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28205056,"num_objects":3443,"num_object_clones":0,"num_object_copies":13772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3443,"num_write_kb":27544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":1646592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003795"},"pg_stats":[{"pgid":"1.f","version":"31'3281","reported_seq":3319,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.147462+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:13.147462+0000","last_peered":"2024-09-16T04:42:13.147462+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:13.147462+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3281,"log_dups_size":0,"ondisk_log_size":3281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26877952,"num_objects":3281,"num_object_clones":0,"num_object_copies":13124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3281,"num_write_kb":26248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3302","reported_seq":3363,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.999818+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:13.999818+0000","last_peered":"2024-09-16T04:42:13.999818+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:13.999818+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3302,"log_dups_size":0,"ondisk_log_size":3302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27014412,"num_objects":3298,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3300,"num_write_kb":26396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1087}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3307","reported_seq":3377,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.671143+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:13.671143+0000","last_peered":"2024-09-16T04:42:13.671143+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:13.671143+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3307,"log_dups_size":0,"ondisk_log_size":3307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27064966,"num_objects":3304,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3306,"num_write_kb":26438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1077}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3357","reported_seq":3418,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.991981+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:13.991981+0000","last_peered":"2024-09-16T04:42:13.991981+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:13.991981+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3357,"log_dups_size":0,"ondisk_log_size":3357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27464972,"num_objects":3353,"num_object_clones":0,"num_object_copies":13412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3355,"num_write_kb":26829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3355","reported_seq":3406,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.218547+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:13.218547+0000","last_peered":"2024-09-16T04:42:13.218547+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:13.218547+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3355,"log_dups_size":0,"ondisk_log_size":3355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27456780,"num_objects":3352,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3353,"num_write_kb":26821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1069}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3237","reported_seq":3276,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.257688+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:13.257688+0000","last_peered":"2024-09-16T04:42:13.257688+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:13.257688+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3237,"log_dups_size":0,"ondisk_log_size":3237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26517504,"num_objects":3237,"num_object_clones":0,"num_object_copies":12948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3237,"num_write_kb":25896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3367","reported_seq":3427,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:14.018964+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:14.018964+0000","last_peered":"2024-09-16T04:42:14.018964+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:14.018964+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3367,"log_dups_size":0,"ondisk_log_size":3367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27530508,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3365,"num_write_kb":26900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1135}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3345","reported_seq":4725,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.989234+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:13.989234+0000","last_peered":"2024-09-16T04:42:13.989234+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:13.989234+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3345,"log_dups_size":0,"ondisk_log_size":3345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27358476,"num_objects":3340,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3343,"num_write_kb":26732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":662,"num_bytes_recovered":5406720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1762}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3432","reported_seq":3484,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.243412+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:13.243412+0000","last_peered":"2024-09-16T04:42:13.243412+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:13.243412+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3432,"log_dups_size":0,"ondisk_log_size":3432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28043800,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3429,"num_write_kb":27409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1084}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3306","reported_seq":3379,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.706259+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:13.706259+0000","last_peered":"2024-09-16T04:42:13.706259+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:13.706259+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3306,"log_dups_size":0,"ondisk_log_size":3306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27063564,"num_objects":3304,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3304,"num_write_kb":26430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1104}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3348","reported_seq":3401,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.219191+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:13.219191+0000","last_peered":"2024-09-16T04:42:13.219191+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:13.219191+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3348,"log_dups_size":0,"ondisk_log_size":3348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27381650,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3345,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1154}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3194","reported_seq":3229,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.261413+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:13.261413+0000","last_peered":"2024-09-16T04:42:13.261413+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:13.261413+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3194,"log_dups_size":0,"ondisk_log_size":3194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26155654,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1065}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3296","reported_seq":3365,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.792096+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:13.792096+0000","last_peered":"2024-09-16T04:42:13.792096+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:13.792096+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3296,"log_dups_size":0,"ondisk_log_size":3296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26991238,"num_objects":3295,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":26359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1070}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3327","reported_seq":3376,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.261453+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:13.261453+0000","last_peered":"2024-09-16T04:42:13.261453+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:13.261453+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3327,"log_dups_size":0,"ondisk_log_size":3327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27219212,"num_objects":3323,"num_object_clones":0,"num_object_copies":13292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3325,"num_write_kb":26596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3205","reported_seq":3278,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.781802+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:13.781802+0000","last_peered":"2024-09-16T04:42:13.781802+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:13.781802+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3205,"log_dups_size":0,"ondisk_log_size":3205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26227980,"num_objects":3202,"num_object_clones":0,"num_object_copies":12808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3203,"num_write_kb":25621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1070}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3317","reported_seq":3378,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.984466+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:13.984466+0000","last_peered":"2024-09-16T04:42:13.984466+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:13.984466+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3317,"log_dups_size":0,"ondisk_log_size":3317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27129100,"num_objects":3312,"num_object_clones":0,"num_object_copies":13248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3315,"num_write_kb":26508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1073}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":433497768,"num_objects":52922,"num_object_clones":0,"num_object_copies":211688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52949,"num_write_kb":423474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":687,"num_bytes_recovered":5406720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":727867392,"data_stored":727867392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52976,"ondisk_log_size":52976,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391444,"kb_used_data":211204,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93980396,"statfs":{"total":96636764160,"available":96235925504,"internally_reserved":0,"allocated":216272896,"data_stored":216102086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184544975},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,12,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":382816,"kb_used_data":211408,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93989024,"statfs":{"total":96636764160,"available":96244760576,"internally_reserved":0,"allocated":216481792,"data_stored":216310982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,2,5,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051595,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":168996,"kb_used_data":74836,"kb_used_omap":4,"kb_used_meta":94139,"kb_avail":94202844,"statfs":{"total":96636764160,"available":96463712256,"internally_reserved":0,"allocated":76632064,"data_stored":76461254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":96399055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391460,"kb_used_data":211280,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93980380,"statfs":{"total":96636764160,"available":96235909120,"internally_reserved":0,"allocated":216350720,"data_stored":216179910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":184479438},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217346048,"data_stored":217346048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77357056,"data_stored":77357056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":216997888,"data_stored":216997888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":216166400,"data_stored":216166400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:17.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:42:17.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3281", 'reported_seq': 3319, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:13.147462+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:42:13.147462+0000', 'last_peered': '2024-09-16T04:42:13.147462+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:42:13.147462+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3281, 'log_dups_size': 0, 'ondisk_log_size': 3281, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26877952, 'num_objects': 3281, 'num_object_clones': 0, 'num_object_copies': 13124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3281, 'num_write_kb': 26248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1062}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:17.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:42:17.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3302", 'reported_seq': 3363, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:13.999818+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:42:13.999818+0000', 'last_peered': '2024-09-16T04:42:13.999818+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:42:13.999818+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3302, 'log_dups_size': 0, 'ondisk_log_size': 3302, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27014412, 'num_objects': 3298, 'num_object_clones': 0, 'num_object_copies': 13192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3298, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3300, 'num_write_kb': 26396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1087}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:17.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:42:17.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3307", 'reported_seq': 3377, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:13.671143+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:42:13.671143+0000', 'last_peered': '2024-09-16T04:42:13.671143+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:42:13.671143+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3307, 'log_dups_size': 0, 'ondisk_log_size': 3307, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27064966, 'num_objects': 3304, 'num_object_clones': 0, 'num_object_copies': 13216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3304, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3306, 'num_write_kb': 26438, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1077}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:17.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:42:17.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3357", 'reported_seq': 3418, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:13.991981+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:42:13.991981+0000', 'last_peered': '2024-09-16T04:42:13.991981+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:42:13.991981+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3357, 'log_dups_size': 0, 'ondisk_log_size': 3357, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27464972, 'num_objects': 3353, 'num_object_clones': 0, 'num_object_copies': 13412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3353, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3355, 'num_write_kb': 26829, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1124}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:17.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:42:17.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3355", 'reported_seq': 3406, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:13.218547+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:42:13.218547+0000', 'last_peered': '2024-09-16T04:42:13.218547+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:42:13.218547+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3355, 'log_dups_size': 0, 'ondisk_log_size': 3355, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27456780, 'num_objects': 3352, 'num_object_clones': 0, 'num_object_copies': 13408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3352, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3353, 'num_write_kb': 26821, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1069}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:17.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:42:17.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3237", 'reported_seq': 3276, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:13.257688+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:42:13.257688+0000', 'last_peered': '2024-09-16T04:42:13.257688+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:42:13.257688+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3237, 'log_dups_size': 0, 'ondisk_log_size': 3237, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26517504, 'num_objects': 3237, 'num_object_clones': 0, 'num_object_copies': 12948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3237, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3237, 'num_write_kb': 25896, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1029}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:17.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:42:17.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3367", 'reported_seq': 3427, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:14.018964+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:42:14.018964+0000', 'last_peered': '2024-09-16T04:42:14.018964+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:42:14.018964+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3367, 'log_dups_size': 0, 'ondisk_log_size': 3367, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27530508, 'num_objects': 3361, 'num_object_clones': 0, 'num_object_copies': 13444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3361, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3365, 'num_write_kb': 26900, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1135}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:17.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:42:17.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3345", 'reported_seq': 4725, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:13.989234+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:42:13.989234+0000', 'last_peered': '2024-09-16T04:42:13.989234+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:42:13.989234+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3345, 'log_dups_size': 0, 'ondisk_log_size': 3345, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27358476, 'num_objects': 3340, 'num_object_clones': 0, 'num_object_copies': 13360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1578, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3343, 'num_write_kb': 26732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 662, 'num_bytes_recovered': 5406720, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1762}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:17.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:42:17.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3432", 'reported_seq': 3484, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:13.243412+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:42:13.243412+0000', 'last_peered': '2024-09-16T04:42:13.243412+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:42:13.243412+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3432, 'log_dups_size': 0, 'ondisk_log_size': 3432, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28043800, 'num_objects': 3424, 'num_object_clones': 0, 'num_object_copies': 13696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3429, 'num_write_kb': 27409, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1084}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:17.149 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:42:17.149 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3306", 'reported_seq': 3379, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:13.706259+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:42:13.706259+0000', 'last_peered': '2024-09-16T04:42:13.706259+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:42:13.706259+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3306, 'log_dups_size': 0, 'ondisk_log_size': 3306, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27063564, 'num_objects': 3304, 'num_object_clones': 0, 'num_object_copies': 13216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3304, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3304, 'num_write_kb': 26430, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1104}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:17.149 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:42:17.149 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3348", 'reported_seq': 3401, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:13.219191+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:42:13.219191+0000', 'last_peered': '2024-09-16T04:42:13.219191+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:42:13.219191+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3348, 'log_dups_size': 0, 'ondisk_log_size': 3348, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27381650, 'num_objects': 3343, 'num_object_clones': 0, 'num_object_copies': 13372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3343, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3345, 'num_write_kb': 26748, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1154}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:17.149 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:42:17.149 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3194", 'reported_seq': 3229, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:13.261413+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:42:13.261413+0000', 'last_peered': '2024-09-16T04:42:13.261413+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:42:13.261413+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3194, 'log_dups_size': 0, 'ondisk_log_size': 3194, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26155654, 'num_objects': 3193, 'num_object_clones': 0, 'num_object_copies': 12772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3193, 'num_write_kb': 25543, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1065}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:17.149 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:42:17.149 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3296", 'reported_seq': 3365, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:13.792096+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:42:13.792096+0000', 'last_peered': '2024-09-16T04:42:13.792096+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:42:13.792096+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3296, 'log_dups_size': 0, 'ondisk_log_size': 3296, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26991238, 'num_objects': 3295, 'num_object_clones': 0, 'num_object_copies': 13180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3295, 'num_write_kb': 26359, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1070}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:17.149 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:42:17.149 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3327", 'reported_seq': 3376, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:13.261453+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:42:13.261453+0000', 'last_peered': '2024-09-16T04:42:13.261453+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:42:13.261453+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3327, 'log_dups_size': 0, 'ondisk_log_size': 3327, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27219212, 'num_objects': 3323, 'num_object_clones': 0, 'num_object_copies': 13292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3323, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3325, 'num_write_kb': 26596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1089}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:17.150 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:42:17.150 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3205", 'reported_seq': 3278, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:13.781802+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:42:13.781802+0000', 'last_peered': '2024-09-16T04:42:13.781802+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:42:13.781802+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3205, 'log_dups_size': 0, 'ondisk_log_size': 3205, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26227980, 'num_objects': 3202, 'num_object_clones': 0, 'num_object_copies': 12808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3203, 'num_write_kb': 25621, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1070}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:17.150 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:42:17.150 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3317", 'reported_seq': 3378, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:13.984466+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:42:13.984466+0000', 'last_peered': '2024-09-16T04:42:13.984466+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:42:13.984466+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3317, 'log_dups_size': 0, 'ondisk_log_size': 3317, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27129100, 'num_objects': 3312, 'num_object_clones': 0, 'num_object_copies': 13248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3312, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3315, 'num_write_kb': 26508, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1073}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:17.150 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:42:17.269 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 83 20 54185 54165 5.09668 2.35938 0.0577555 0.0306455 2024-09-16T04:42:17.850 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:17.850 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34893/211688 objects degraded (16.483%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":150,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":52922,"data_bytes":433497768,"bytes_used":1366749184,"bytes_avail":385180307456,"bytes_total":386547056640,"degraded_objects":34893,"degraded_total":211688,"degraded_ratio":0.16483220588791048,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":137172,"recovering_keys_per_sec":0,"num_objects_recovered":201,"num_bytes_recovered":1646592,"num_keys_recovered":0,"write_bytes_sec":2349678,"read_op_per_sec":0,"write_op_per_sec":286},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T04:41:44.568132+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:42:17.877 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:42:17.877 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:18.270 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 84 20 54507 54487 5.06596 2.51562 0.0379623 0.0308262 2024-09-16T04:42:18.464 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:18.464 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:18.497 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-09-16T04:42:16.582461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":433497768,"num_objects":52922,"num_object_clones":0,"num_object_copies":211688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52949,"num_write_kb":423474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":687,"num_bytes_recovered":5406720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52976,"ondisk_log_size":52976,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1334716,"kb_used_data":708728,"kb_used_omap":14,"kb_used_meta":625905,"kb_avail":376152644,"statfs":{"total":386547056640,"available":385180307456,"internally_reserved":0,"allocated":725737472,"data_stored":725054232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":640927281},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,7,9,20,9,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28205056,"num_objects":3443,"num_object_clones":0,"num_object_copies":13772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3443,"num_write_kb":27544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":1646592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003795"},"pg_stats":[{"pgid":"1.f","version":"31'3281","reported_seq":3319,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.147462+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:13.147462+0000","last_peered":"2024-09-16T04:42:13.147462+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:13.147462+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3281,"log_dups_size":0,"ondisk_log_size":3281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26877952,"num_objects":3281,"num_object_clones":0,"num_object_copies":13124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3281,"num_write_kb":26248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3302","reported_seq":3363,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.999818+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:13.999818+0000","last_peered":"2024-09-16T04:42:13.999818+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:13.999818+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3302,"log_dups_size":0,"ondisk_log_size":3302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27014412,"num_objects":3298,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3300,"num_write_kb":26396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1087}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3307","reported_seq":3377,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.671143+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:13.671143+0000","last_peered":"2024-09-16T04:42:13.671143+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:13.671143+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3307,"log_dups_size":0,"ondisk_log_size":3307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27064966,"num_objects":3304,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3306,"num_write_kb":26438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1077}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3357","reported_seq":3418,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.991981+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:13.991981+0000","last_peered":"2024-09-16T04:42:13.991981+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:13.991981+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3357,"log_dups_size":0,"ondisk_log_size":3357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27464972,"num_objects":3353,"num_object_clones":0,"num_object_copies":13412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3355,"num_write_kb":26829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3355","reported_seq":3406,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.218547+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:13.218547+0000","last_peered":"2024-09-16T04:42:13.218547+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:13.218547+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3355,"log_dups_size":0,"ondisk_log_size":3355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27456780,"num_objects":3352,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3353,"num_write_kb":26821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1069}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3237","reported_seq":3276,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.257688+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:13.257688+0000","last_peered":"2024-09-16T04:42:13.257688+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:13.257688+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3237,"log_dups_size":0,"ondisk_log_size":3237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26517504,"num_objects":3237,"num_object_clones":0,"num_object_copies":12948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3237,"num_write_kb":25896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3367","reported_seq":3427,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:14.018964+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:14.018964+0000","last_peered":"2024-09-16T04:42:14.018964+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:14.018964+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3367,"log_dups_size":0,"ondisk_log_size":3367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27530508,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3365,"num_write_kb":26900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1135}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3345","reported_seq":4725,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.989234+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:13.989234+0000","last_peered":"2024-09-16T04:42:13.989234+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:13.989234+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3345,"log_dups_size":0,"ondisk_log_size":3345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27358476,"num_objects":3340,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3343,"num_write_kb":26732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":662,"num_bytes_recovered":5406720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1762}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3432","reported_seq":3484,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.243412+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:13.243412+0000","last_peered":"2024-09-16T04:42:13.243412+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:13.243412+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3432,"log_dups_size":0,"ondisk_log_size":3432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28043800,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3429,"num_write_kb":27409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1084}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3306","reported_seq":3379,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.706259+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:13.706259+0000","last_peered":"2024-09-16T04:42:13.706259+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:13.706259+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3306,"log_dups_size":0,"ondisk_log_size":3306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27063564,"num_objects":3304,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3304,"num_write_kb":26430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1104}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3348","reported_seq":3401,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.219191+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:13.219191+0000","last_peered":"2024-09-16T04:42:13.219191+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:13.219191+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3348,"log_dups_size":0,"ondisk_log_size":3348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27381650,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3345,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1154}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3194","reported_seq":3229,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.261413+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:13.261413+0000","last_peered":"2024-09-16T04:42:13.261413+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:13.261413+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3194,"log_dups_size":0,"ondisk_log_size":3194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26155654,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1065}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3296","reported_seq":3365,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.792096+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:13.792096+0000","last_peered":"2024-09-16T04:42:13.792096+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:13.792096+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3296,"log_dups_size":0,"ondisk_log_size":3296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26991238,"num_objects":3295,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":26359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1070}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3327","reported_seq":3376,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.261453+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:13.261453+0000","last_peered":"2024-09-16T04:42:13.261453+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:13.261453+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3327,"log_dups_size":0,"ondisk_log_size":3327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27219212,"num_objects":3323,"num_object_clones":0,"num_object_copies":13292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3325,"num_write_kb":26596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3205","reported_seq":3278,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.781802+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:13.781802+0000","last_peered":"2024-09-16T04:42:13.781802+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:13.781802+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3205,"log_dups_size":0,"ondisk_log_size":3205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26227980,"num_objects":3202,"num_object_clones":0,"num_object_copies":12808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3203,"num_write_kb":25621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1070}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3317","reported_seq":3378,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:13.984466+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:13.984466+0000","last_peered":"2024-09-16T04:42:13.984466+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:13.984466+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3317,"log_dups_size":0,"ondisk_log_size":3317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27129100,"num_objects":3312,"num_object_clones":0,"num_object_copies":13248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3315,"num_write_kb":26508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1073}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":433497768,"num_objects":52922,"num_object_clones":0,"num_object_copies":211688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52949,"num_write_kb":423474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":687,"num_bytes_recovered":5406720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":727867392,"data_stored":727867392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52976,"ondisk_log_size":52976,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391444,"kb_used_data":211204,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93980396,"statfs":{"total":96636764160,"available":96235925504,"internally_reserved":0,"allocated":216272896,"data_stored":216102086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":184544975},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,12,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":382816,"kb_used_data":211408,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93989024,"statfs":{"total":96636764160,"available":96244760576,"internally_reserved":0,"allocated":216481792,"data_stored":216310982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,2,5,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051595,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":168996,"kb_used_data":74836,"kb_used_omap":4,"kb_used_meta":94139,"kb_avail":94202844,"statfs":{"total":96636764160,"available":96463712256,"internally_reserved":0,"allocated":76632064,"data_stored":76461254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":96399055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391460,"kb_used_data":211280,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93980380,"statfs":{"total":96636764160,"available":96235909120,"internally_reserved":0,"allocated":216350720,"data_stored":216179910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":184479438},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217346048,"data_stored":217346048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77357056,"data_stored":77357056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":216997888,"data_stored":216997888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":216166400,"data_stored":216166400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:19.271 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 85 20 54872 54852 5.03991 2.85156 0.0339541 0.0309845 2024-09-16T04:42:20.270 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 86 20 55244 55224 5.0151 2.90625 0.0601659 0.03114 2024-09-16T04:42:21.270 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 87 20 55612 55592 4.99051 2.875 0.0358331 0.0312955 2024-09-16T04:42:21.499 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:22.057 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:22.058 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:22.089 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-09-16T04:42:20.584136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":447833768,"num_objects":54672,"num_object_clones":0,"num_object_copies":218688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54699,"num_write_kb":437474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":760,"num_bytes_recovered":6004736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54726,"ondisk_log_size":54726,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1368308,"kb_used_data":729840,"kb_used_omap":14,"kb_used_meta":638385,"kb_avail":376119052,"statfs":{"total":386547056640,"available":385145909248,"internally_reserved":0,"allocated":747356160,"data_stored":746672920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":653706801},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,4,11,13,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35684352,"num_objects":4356,"num_object_clones":0,"num_object_copies":17424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4356,"num_write_kb":34848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":274,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004391"},"pg_stats":[{"pgid":"1.f","version":"31'3400","reported_seq":3439,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.280177+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:18.280177+0000","last_peered":"2024-09-16T04:42:18.280177+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:18.280177+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3400,"log_dups_size":0,"ondisk_log_size":3400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27852800,"num_objects":3400,"num_object_clones":0,"num_object_copies":13600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3400,"num_write_kb":27200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3406","reported_seq":3467,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:19.064536+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:19.064536+0000","last_peered":"2024-09-16T04:42:19.064536+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:19.064536+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3406,"log_dups_size":0,"ondisk_log_size":3406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27866380,"num_objects":3402,"num_object_clones":0,"num_object_copies":13608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3404,"num_write_kb":27228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1191}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3412","reported_seq":3482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.813736+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:18.813736+0000","last_peered":"2024-09-16T04:42:18.813736+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:18.813736+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3412,"log_dups_size":0,"ondisk_log_size":3412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27925126,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3411,"num_write_kb":27278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1182}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3473","reported_seq":3534,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.986603+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:18.986603+0000","last_peered":"2024-09-16T04:42:18.986603+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:18.986603+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3473,"log_dups_size":0,"ondisk_log_size":3473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28415244,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":27757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1240}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3470","reported_seq":3521,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.313270+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:18.313270+0000","last_peered":"2024-09-16T04:42:18.313270+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:18.313270+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28398860,"num_objects":3467,"num_object_clones":0,"num_object_copies":13868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3468,"num_write_kb":27741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3350","reported_seq":3389,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.368606+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:18.368606+0000","last_peered":"2024-09-16T04:42:18.368606+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:18.368606+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27443200,"num_objects":3350,"num_object_clones":0,"num_object_copies":13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3350,"num_write_kb":26800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1142}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3481","reported_seq":3541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:19.054644+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:19.054644+0000","last_peered":"2024-09-16T04:42:19.054644+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:19.054644+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28464396,"num_objects":3475,"num_object_clones":0,"num_object_copies":13900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3479,"num_write_kb":27812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1249}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3466","reported_seq":4991,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:19.079164+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:19.079164+0000","last_peered":"2024-09-16T04:42:19.079164+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:19.079164+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3466,"log_dups_size":0,"ondisk_log_size":3466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28349708,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3464,"num_write_kb":27700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":735,"num_bytes_recovered":6004736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1956}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3526","reported_seq":3578,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.254059+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:18.254059+0000","last_peered":"2024-09-16T04:42:18.254059+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:18.254059+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28813848,"num_objects":3518,"num_object_clones":0,"num_object_copies":14072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3523,"num_write_kb":28161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3418","reported_seq":3491,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.703028+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:18.703028+0000","last_peered":"2024-09-16T04:42:18.703028+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:18.703028+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3418,"log_dups_size":0,"ondisk_log_size":3418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27981068,"num_objects":3416,"num_object_clones":0,"num_object_copies":13664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3416,"num_write_kb":27326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1216}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3451","reported_seq":3504,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.327466+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:18.327466+0000","last_peered":"2024-09-16T04:42:18.327466+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:18.327466+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3451,"log_dups_size":0,"ondisk_log_size":3451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28225426,"num_objects":3446,"num_object_clones":0,"num_object_copies":13784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3448,"num_write_kb":27572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3314","reported_seq":3348,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.325399+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:18.325399+0000","last_peered":"2024-09-16T04:42:18.325399+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:18.325399+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3314,"log_dups_size":0,"ondisk_log_size":3314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27138694,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3313,"num_write_kb":26503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1185}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3400","reported_seq":3469,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.747745+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:18.747745+0000","last_peered":"2024-09-16T04:42:18.747745+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:18.747745+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3400,"log_dups_size":0,"ondisk_log_size":3400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27843206,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3399,"num_write_kb":27191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1174}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3440","reported_seq":3489,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.327903+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:18.327903+0000","last_peered":"2024-09-16T04:42:18.327903+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:18.327903+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3440,"log_dups_size":0,"ondisk_log_size":3440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28144908,"num_objects":3436,"num_object_clones":0,"num_object_copies":13744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3438,"num_write_kb":27500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1202}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3308","reported_seq":3382,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.770935+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:18.770935+0000","last_peered":"2024-09-16T04:42:18.770935+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:18.770935+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3308,"log_dups_size":0,"ondisk_log_size":3308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27071756,"num_objects":3305,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3306,"num_write_kb":26445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1173}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3411","reported_seq":3472,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.919816+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:18.919816+0000","last_peered":"2024-09-16T04:42:18.919816+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:18.919816+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27899148,"num_objects":3406,"num_object_clones":0,"num_object_copies":13624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3409,"num_write_kb":27260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":447833768,"num_objects":54672,"num_object_clones":0,"num_object_copies":218688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54699,"num_write_kb":437474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":760,"num_bytes_recovered":6004736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":749391872,"data_stored":749391872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54726,"ondisk_log_size":54726,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402596,"kb_used_data":218196,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93969244,"statfs":{"total":96636764160,"available":96224505856,"internally_reserved":0,"allocated":223432704,"data_stored":223261894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188804815},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393992,"kb_used_data":218420,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93977848,"statfs":{"total":96636764160,"available":96233316352,"internally_reserved":0,"allocated":223662080,"data_stored":223491270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051595,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":168996,"kb_used_data":74836,"kb_used_omap":4,"kb_used_meta":94139,"kb_avail":94202844,"statfs":{"total":96636764160,"available":96463712256,"internally_reserved":0,"allocated":76632064,"data_stored":76461254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":96399055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402724,"kb_used_data":218388,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93969116,"statfs":{"total":96636764160,"available":96224374784,"internally_reserved":0,"allocated":223629312,"data_stored":223458502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":188739278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,6,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":224493568,"data_stored":224493568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77357056,"data_stored":77357056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":224096256,"data_stored":224096256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":223444992,"data_stored":223444992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:22.090 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:22.273 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 88 20 55945 55925 4.9632 2.60156 0.0670982 0.0314662 2024-09-16T04:42:22.690 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:22.690 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:22.715 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-09-16T04:42:22.584902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":447833768,"num_objects":54672,"num_object_clones":0,"num_object_copies":218688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54699,"num_write_kb":437474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":760,"num_bytes_recovered":6004736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54726,"ondisk_log_size":54726,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1380912,"kb_used_data":736876,"kb_used_omap":14,"kb_used_meta":643953,"kb_avail":376106448,"statfs":{"total":386547056640,"available":385133002752,"internally_reserved":0,"allocated":754561024,"data_stored":753877784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":659408433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,6,7,12,17,14],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27582464,"num_objects":3367,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3367,"num_write_kb":26936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1728512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004246"},"pg_stats":[{"pgid":"1.f","version":"31'3400","reported_seq":3439,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.280177+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:18.280177+0000","last_peered":"2024-09-16T04:42:18.280177+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:18.280177+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3400,"log_dups_size":0,"ondisk_log_size":3400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27852800,"num_objects":3400,"num_object_clones":0,"num_object_copies":13600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3400,"num_write_kb":27200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3406","reported_seq":3467,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:19.064536+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:19.064536+0000","last_peered":"2024-09-16T04:42:19.064536+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:19.064536+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3406,"log_dups_size":0,"ondisk_log_size":3406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27866380,"num_objects":3402,"num_object_clones":0,"num_object_copies":13608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3404,"num_write_kb":27228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1191}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3412","reported_seq":3482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.813736+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:18.813736+0000","last_peered":"2024-09-16T04:42:18.813736+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:18.813736+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3412,"log_dups_size":0,"ondisk_log_size":3412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27925126,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3411,"num_write_kb":27278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1182}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3473","reported_seq":3534,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.986603+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:18.986603+0000","last_peered":"2024-09-16T04:42:18.986603+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:18.986603+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3473,"log_dups_size":0,"ondisk_log_size":3473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28415244,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":27757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1240}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3470","reported_seq":3521,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.313270+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:18.313270+0000","last_peered":"2024-09-16T04:42:18.313270+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:18.313270+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28398860,"num_objects":3467,"num_object_clones":0,"num_object_copies":13868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3468,"num_write_kb":27741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3350","reported_seq":3389,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.368606+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:18.368606+0000","last_peered":"2024-09-16T04:42:18.368606+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:18.368606+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27443200,"num_objects":3350,"num_object_clones":0,"num_object_copies":13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3350,"num_write_kb":26800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1142}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3481","reported_seq":3541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:19.054644+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:19.054644+0000","last_peered":"2024-09-16T04:42:19.054644+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:19.054644+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28464396,"num_objects":3475,"num_object_clones":0,"num_object_copies":13900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3479,"num_write_kb":27812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1249}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3466","reported_seq":4991,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:19.079164+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:19.079164+0000","last_peered":"2024-09-16T04:42:19.079164+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:19.079164+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3466,"log_dups_size":0,"ondisk_log_size":3466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28349708,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3464,"num_write_kb":27700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":735,"num_bytes_recovered":6004736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1956}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3526","reported_seq":3578,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.254059+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:18.254059+0000","last_peered":"2024-09-16T04:42:18.254059+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:18.254059+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28813848,"num_objects":3518,"num_object_clones":0,"num_object_copies":14072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3523,"num_write_kb":28161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3418","reported_seq":3491,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.703028+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:18.703028+0000","last_peered":"2024-09-16T04:42:18.703028+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:18.703028+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3418,"log_dups_size":0,"ondisk_log_size":3418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27981068,"num_objects":3416,"num_object_clones":0,"num_object_copies":13664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3416,"num_write_kb":27326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1216}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3451","reported_seq":3504,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.327466+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:18.327466+0000","last_peered":"2024-09-16T04:42:18.327466+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:18.327466+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3451,"log_dups_size":0,"ondisk_log_size":3451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28225426,"num_objects":3446,"num_object_clones":0,"num_object_copies":13784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3448,"num_write_kb":27572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3314","reported_seq":3348,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.325399+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:18.325399+0000","last_peered":"2024-09-16T04:42:18.325399+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:18.325399+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3314,"log_dups_size":0,"ondisk_log_size":3314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27138694,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3313,"num_write_kb":26503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1185}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3400","reported_seq":3469,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.747745+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:18.747745+0000","last_peered":"2024-09-16T04:42:18.747745+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:18.747745+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3400,"log_dups_size":0,"ondisk_log_size":3400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27843206,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3399,"num_write_kb":27191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1174}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3440","reported_seq":3489,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.327903+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:18.327903+0000","last_peered":"2024-09-16T04:42:18.327903+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:18.327903+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3440,"log_dups_size":0,"ondisk_log_size":3440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28144908,"num_objects":3436,"num_object_clones":0,"num_object_copies":13744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3438,"num_write_kb":27500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1202}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3308","reported_seq":3382,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.770935+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:18.770935+0000","last_peered":"2024-09-16T04:42:18.770935+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:18.770935+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3308,"log_dups_size":0,"ondisk_log_size":3308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27071756,"num_objects":3305,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3306,"num_write_kb":26445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1173}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3411","reported_seq":3472,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.919816+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:18.919816+0000","last_peered":"2024-09-16T04:42:18.919816+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:18.919816+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27899148,"num_objects":3406,"num_object_clones":0,"num_object_copies":13624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3409,"num_write_kb":27260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":447833768,"num_objects":54672,"num_object_clones":0,"num_object_copies":218688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54699,"num_write_kb":437474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":760,"num_bytes_recovered":6004736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":756789248,"data_stored":756789248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54726,"ondisk_log_size":54726,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402596,"kb_used_data":218196,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93969244,"statfs":{"total":96636764160,"available":96224505856,"internally_reserved":0,"allocated":223432704,"data_stored":223261894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188804815},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393992,"kb_used_data":218420,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93977848,"statfs":{"total":96636764160,"available":96233316352,"internally_reserved":0,"allocated":223662080,"data_stored":223491270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181600,"kb_used_data":81872,"kb_used_omap":4,"kb_used_meta":99707,"kb_avail":94190240,"statfs":{"total":96636764160,"available":96450805760,"internally_reserved":0,"allocated":83836928,"data_stored":83666118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":102100687},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,4,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402724,"kb_used_data":218388,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93969116,"statfs":{"total":96636764160,"available":96224374784,"internally_reserved":0,"allocated":223629312,"data_stored":223458502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":188739278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,6,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":224493568,"data_stored":224493568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":84754432,"data_stored":84754432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":224096256,"data_stored":224096256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":223444992,"data_stored":223444992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:22.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:42:22.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3400", 'reported_seq': 3439, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:18.280177+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:42:18.280177+0000', 'last_peered': '2024-09-16T04:42:18.280177+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:42:18.280177+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3400, 'log_dups_size': 0, 'ondisk_log_size': 3400, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27852800, 'num_objects': 3400, 'num_object_clones': 0, 'num_object_copies': 13600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3400, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3400, 'num_write_kb': 27200, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1181}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:22.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:42:22.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3406", 'reported_seq': 3467, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:19.064536+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:42:19.064536+0000', 'last_peered': '2024-09-16T04:42:19.064536+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:42:19.064536+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3406, 'log_dups_size': 0, 'ondisk_log_size': 3406, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27866380, 'num_objects': 3402, 'num_object_clones': 0, 'num_object_copies': 13608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3404, 'num_write_kb': 27228, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1191}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:22.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:42:22.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3412", 'reported_seq': 3482, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:18.813736+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:42:18.813736+0000', 'last_peered': '2024-09-16T04:42:18.813736+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:42:18.813736+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3412, 'log_dups_size': 0, 'ondisk_log_size': 3412, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27925126, 'num_objects': 3409, 'num_object_clones': 0, 'num_object_copies': 13636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3411, 'num_write_kb': 27278, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1182}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:22.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:42:22.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3473", 'reported_seq': 3534, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:18.986603+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:42:18.986603+0000', 'last_peered': '2024-09-16T04:42:18.986603+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:42:18.986603+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3473, 'log_dups_size': 0, 'ondisk_log_size': 3473, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28415244, 'num_objects': 3469, 'num_object_clones': 0, 'num_object_copies': 13876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3471, 'num_write_kb': 27757, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1240}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:22.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:42:22.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3470", 'reported_seq': 3521, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:18.313270+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:42:18.313270+0000', 'last_peered': '2024-09-16T04:42:18.313270+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:42:18.313270+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3470, 'log_dups_size': 0, 'ondisk_log_size': 3470, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28398860, 'num_objects': 3467, 'num_object_clones': 0, 'num_object_copies': 13868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3467, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3468, 'num_write_kb': 27741, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1184}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:22.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:42:22.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3350", 'reported_seq': 3389, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:18.368606+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:42:18.368606+0000', 'last_peered': '2024-09-16T04:42:18.368606+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:42:18.368606+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3350, 'log_dups_size': 0, 'ondisk_log_size': 3350, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27443200, 'num_objects': 3350, 'num_object_clones': 0, 'num_object_copies': 13400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3350, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3350, 'num_write_kb': 26800, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1142}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:22.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:42:22.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3481", 'reported_seq': 3541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:19.054644+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:42:19.054644+0000', 'last_peered': '2024-09-16T04:42:19.054644+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:42:19.054644+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3481, 'log_dups_size': 0, 'ondisk_log_size': 3481, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28464396, 'num_objects': 3475, 'num_object_clones': 0, 'num_object_copies': 13900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3475, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3479, 'num_write_kb': 27812, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1249}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:22.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:42:22.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3466", 'reported_seq': 4991, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:19.079164+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:42:19.079164+0000', 'last_peered': '2024-09-16T04:42:19.079164+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:42:19.079164+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3466, 'log_dups_size': 0, 'ondisk_log_size': 3466, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28349708, 'num_objects': 3461, 'num_object_clones': 0, 'num_object_copies': 13844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1505, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3464, 'num_write_kb': 27700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 735, 'num_bytes_recovered': 6004736, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1956}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:22.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:42:22.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3526", 'reported_seq': 3578, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:18.254059+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:42:18.254059+0000', 'last_peered': '2024-09-16T04:42:18.254059+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:42:18.254059+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3526, 'log_dups_size': 0, 'ondisk_log_size': 3526, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28813848, 'num_objects': 3518, 'num_object_clones': 0, 'num_object_copies': 14072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3523, 'num_write_kb': 28161, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1178}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:22.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:42:22.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3418", 'reported_seq': 3491, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:18.703028+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:42:18.703028+0000', 'last_peered': '2024-09-16T04:42:18.703028+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:42:18.703028+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3418, 'log_dups_size': 0, 'ondisk_log_size': 3418, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27981068, 'num_objects': 3416, 'num_object_clones': 0, 'num_object_copies': 13664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3416, 'num_write_kb': 27326, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1216}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:22.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:42:22.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3451", 'reported_seq': 3504, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:18.327466+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:42:18.327466+0000', 'last_peered': '2024-09-16T04:42:18.327466+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:42:18.327466+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3451, 'log_dups_size': 0, 'ondisk_log_size': 3451, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28225426, 'num_objects': 3446, 'num_object_clones': 0, 'num_object_copies': 13784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3446, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3448, 'num_write_kb': 27572, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1257}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:22.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:42:22.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3314", 'reported_seq': 3348, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:18.325399+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:42:18.325399+0000', 'last_peered': '2024-09-16T04:42:18.325399+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:42:18.325399+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3314, 'log_dups_size': 0, 'ondisk_log_size': 3314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27138694, 'num_objects': 3313, 'num_object_clones': 0, 'num_object_copies': 13252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3313, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3313, 'num_write_kb': 26503, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1185}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:22.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:42:22.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3400", 'reported_seq': 3469, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:18.747745+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:42:18.747745+0000', 'last_peered': '2024-09-16T04:42:18.747745+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:42:18.747745+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3400, 'log_dups_size': 0, 'ondisk_log_size': 3400, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27843206, 'num_objects': 3399, 'num_object_clones': 0, 'num_object_copies': 13596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3399, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3399, 'num_write_kb': 27191, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1174}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:22.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:42:22.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3440", 'reported_seq': 3489, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:18.327903+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:42:18.327903+0000', 'last_peered': '2024-09-16T04:42:18.327903+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:42:18.327903+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3440, 'log_dups_size': 0, 'ondisk_log_size': 3440, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28144908, 'num_objects': 3436, 'num_object_clones': 0, 'num_object_copies': 13744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3436, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3438, 'num_write_kb': 27500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1202}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:22.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:42:22.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3308", 'reported_seq': 3382, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:18.770935+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:42:18.770935+0000', 'last_peered': '2024-09-16T04:42:18.770935+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:42:18.770935+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3308, 'log_dups_size': 0, 'ondisk_log_size': 3308, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27071756, 'num_objects': 3305, 'num_object_clones': 0, 'num_object_copies': 13220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3306, 'num_write_kb': 26445, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1173}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:22.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:42:22.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3411", 'reported_seq': 3472, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:18.919816+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:42:18.919816+0000', 'last_peered': '2024-09-16T04:42:18.919816+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:42:18.919816+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3411, 'log_dups_size': 0, 'ondisk_log_size': 3411, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27899148, 'num_objects': 3406, 'num_object_clones': 0, 'num_object_copies': 13624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3409, 'num_write_kb': 27260, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:22.721 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:42:23.273 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 89 20 56251 56231 4.9343 2.39062 0.13513 0.0316438 2024-09-16T04:42:23.382 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:23.382 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34820/218688 objects degraded (15.922%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":155,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54672,"data_bytes":447833768,"bytes_used":1414053888,"bytes_avail":385133002752,"bytes_total":386547056640,"degraded_objects":34820,"degraded_total":218688,"degraded_ratio":0.15922227099795142,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":143991,"recovering_keys_per_sec":0,"num_objects_recovered":211,"num_bytes_recovered":1728512,"num_keys_recovered":0,"write_bytes_sec":2297725,"read_op_per_sec":0,"write_op_per_sec":280},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T04:41:44.568132+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:42:23.406 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:42:23.406 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:24.025 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:24.025 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:24.057 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-09-16T04:42:22.584902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":447833768,"num_objects":54672,"num_object_clones":0,"num_object_copies":218688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54699,"num_write_kb":437474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":760,"num_bytes_recovered":6004736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54726,"ondisk_log_size":54726,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1380912,"kb_used_data":736876,"kb_used_omap":14,"kb_used_meta":643953,"kb_avail":376106448,"statfs":{"total":386547056640,"available":385133002752,"internally_reserved":0,"allocated":754561024,"data_stored":753877784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":659408433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,6,7,12,17,14],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27582464,"num_objects":3367,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3367,"num_write_kb":26936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1728512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004246"},"pg_stats":[{"pgid":"1.f","version":"31'3400","reported_seq":3439,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.280177+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:18.280177+0000","last_peered":"2024-09-16T04:42:18.280177+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:18.280177+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3400,"log_dups_size":0,"ondisk_log_size":3400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27852800,"num_objects":3400,"num_object_clones":0,"num_object_copies":13600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3400,"num_write_kb":27200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3406","reported_seq":3467,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:19.064536+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:19.064536+0000","last_peered":"2024-09-16T04:42:19.064536+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:19.064536+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3406,"log_dups_size":0,"ondisk_log_size":3406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27866380,"num_objects":3402,"num_object_clones":0,"num_object_copies":13608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3404,"num_write_kb":27228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1191}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3412","reported_seq":3482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.813736+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:18.813736+0000","last_peered":"2024-09-16T04:42:18.813736+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:18.813736+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3412,"log_dups_size":0,"ondisk_log_size":3412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27925126,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3411,"num_write_kb":27278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1182}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3473","reported_seq":3534,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.986603+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:18.986603+0000","last_peered":"2024-09-16T04:42:18.986603+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:18.986603+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3473,"log_dups_size":0,"ondisk_log_size":3473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28415244,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":27757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1240}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3470","reported_seq":3521,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.313270+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:18.313270+0000","last_peered":"2024-09-16T04:42:18.313270+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:18.313270+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28398860,"num_objects":3467,"num_object_clones":0,"num_object_copies":13868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3468,"num_write_kb":27741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3350","reported_seq":3389,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.368606+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:18.368606+0000","last_peered":"2024-09-16T04:42:18.368606+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:18.368606+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27443200,"num_objects":3350,"num_object_clones":0,"num_object_copies":13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3350,"num_write_kb":26800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1142}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3481","reported_seq":3541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:19.054644+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:19.054644+0000","last_peered":"2024-09-16T04:42:19.054644+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:19.054644+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28464396,"num_objects":3475,"num_object_clones":0,"num_object_copies":13900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3479,"num_write_kb":27812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1249}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3466","reported_seq":4991,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:19.079164+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:19.079164+0000","last_peered":"2024-09-16T04:42:19.079164+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:19.079164+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3466,"log_dups_size":0,"ondisk_log_size":3466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28349708,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3464,"num_write_kb":27700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":735,"num_bytes_recovered":6004736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1956}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3526","reported_seq":3578,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.254059+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:18.254059+0000","last_peered":"2024-09-16T04:42:18.254059+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:18.254059+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28813848,"num_objects":3518,"num_object_clones":0,"num_object_copies":14072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3523,"num_write_kb":28161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3418","reported_seq":3491,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.703028+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:18.703028+0000","last_peered":"2024-09-16T04:42:18.703028+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:18.703028+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3418,"log_dups_size":0,"ondisk_log_size":3418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27981068,"num_objects":3416,"num_object_clones":0,"num_object_copies":13664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3416,"num_write_kb":27326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1216}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3451","reported_seq":3504,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.327466+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:18.327466+0000","last_peered":"2024-09-16T04:42:18.327466+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:18.327466+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3451,"log_dups_size":0,"ondisk_log_size":3451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28225426,"num_objects":3446,"num_object_clones":0,"num_object_copies":13784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3448,"num_write_kb":27572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3314","reported_seq":3348,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.325399+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:18.325399+0000","last_peered":"2024-09-16T04:42:18.325399+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:18.325399+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3314,"log_dups_size":0,"ondisk_log_size":3314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27138694,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3313,"num_write_kb":26503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1185}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3400","reported_seq":3469,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.747745+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:18.747745+0000","last_peered":"2024-09-16T04:42:18.747745+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:18.747745+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3400,"log_dups_size":0,"ondisk_log_size":3400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27843206,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3399,"num_write_kb":27191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1174}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3440","reported_seq":3489,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.327903+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:18.327903+0000","last_peered":"2024-09-16T04:42:18.327903+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:18.327903+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3440,"log_dups_size":0,"ondisk_log_size":3440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28144908,"num_objects":3436,"num_object_clones":0,"num_object_copies":13744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3438,"num_write_kb":27500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1202}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3308","reported_seq":3382,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.770935+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:18.770935+0000","last_peered":"2024-09-16T04:42:18.770935+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:18.770935+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3308,"log_dups_size":0,"ondisk_log_size":3308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27071756,"num_objects":3305,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3306,"num_write_kb":26445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1173}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3411","reported_seq":3472,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:18.919816+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:18.919816+0000","last_peered":"2024-09-16T04:42:18.919816+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:18.919816+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27899148,"num_objects":3406,"num_object_clones":0,"num_object_copies":13624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3409,"num_write_kb":27260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":447833768,"num_objects":54672,"num_object_clones":0,"num_object_copies":218688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54699,"num_write_kb":437474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":760,"num_bytes_recovered":6004736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":756789248,"data_stored":756789248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54726,"ondisk_log_size":54726,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402596,"kb_used_data":218196,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93969244,"statfs":{"total":96636764160,"available":96224505856,"internally_reserved":0,"allocated":223432704,"data_stored":223261894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":188804815},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393992,"kb_used_data":218420,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93977848,"statfs":{"total":96636764160,"available":96233316352,"internally_reserved":0,"allocated":223662080,"data_stored":223491270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181600,"kb_used_data":81872,"kb_used_omap":4,"kb_used_meta":99707,"kb_avail":94190240,"statfs":{"total":96636764160,"available":96450805760,"internally_reserved":0,"allocated":83836928,"data_stored":83666118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":102100687},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,4,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402724,"kb_used_data":218388,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93969116,"statfs":{"total":96636764160,"available":96224374784,"internally_reserved":0,"allocated":223629312,"data_stored":223458502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":188739278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,6,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":224493568,"data_stored":224493568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":84754432,"data_stored":84754432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":224096256,"data_stored":224096256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":223444992,"data_stored":223444992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:24.273 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 90 19 56586 56567 4.90864 2.625 0.0573952 0.0318183 2024-09-16T04:42:25.273 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 91 20 56959 56939 4.88664 2.90625 0.0790946 0.0319605 2024-09-16T04:42:26.273 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 92 20 57323 57303 4.86444 2.84375 0.055836 0.0321062 2024-09-16T04:42:27.059 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:27.273 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 93 20 57677 57657 4.84187 2.76562 0.0802917 0.0322551 2024-09-16T04:42:27.648 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:27.649 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:27.670 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-16T04:42:26.585774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":462169768,"num_objects":56422,"num_object_clones":0,"num_object_copies":225688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56449,"num_write_kb":451474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":6602752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56476,"ondisk_log_size":56476,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1392140,"kb_used_data":764748,"kb_used_omap":14,"kb_used_meta":627313,"kb_avail":376095220,"statfs":{"total":386547056640,"available":385121505280,"internally_reserved":0,"allocated":783101952,"data_stored":782418712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":642369073},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,11,5,6,12,16,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28672000,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3500,"num_write_kb":28000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004053"},"pg_stats":[{"pgid":"1.f","version":"31'3507","reported_seq":3546,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.347251+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:23.347251+0000","last_peered":"2024-09-16T04:42:23.347251+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:23.347251+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3507,"log_dups_size":0,"ondisk_log_size":3507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28729344,"num_objects":3507,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3507,"num_write_kb":28056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1288}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3518","reported_seq":3579,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:24.091907+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:24.091907+0000","last_peered":"2024-09-16T04:42:24.091907+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:24.091907+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3518,"log_dups_size":0,"ondisk_log_size":3518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28783884,"num_objects":3514,"num_object_clones":0,"num_object_copies":14056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3516,"num_write_kb":28124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1303}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3533","reported_seq":3603,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.797893+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:23.797893+0000","last_peered":"2024-09-16T04:42:23.797893+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:23.797893+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3533,"log_dups_size":0,"ondisk_log_size":3533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28916358,"num_objects":3530,"num_object_clones":0,"num_object_copies":14120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3532,"num_write_kb":28246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1303}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3591","reported_seq":3652,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:24.067068+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:24.067068+0000","last_peered":"2024-09-16T04:42:24.067068+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:24.067068+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3591,"log_dups_size":0,"ondisk_log_size":3591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29381900,"num_objects":3587,"num_object_clones":0,"num_object_copies":14348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3589,"num_write_kb":28701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1358}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3577","reported_seq":3628,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.398940+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:23.398940+0000","last_peered":"2024-09-16T04:42:23.398940+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:23.398940+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3577,"log_dups_size":0,"ondisk_log_size":3577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29275404,"num_objects":3574,"num_object_clones":0,"num_object_copies":14296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3575,"num_write_kb":28597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3461","reported_seq":3500,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.351709+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:23.351709+0000","last_peered":"2024-09-16T04:42:23.351709+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:23.351709+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3461,"log_dups_size":0,"ondisk_log_size":3461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28352512,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3461,"num_write_kb":27688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1253}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3589","reported_seq":3649,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:24.086892+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:24.086892+0000","last_peered":"2024-09-16T04:42:24.086892+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:24.086892+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3589,"log_dups_size":0,"ondisk_log_size":3589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29349132,"num_objects":3583,"num_object_clones":0,"num_object_copies":14332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3587,"num_write_kb":28676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1357}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3553","reported_seq":5225,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:24.076152+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:24.076152+0000","last_peered":"2024-09-16T04:42:24.076152+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:24.076152+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062412,"num_objects":3548,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":808,"num_bytes_recovered":6602752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2116}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3637","reported_seq":3689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.405418+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:23.405418+0000","last_peered":"2024-09-16T04:42:23.405418+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:23.405418+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3637,"log_dups_size":0,"ondisk_log_size":3637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29723160,"num_objects":3629,"num_object_clones":0,"num_object_copies":14516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3634,"num_write_kb":29049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1289}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3524","reported_seq":3598,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.634030+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:23.634030+0000","last_peered":"2024-09-16T04:42:23.634030+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:23.634030+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28849420,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3522,"num_write_kb":28174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1322}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3545","reported_seq":3598,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.352020+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:23.352020+0000","last_peered":"2024-09-16T04:42:23.352020+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:23.352020+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3545,"log_dups_size":0,"ondisk_log_size":3545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28995474,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3542,"num_write_kb":28324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3438","reported_seq":3473,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.419500+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:23.419500+0000","last_peered":"2024-09-16T04:42:23.419500+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:23.419500+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3438,"log_dups_size":0,"ondisk_log_size":3438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28154502,"num_objects":3437,"num_object_clones":0,"num_object_copies":13748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3437,"num_write_kb":27495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1309}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3517","reported_seq":3586,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.815399+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:23.815399+0000","last_peered":"2024-09-16T04:42:23.815399+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:23.815399+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28801670,"num_objects":3516,"num_object_clones":0,"num_object_copies":14064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3516,"num_write_kb":28127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1291}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3573","reported_seq":3622,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.353075+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:23.353075+0000","last_peered":"2024-09-16T04:42:23.353075+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:23.353075+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3573,"log_dups_size":0,"ondisk_log_size":3573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29234444,"num_objects":3569,"num_object_clones":0,"num_object_copies":14276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3571,"num_write_kb":28564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1335}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3408","reported_seq":3482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.661623+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:23.661623+0000","last_peered":"2024-09-16T04:42:23.661623+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:23.661623+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3408,"log_dups_size":0,"ondisk_log_size":3408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27890956,"num_objects":3405,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3406,"num_write_kb":27245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3505","reported_seq":3566,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.998709+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:23.998709+0000","last_peered":"2024-09-16T04:42:23.998709+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:23.998709+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3505,"log_dups_size":0,"ondisk_log_size":3505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28669196,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":28012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1261}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":462169768,"num_objects":56422,"num_object_clones":0,"num_object_copies":225688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56449,"num_write_kb":451474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":6602752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":785408000,"data_stored":785408000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56476,"ondisk_log_size":56476,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397088,"kb_used_data":225168,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93974752,"statfs":{"total":96636764160,"available":96230146048,"internally_reserved":0,"allocated":230572032,"data_stored":230401222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":176025295},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405092,"kb_used_data":225364,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93966748,"statfs":{"total":96636764160,"available":96221949952,"internally_reserved":0,"allocated":230772736,"data_stored":230601926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":192748,"kb_used_data":88860,"kb_used_omap":4,"kb_used_meta":103867,"kb_avail":94179092,"statfs":{"total":96636764160,"available":96439390208,"internally_reserved":0,"allocated":90992640,"data_stored":90821830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":106360527},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,3,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397212,"kb_used_data":225356,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":93974628,"statfs":{"total":96636764160,"available":96230019072,"internally_reserved":0,"allocated":230764544,"data_stored":230593734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":175959758},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,1,6,4,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":231526400,"data_stored":231526400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":92053504,"data_stored":92053504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":231182336,"data_stored":231182336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230645760,"data_stored":230645760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:27.671 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:28.248 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:28.248 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:28.273 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 94 20 57994 57974 4.81671 2.47656 0.0839158 0.0324235 2024-09-16T04:42:28.276 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-16T04:42:26.585774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":462169768,"num_objects":56422,"num_object_clones":0,"num_object_copies":225688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56449,"num_write_kb":451474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":6602752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56476,"ondisk_log_size":56476,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1392140,"kb_used_data":764748,"kb_used_omap":14,"kb_used_meta":627313,"kb_avail":376095220,"statfs":{"total":386547056640,"available":385121505280,"internally_reserved":0,"allocated":783101952,"data_stored":782418712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":642369073},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,11,5,6,12,16,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28672000,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3500,"num_write_kb":28000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004053"},"pg_stats":[{"pgid":"1.f","version":"31'3507","reported_seq":3546,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.347251+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:23.347251+0000","last_peered":"2024-09-16T04:42:23.347251+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:23.347251+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3507,"log_dups_size":0,"ondisk_log_size":3507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28729344,"num_objects":3507,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3507,"num_write_kb":28056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1288}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3518","reported_seq":3579,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:24.091907+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:24.091907+0000","last_peered":"2024-09-16T04:42:24.091907+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:24.091907+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3518,"log_dups_size":0,"ondisk_log_size":3518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28783884,"num_objects":3514,"num_object_clones":0,"num_object_copies":14056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3516,"num_write_kb":28124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1303}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3533","reported_seq":3603,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.797893+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:23.797893+0000","last_peered":"2024-09-16T04:42:23.797893+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:23.797893+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3533,"log_dups_size":0,"ondisk_log_size":3533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28916358,"num_objects":3530,"num_object_clones":0,"num_object_copies":14120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3532,"num_write_kb":28246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1303}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3591","reported_seq":3652,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:24.067068+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:24.067068+0000","last_peered":"2024-09-16T04:42:24.067068+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:24.067068+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3591,"log_dups_size":0,"ondisk_log_size":3591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29381900,"num_objects":3587,"num_object_clones":0,"num_object_copies":14348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3589,"num_write_kb":28701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1358}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3577","reported_seq":3628,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.398940+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:23.398940+0000","last_peered":"2024-09-16T04:42:23.398940+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:23.398940+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3577,"log_dups_size":0,"ondisk_log_size":3577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29275404,"num_objects":3574,"num_object_clones":0,"num_object_copies":14296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3575,"num_write_kb":28597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3461","reported_seq":3500,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.351709+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:23.351709+0000","last_peered":"2024-09-16T04:42:23.351709+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:23.351709+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3461,"log_dups_size":0,"ondisk_log_size":3461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28352512,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3461,"num_write_kb":27688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1253}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3589","reported_seq":3649,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:24.086892+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:24.086892+0000","last_peered":"2024-09-16T04:42:24.086892+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:24.086892+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3589,"log_dups_size":0,"ondisk_log_size":3589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29349132,"num_objects":3583,"num_object_clones":0,"num_object_copies":14332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3587,"num_write_kb":28676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1357}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3553","reported_seq":5225,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:24.076152+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:24.076152+0000","last_peered":"2024-09-16T04:42:24.076152+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:24.076152+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062412,"num_objects":3548,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":808,"num_bytes_recovered":6602752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2116}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3637","reported_seq":3689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.405418+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:23.405418+0000","last_peered":"2024-09-16T04:42:23.405418+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:23.405418+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3637,"log_dups_size":0,"ondisk_log_size":3637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29723160,"num_objects":3629,"num_object_clones":0,"num_object_copies":14516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3634,"num_write_kb":29049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1289}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3524","reported_seq":3598,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.634030+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:23.634030+0000","last_peered":"2024-09-16T04:42:23.634030+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:23.634030+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28849420,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3522,"num_write_kb":28174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1322}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3545","reported_seq":3598,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.352020+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:23.352020+0000","last_peered":"2024-09-16T04:42:23.352020+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:23.352020+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3545,"log_dups_size":0,"ondisk_log_size":3545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28995474,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3542,"num_write_kb":28324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3438","reported_seq":3473,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.419500+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:23.419500+0000","last_peered":"2024-09-16T04:42:23.419500+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:23.419500+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3438,"log_dups_size":0,"ondisk_log_size":3438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28154502,"num_objects":3437,"num_object_clones":0,"num_object_copies":13748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3437,"num_write_kb":27495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1309}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3517","reported_seq":3586,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.815399+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:23.815399+0000","last_peered":"2024-09-16T04:42:23.815399+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:23.815399+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28801670,"num_objects":3516,"num_object_clones":0,"num_object_copies":14064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3516,"num_write_kb":28127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1291}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3573","reported_seq":3622,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.353075+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:23.353075+0000","last_peered":"2024-09-16T04:42:23.353075+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:23.353075+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3573,"log_dups_size":0,"ondisk_log_size":3573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29234444,"num_objects":3569,"num_object_clones":0,"num_object_copies":14276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3571,"num_write_kb":28564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1335}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3408","reported_seq":3482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.661623+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:23.661623+0000","last_peered":"2024-09-16T04:42:23.661623+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:23.661623+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3408,"log_dups_size":0,"ondisk_log_size":3408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27890956,"num_objects":3405,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3406,"num_write_kb":27245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3505","reported_seq":3566,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.998709+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:23.998709+0000","last_peered":"2024-09-16T04:42:23.998709+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:23.998709+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3505,"log_dups_size":0,"ondisk_log_size":3505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28669196,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":28012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1261}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":462169768,"num_objects":56422,"num_object_clones":0,"num_object_copies":225688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56449,"num_write_kb":451474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":6602752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":785408000,"data_stored":785408000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56476,"ondisk_log_size":56476,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397088,"kb_used_data":225168,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93974752,"statfs":{"total":96636764160,"available":96230146048,"internally_reserved":0,"allocated":230572032,"data_stored":230401222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":176025295},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405092,"kb_used_data":225364,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93966748,"statfs":{"total":96636764160,"available":96221949952,"internally_reserved":0,"allocated":230772736,"data_stored":230601926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":192748,"kb_used_data":88860,"kb_used_omap":4,"kb_used_meta":103867,"kb_avail":94179092,"statfs":{"total":96636764160,"available":96439390208,"internally_reserved":0,"allocated":90992640,"data_stored":90821830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":106360527},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,3,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397212,"kb_used_data":225356,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":93974628,"statfs":{"total":96636764160,"available":96230019072,"internally_reserved":0,"allocated":230764544,"data_stored":230593734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":175959758},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,1,6,4,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":231526400,"data_stored":231526400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":92053504,"data_stored":92053504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":231182336,"data_stored":231182336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230645760,"data_stored":230645760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:28.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:42:28.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3507", 'reported_seq': 3546, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:23.347251+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:42:23.347251+0000', 'last_peered': '2024-09-16T04:42:23.347251+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:42:23.347251+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3507, 'log_dups_size': 0, 'ondisk_log_size': 3507, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28729344, 'num_objects': 3507, 'num_object_clones': 0, 'num_object_copies': 14028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3507, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3507, 'num_write_kb': 28056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1288}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:28.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:42:28.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3518", 'reported_seq': 3579, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:24.091907+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:42:24.091907+0000', 'last_peered': '2024-09-16T04:42:24.091907+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:42:24.091907+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3518, 'log_dups_size': 0, 'ondisk_log_size': 3518, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28783884, 'num_objects': 3514, 'num_object_clones': 0, 'num_object_copies': 14056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3514, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3516, 'num_write_kb': 28124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1303}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:28.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:42:28.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3533", 'reported_seq': 3603, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:23.797893+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:42:23.797893+0000', 'last_peered': '2024-09-16T04:42:23.797893+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:42:23.797893+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3533, 'log_dups_size': 0, 'ondisk_log_size': 3533, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28916358, 'num_objects': 3530, 'num_object_clones': 0, 'num_object_copies': 14120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3530, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3532, 'num_write_kb': 28246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1303}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:28.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:42:28.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3591", 'reported_seq': 3652, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:24.067068+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:42:24.067068+0000', 'last_peered': '2024-09-16T04:42:24.067068+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:42:24.067068+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3591, 'log_dups_size': 0, 'ondisk_log_size': 3591, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29381900, 'num_objects': 3587, 'num_object_clones': 0, 'num_object_copies': 14348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3587, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3589, 'num_write_kb': 28701, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1358}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:28.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:42:28.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3577", 'reported_seq': 3628, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:23.398940+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:42:23.398940+0000', 'last_peered': '2024-09-16T04:42:23.398940+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:42:23.398940+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3577, 'log_dups_size': 0, 'ondisk_log_size': 3577, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29275404, 'num_objects': 3574, 'num_object_clones': 0, 'num_object_copies': 14296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3574, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3575, 'num_write_kb': 28597, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:28.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:42:28.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3461", 'reported_seq': 3500, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:23.351709+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:42:23.351709+0000', 'last_peered': '2024-09-16T04:42:23.351709+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:42:23.351709+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3461, 'log_dups_size': 0, 'ondisk_log_size': 3461, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28352512, 'num_objects': 3461, 'num_object_clones': 0, 'num_object_copies': 13844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3461, 'num_write_kb': 27688, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1253}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:28.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:42:28.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3589", 'reported_seq': 3649, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:24.086892+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:42:24.086892+0000', 'last_peered': '2024-09-16T04:42:24.086892+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:42:24.086892+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3589, 'log_dups_size': 0, 'ondisk_log_size': 3589, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29349132, 'num_objects': 3583, 'num_object_clones': 0, 'num_object_copies': 14332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3587, 'num_write_kb': 28676, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1357}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:28.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:42:28.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3553", 'reported_seq': 5225, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:24.076152+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:42:24.076152+0000', 'last_peered': '2024-09-16T04:42:24.076152+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:42:24.076152+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3553, 'log_dups_size': 0, 'ondisk_log_size': 3553, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29062412, 'num_objects': 3548, 'num_object_clones': 0, 'num_object_copies': 14192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1432, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3551, 'num_write_kb': 28396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 808, 'num_bytes_recovered': 6602752, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2116}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:28.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:42:28.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3637", 'reported_seq': 3689, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:23.405418+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:42:23.405418+0000', 'last_peered': '2024-09-16T04:42:23.405418+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:42:23.405418+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3637, 'log_dups_size': 0, 'ondisk_log_size': 3637, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29723160, 'num_objects': 3629, 'num_object_clones': 0, 'num_object_copies': 14516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3629, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3634, 'num_write_kb': 29049, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1289}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:28.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:42:28.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3524", 'reported_seq': 3598, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:23.634030+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:42:23.634030+0000', 'last_peered': '2024-09-16T04:42:23.634030+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:42:23.634030+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3524, 'log_dups_size': 0, 'ondisk_log_size': 3524, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28849420, 'num_objects': 3522, 'num_object_clones': 0, 'num_object_copies': 14088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3522, 'num_write_kb': 28174, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1322}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:28.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:42:28.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3545", 'reported_seq': 3598, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:23.352020+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:42:23.352020+0000', 'last_peered': '2024-09-16T04:42:23.352020+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:42:23.352020+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3545, 'log_dups_size': 0, 'ondisk_log_size': 3545, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28995474, 'num_objects': 3540, 'num_object_clones': 0, 'num_object_copies': 14160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3542, 'num_write_kb': 28324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1351}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:28.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:42:28.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3438", 'reported_seq': 3473, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:23.419500+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:42:23.419500+0000', 'last_peered': '2024-09-16T04:42:23.419500+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:42:23.419500+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3438, 'log_dups_size': 0, 'ondisk_log_size': 3438, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28154502, 'num_objects': 3437, 'num_object_clones': 0, 'num_object_copies': 13748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3437, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3437, 'num_write_kb': 27495, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:28.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:42:28.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3517", 'reported_seq': 3586, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:23.815399+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:42:23.815399+0000', 'last_peered': '2024-09-16T04:42:23.815399+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:42:23.815399+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3517, 'log_dups_size': 0, 'ondisk_log_size': 3517, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28801670, 'num_objects': 3516, 'num_object_clones': 0, 'num_object_copies': 14064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3516, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3516, 'num_write_kb': 28127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1291}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:28.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:42:28.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3573", 'reported_seq': 3622, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:23.353075+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:42:23.353075+0000', 'last_peered': '2024-09-16T04:42:23.353075+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:42:23.353075+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3573, 'log_dups_size': 0, 'ondisk_log_size': 3573, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29234444, 'num_objects': 3569, 'num_object_clones': 0, 'num_object_copies': 14276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3569, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3571, 'num_write_kb': 28564, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1335}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:28.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:42:28.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3408", 'reported_seq': 3482, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:23.661623+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:42:23.661623+0000', 'last_peered': '2024-09-16T04:42:23.661623+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:42:23.661623+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3408, 'log_dups_size': 0, 'ondisk_log_size': 3408, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27890956, 'num_objects': 3405, 'num_object_clones': 0, 'num_object_copies': 13620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3406, 'num_write_kb': 27245, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1273}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:28.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:42:28.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3505", 'reported_seq': 3566, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:23.998709+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:42:23.998709+0000', 'last_peered': '2024-09-16T04:42:23.998709+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:42:23.998709+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3505, 'log_dups_size': 0, 'ondisk_log_size': 3505, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28669196, 'num_objects': 3500, 'num_object_clones': 0, 'num_object_copies': 14000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3500, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3503, 'num_write_kb': 28012, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1261}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:28.280 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:42:28.981 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:29.449 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34747/228588 objects degraded (15.201%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":161,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":57147,"data_bytes":468108968,"bytes_used":1436979200,"bytes_avail":385110077440,"bytes_total":386547056640,"degraded_objects":34747,"degraded_total":228588,"degraded_ratio":0.15200710448492485,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":99638,"recovering_keys_per_sec":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"write_bytes_sec":2883368,"read_op_per_sec":0,"write_op_per_sec":351},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:42:29.450 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:42:29.450 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:29.451 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 95 20 58331 58311 4.79373 2.63281 0.0627081 0.032584 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:30.118 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-09-16T04:42:28.586197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":468108968,"num_objects":57147,"num_object_clones":0,"num_object_copies":228588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57174,"num_write_kb":457274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":6602752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57201,"ondisk_log_size":57201,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1403300,"kb_used_data":771748,"kb_used_omap":14,"kb_used_meta":631473,"kb_avail":376084060,"statfs":{"total":386547056640,"available":385110077440,"internally_reserved":0,"allocated":790269952,"data_stored":789586712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":646628913},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,11,4,10,15,10,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34611200,"num_objects":4225,"num_object_clones":0,"num_object_copies":16900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4225,"num_write_kb":33800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003736"},"pg_stats":[{"pgid":"1.f","version":"31'3606","reported_seq":3644,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.279561+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:28.279561+0000","last_peered":"2024-09-16T04:42:28.279561+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:28.279561+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29540352,"num_objects":3606,"num_object_clones":0,"num_object_copies":14424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3606,"num_write_kb":28848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1387}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3518","reported_seq":3579,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:24.091907+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:24.091907+0000","last_peered":"2024-09-16T04:42:24.091907+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:24.091907+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3518,"log_dups_size":0,"ondisk_log_size":3518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28783884,"num_objects":3514,"num_object_clones":0,"num_object_copies":14056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3516,"num_write_kb":28124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1303}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3533","reported_seq":3603,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.797893+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:23.797893+0000","last_peered":"2024-09-16T04:42:23.797893+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:23.797893+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3533,"log_dups_size":0,"ondisk_log_size":3533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28916358,"num_objects":3530,"num_object_clones":0,"num_object_copies":14120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3532,"num_write_kb":28246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1303}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3591","reported_seq":3652,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:24.067068+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:24.067068+0000","last_peered":"2024-09-16T04:42:24.067068+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:24.067068+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3591,"log_dups_size":0,"ondisk_log_size":3591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29381900,"num_objects":3587,"num_object_clones":0,"num_object_copies":14348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3589,"num_write_kb":28701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1358}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3678","reported_seq":3729,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.238858+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:28.238858+0000","last_peered":"2024-09-16T04:42:28.238858+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:28.238858+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3678,"log_dups_size":0,"ondisk_log_size":3678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30102796,"num_objects":3675,"num_object_clones":0,"num_object_copies":14700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3676,"num_write_kb":29405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3599","reported_seq":3638,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.386503+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:28.386503+0000","last_peered":"2024-09-16T04:42:28.386503+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:28.386503+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29483008,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":28792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1391}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3589","reported_seq":3649,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:24.086892+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:24.086892+0000","last_peered":"2024-09-16T04:42:24.086892+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:24.086892+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3589,"log_dups_size":0,"ondisk_log_size":3589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29349132,"num_objects":3583,"num_object_clones":0,"num_object_copies":14332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3587,"num_write_kb":28676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1357}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3553","reported_seq":5225,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:24.076152+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:24.076152+0000","last_peered":"2024-09-16T04:42:24.076152+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:24.076152+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062412,"num_objects":3548,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3551,"num_write_kb":28396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":808,"num_bytes_recovered":6602752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2116}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3723","reported_seq":3775,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.290275+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:28.290275+0000","last_peered":"2024-09-16T04:42:28.290275+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:28.290275+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30427672,"num_objects":3715,"num_object_clones":0,"num_object_copies":14860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3720,"num_write_kb":29737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1375}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3524","reported_seq":3598,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.634030+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:23.634030+0000","last_peered":"2024-09-16T04:42:23.634030+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:23.634030+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28849420,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3522,"num_write_kb":28174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1322}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3638","reported_seq":3691,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.414944+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:28.414944+0000","last_peered":"2024-09-16T04:42:28.414944+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:28.414944+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3638,"log_dups_size":0,"ondisk_log_size":3638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29757330,"num_objects":3633,"num_object_clones":0,"num_object_copies":14532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3635,"num_write_kb":29068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1444}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3541","reported_seq":3576,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.369184+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:28.369184+0000","last_peered":"2024-09-16T04:42:28.369184+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:28.369184+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3541,"log_dups_size":0,"ondisk_log_size":3541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28998278,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3540,"num_write_kb":28319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1412}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3517","reported_seq":3586,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.815399+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:23.815399+0000","last_peered":"2024-09-16T04:42:23.815399+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:23.815399+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28801670,"num_objects":3516,"num_object_clones":0,"num_object_copies":14064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3516,"num_write_kb":28127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1291}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3678","reported_seq":3727,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.381782+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:28.381782+0000","last_peered":"2024-09-16T04:42:28.381782+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:28.381782+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3678,"log_dups_size":0,"ondisk_log_size":3678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30094604,"num_objects":3674,"num_object_clones":0,"num_object_copies":14696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3676,"num_write_kb":29404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1440}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3408","reported_seq":3482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.661623+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:23.661623+0000","last_peered":"2024-09-16T04:42:23.661623+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:23.661623+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3408,"log_dups_size":0,"ondisk_log_size":3408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27890956,"num_objects":3405,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3406,"num_write_kb":27245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3505","reported_seq":3566,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:23.998709+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:23.998709+0000","last_peered":"2024-09-16T04:42:23.998709+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:23.998709+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3505,"log_dups_size":0,"ondisk_log_size":3505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28669196,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":28012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1261}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":468108968,"num_objects":57147,"num_object_clones":0,"num_object_copies":228588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57174,"num_write_kb":457274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":6602752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":792547328,"data_stored":792547328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57201,"ondisk_log_size":57201,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408248,"kb_used_data":232168,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93963592,"statfs":{"total":96636764160,"available":96218718208,"internally_reserved":0,"allocated":237740032,"data_stored":237569222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180285135},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,6,5,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405092,"kb_used_data":225364,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93966748,"statfs":{"total":96636764160,"available":96221949952,"internally_reserved":0,"allocated":230772736,"data_stored":230601926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":192748,"kb_used_data":88860,"kb_used_omap":4,"kb_used_meta":103867,"kb_avail":94179092,"statfs":{"total":96636764160,"available":96439390208,"internally_reserved":0,"allocated":90992640,"data_stored":90821830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":106360527},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,3,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397212,"kb_used_data":225356,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":93974628,"statfs":{"total":96636764160,"available":96230019072,"internally_reserved":0,"allocated":230764544,"data_stored":230593734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":175959758},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,1,6,4,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":231526400,"data_stored":231526400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":92053504,"data_stored":92053504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":231182336,"data_stored":231182336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":237785088,"data_stored":237785088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:30.274 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 96 20 58655 58635 4.77016 2.53125 0.0649402 0.0327391 2024-09-16T04:42:31.274 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 97 20 59015 58995 4.74999 2.8125 0.0705303 0.0328802 2024-09-16T04:42:32.274 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 98 20 59368 59348 4.72966 2.75781 0.0472262 0.0330176 2024-09-16T04:42:33.120 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:33.274 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 99 20 59726 59706 4.71014 2.79688 0.0523041 0.033153 2024-09-16T04:42:33.714 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:33.715 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:33.738 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-09-16T04:42:32.587242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":476563112,"num_objects":58179,"num_object_clones":0,"num_object_copies":232716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58206,"num_write_kb":465530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":895,"num_bytes_recovered":7110656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58233,"ondisk_log_size":58233,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1459104,"kb_used_data":792864,"kb_used_omap":14,"kb_used_meta":666161,"kb_avail":376028256,"statfs":{"total":386547056640,"available":385052934144,"internally_reserved":0,"allocated":811892736,"data_stored":811209496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":682149425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,19,5,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28729344,"num_objects":3507,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3507,"num_write_kb":28056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":1105920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003106"},"pg_stats":[{"pgid":"1.f","version":"31'3606","reported_seq":3644,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.279561+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:28.279561+0000","last_peered":"2024-09-16T04:42:28.279561+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:28.279561+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29540352,"num_objects":3606,"num_object_clones":0,"num_object_copies":14424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3606,"num_write_kb":28848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1387}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3639","reported_seq":3700,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:29.092705+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:29.092705+0000","last_peered":"2024-09-16T04:42:29.092705+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:29.092705+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3639,"log_dups_size":0,"ondisk_log_size":3639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29775116,"num_objects":3635,"num_object_clones":0,"num_object_copies":14540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3637,"num_write_kb":29092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3655","reported_seq":3726,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.853770+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:28.853770+0000","last_peered":"2024-09-16T04:42:28.853770+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:28.853770+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3655,"log_dups_size":0,"ondisk_log_size":3655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29915782,"num_objects":3652,"num_object_clones":0,"num_object_copies":14608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3654,"num_write_kb":29222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1425}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3701","reported_seq":3762,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:29.052759+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:29.052759+0000","last_peered":"2024-09-16T04:42:29.052759+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:29.052759+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3701,"log_dups_size":0,"ondisk_log_size":3701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30283020,"num_objects":3697,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3699,"num_write_kb":29581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1468}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3678","reported_seq":3729,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.238858+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:28.238858+0000","last_peered":"2024-09-16T04:42:28.238858+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:28.238858+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3678,"log_dups_size":0,"ondisk_log_size":3678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30102796,"num_objects":3675,"num_object_clones":0,"num_object_copies":14700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3676,"num_write_kb":29405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3599","reported_seq":3638,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.386503+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:28.386503+0000","last_peered":"2024-09-16T04:42:28.386503+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:28.386503+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29483008,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":28792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1391}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3705","reported_seq":3765,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:29.086888+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:29.086888+0000","last_peered":"2024-09-16T04:42:29.086888+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:29.086888+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3705,"log_dups_size":0,"ondisk_log_size":3705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30299404,"num_objects":3699,"num_object_clones":0,"num_object_copies":14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3703,"num_write_kb":29604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1473}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3673","reported_seq":5469,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:29.145415+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:29.145415+0000","last_peered":"2024-09-16T04:42:29.145415+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:29.145415+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3673,"log_dups_size":0,"ondisk_log_size":3673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30045452,"num_objects":3668,"num_object_clones":0,"num_object_copies":14672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3671,"num_write_kb":29356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":870,"num_bytes_recovered":7110656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2298}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3723","reported_seq":3775,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.290275+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:28.290275+0000","last_peered":"2024-09-16T04:42:28.290275+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:28.290275+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30427672,"num_objects":3715,"num_object_clones":0,"num_object_copies":14860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3720,"num_write_kb":29737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1375}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3621","reported_seq":3694,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.810574+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:28.810574+0000","last_peered":"2024-09-16T04:42:28.810574+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:28.810574+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3621,"log_dups_size":0,"ondisk_log_size":3621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29644044,"num_objects":3619,"num_object_clones":0,"num_object_copies":14476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3619,"num_write_kb":28950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1419}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3638","reported_seq":3691,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.414944+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:28.414944+0000","last_peered":"2024-09-16T04:42:28.414944+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:28.414944+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3638,"log_dups_size":0,"ondisk_log_size":3638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29757330,"num_objects":3633,"num_object_clones":0,"num_object_copies":14532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3635,"num_write_kb":29068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1444}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3541","reported_seq":3576,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.369184+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:28.369184+0000","last_peered":"2024-09-16T04:42:28.369184+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:28.369184+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3541,"log_dups_size":0,"ondisk_log_size":3541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28998278,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3540,"num_write_kb":28319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1412}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3628","reported_seq":3696,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.826863+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:28.826863+0000","last_peered":"2024-09-16T04:42:28.826863+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:28.826863+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3628,"log_dups_size":0,"ondisk_log_size":3628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29710982,"num_objects":3627,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3627,"num_write_kb":29015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1402}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3678","reported_seq":3727,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.381782+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:28.381782+0000","last_peered":"2024-09-16T04:42:28.381782+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:28.381782+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3678,"log_dups_size":0,"ondisk_log_size":3678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30094604,"num_objects":3674,"num_object_clones":0,"num_object_copies":14696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3676,"num_write_kb":29404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1440}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3526","reported_seq":3600,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.674534+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:28.674534+0000","last_peered":"2024-09-16T04:42:28.674534+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:28.674534+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28857612,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3524,"num_write_kb":28189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1391}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3622","reported_seq":3682,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:29.139674+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:29.139674+0000","last_peered":"2024-09-16T04:42:29.139674+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:29.139674+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29627660,"num_objects":3617,"num_object_clones":0,"num_object_copies":14468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":28948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1378}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":476563112,"num_objects":58179,"num_object_clones":0,"num_object_copies":232716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58206,"num_write_kb":465530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":895,"num_bytes_recovered":7110656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":814145536,"data_stored":814145536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58233,"ondisk_log_size":58233,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408248,"kb_used_data":232168,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93963592,"statfs":{"total":96636764160,"available":96218718208,"internally_reserved":0,"allocated":237740032,"data_stored":237569222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180285135},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,6,5,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416168,"kb_used_data":232280,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93955672,"statfs":{"total":96636764160,"available":96210608128,"internally_reserved":0,"allocated":237854720,"data_stored":237683910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,0,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051598,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":226424,"kb_used_data":96168,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94145416,"statfs":{"total":96636764160,"available":96404905984,"internally_reserved":0,"allocated":98476032,"data_stored":98305222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133361359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408264,"kb_used_data":232248,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93963576,"statfs":{"total":96636764160,"available":96218701824,"internally_reserved":0,"allocated":237821952,"data_stored":237651142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":180219598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":238743552,"data_stored":238743552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99291136,"data_stored":99291136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":238325760,"data_stored":238325760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":237785088,"data_stored":237785088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:33.739 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:34.274 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:2024-09-16T04:42:34.273735+0000 min lat: 0.00269411 max lat: 1.73107 avg lat: 0.0333176 2024-09-16T04:42:34.274 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T04:42:34.274 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 100 20 60034 60014 4.6871 2.40625 0.0383725 0.0333176 2024-09-16T04:42:34.319 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:34.320 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:34.341 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-09-16T04:42:32.587242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":476563112,"num_objects":58179,"num_object_clones":0,"num_object_copies":232716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58206,"num_write_kb":465530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":895,"num_bytes_recovered":7110656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58233,"ondisk_log_size":58233,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1459104,"kb_used_data":792864,"kb_used_omap":14,"kb_used_meta":666161,"kb_avail":376028256,"statfs":{"total":386547056640,"available":385052934144,"internally_reserved":0,"allocated":811892736,"data_stored":811209496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":682149425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,19,5,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28729344,"num_objects":3507,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3507,"num_write_kb":28056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":1105920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003106"},"pg_stats":[{"pgid":"1.f","version":"31'3606","reported_seq":3644,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.279561+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:28.279561+0000","last_peered":"2024-09-16T04:42:28.279561+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:28.279561+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29540352,"num_objects":3606,"num_object_clones":0,"num_object_copies":14424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3606,"num_write_kb":28848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1387}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3639","reported_seq":3700,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:29.092705+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:29.092705+0000","last_peered":"2024-09-16T04:42:29.092705+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:29.092705+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3639,"log_dups_size":0,"ondisk_log_size":3639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29775116,"num_objects":3635,"num_object_clones":0,"num_object_copies":14540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3637,"num_write_kb":29092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3655","reported_seq":3726,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.853770+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:28.853770+0000","last_peered":"2024-09-16T04:42:28.853770+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:28.853770+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3655,"log_dups_size":0,"ondisk_log_size":3655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29915782,"num_objects":3652,"num_object_clones":0,"num_object_copies":14608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3654,"num_write_kb":29222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1425}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3701","reported_seq":3762,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:29.052759+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:29.052759+0000","last_peered":"2024-09-16T04:42:29.052759+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:29.052759+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3701,"log_dups_size":0,"ondisk_log_size":3701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30283020,"num_objects":3697,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3699,"num_write_kb":29581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1468}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3678","reported_seq":3729,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.238858+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:28.238858+0000","last_peered":"2024-09-16T04:42:28.238858+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:28.238858+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3678,"log_dups_size":0,"ondisk_log_size":3678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30102796,"num_objects":3675,"num_object_clones":0,"num_object_copies":14700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3676,"num_write_kb":29405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3599","reported_seq":3638,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.386503+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:28.386503+0000","last_peered":"2024-09-16T04:42:28.386503+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:28.386503+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29483008,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":28792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1391}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3705","reported_seq":3765,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:29.086888+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:29.086888+0000","last_peered":"2024-09-16T04:42:29.086888+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:29.086888+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3705,"log_dups_size":0,"ondisk_log_size":3705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30299404,"num_objects":3699,"num_object_clones":0,"num_object_copies":14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3703,"num_write_kb":29604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1473}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3673","reported_seq":5469,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:29.145415+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:29.145415+0000","last_peered":"2024-09-16T04:42:29.145415+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:29.145415+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3673,"log_dups_size":0,"ondisk_log_size":3673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30045452,"num_objects":3668,"num_object_clones":0,"num_object_copies":14672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3671,"num_write_kb":29356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":870,"num_bytes_recovered":7110656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2298}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3723","reported_seq":3775,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.290275+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:28.290275+0000","last_peered":"2024-09-16T04:42:28.290275+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:28.290275+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30427672,"num_objects":3715,"num_object_clones":0,"num_object_copies":14860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3720,"num_write_kb":29737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1375}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3621","reported_seq":3694,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.810574+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:28.810574+0000","last_peered":"2024-09-16T04:42:28.810574+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:28.810574+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3621,"log_dups_size":0,"ondisk_log_size":3621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29644044,"num_objects":3619,"num_object_clones":0,"num_object_copies":14476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3619,"num_write_kb":28950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1419}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3638","reported_seq":3691,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.414944+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:28.414944+0000","last_peered":"2024-09-16T04:42:28.414944+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:28.414944+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3638,"log_dups_size":0,"ondisk_log_size":3638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29757330,"num_objects":3633,"num_object_clones":0,"num_object_copies":14532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3635,"num_write_kb":29068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1444}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3541","reported_seq":3576,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.369184+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:28.369184+0000","last_peered":"2024-09-16T04:42:28.369184+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:28.369184+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3541,"log_dups_size":0,"ondisk_log_size":3541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28998278,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3540,"num_write_kb":28319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1412}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3628","reported_seq":3696,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.826863+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:28.826863+0000","last_peered":"2024-09-16T04:42:28.826863+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:28.826863+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3628,"log_dups_size":0,"ondisk_log_size":3628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29710982,"num_objects":3627,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3627,"num_write_kb":29015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1402}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3678","reported_seq":3727,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.381782+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:28.381782+0000","last_peered":"2024-09-16T04:42:28.381782+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:28.381782+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3678,"log_dups_size":0,"ondisk_log_size":3678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30094604,"num_objects":3674,"num_object_clones":0,"num_object_copies":14696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3676,"num_write_kb":29404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1440}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3526","reported_seq":3600,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:28.674534+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:28.674534+0000","last_peered":"2024-09-16T04:42:28.674534+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:28.674534+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28857612,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3524,"num_write_kb":28189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1391}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3622","reported_seq":3682,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:29.139674+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:29.139674+0000","last_peered":"2024-09-16T04:42:29.139674+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:29.139674+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29627660,"num_objects":3617,"num_object_clones":0,"num_object_copies":14468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":28948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1378}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":476563112,"num_objects":58179,"num_object_clones":0,"num_object_copies":232716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58206,"num_write_kb":465530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":895,"num_bytes_recovered":7110656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":814145536,"data_stored":814145536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58233,"ondisk_log_size":58233,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408248,"kb_used_data":232168,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93963592,"statfs":{"total":96636764160,"available":96218718208,"internally_reserved":0,"allocated":237740032,"data_stored":237569222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":180285135},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,6,5,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416168,"kb_used_data":232280,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93955672,"statfs":{"total":96636764160,"available":96210608128,"internally_reserved":0,"allocated":237854720,"data_stored":237683910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,0,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051598,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":226424,"kb_used_data":96168,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94145416,"statfs":{"total":96636764160,"available":96404905984,"internally_reserved":0,"allocated":98476032,"data_stored":98305222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133361359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408264,"kb_used_data":232248,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93963576,"statfs":{"total":96636764160,"available":96218701824,"internally_reserved":0,"allocated":237821952,"data_stored":237651142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":180219598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":238743552,"data_stored":238743552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99291136,"data_stored":99291136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":238325760,"data_stored":238325760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":237785088,"data_stored":237785088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:34.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:42:34.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3606", 'reported_seq': 3644, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:28.279561+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:42:28.279561+0000', 'last_peered': '2024-09-16T04:42:28.279561+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:42:28.279561+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3606, 'log_dups_size': 0, 'ondisk_log_size': 3606, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29540352, 'num_objects': 3606, 'num_object_clones': 0, 'num_object_copies': 14424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3606, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3606, 'num_write_kb': 28848, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1387}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:34.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:42:34.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3639", 'reported_seq': 3700, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:29.092705+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:42:29.092705+0000', 'last_peered': '2024-09-16T04:42:29.092705+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:42:29.092705+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3639, 'log_dups_size': 0, 'ondisk_log_size': 3639, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29775116, 'num_objects': 3635, 'num_object_clones': 0, 'num_object_copies': 14540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3635, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3637, 'num_write_kb': 29092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1424}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:34.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:42:34.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3655", 'reported_seq': 3726, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:28.853770+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:42:28.853770+0000', 'last_peered': '2024-09-16T04:42:28.853770+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:42:28.853770+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3655, 'log_dups_size': 0, 'ondisk_log_size': 3655, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29915782, 'num_objects': 3652, 'num_object_clones': 0, 'num_object_copies': 14608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3652, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3654, 'num_write_kb': 29222, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1425}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:34.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:42:34.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3701", 'reported_seq': 3762, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:29.052759+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:42:29.052759+0000', 'last_peered': '2024-09-16T04:42:29.052759+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:42:29.052759+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3701, 'log_dups_size': 0, 'ondisk_log_size': 3701, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30283020, 'num_objects': 3697, 'num_object_clones': 0, 'num_object_copies': 14788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3699, 'num_write_kb': 29581, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1468}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:34.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:42:34.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3678", 'reported_seq': 3729, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:28.238858+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:42:28.238858+0000', 'last_peered': '2024-09-16T04:42:28.238858+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:42:28.238858+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3678, 'log_dups_size': 0, 'ondisk_log_size': 3678, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30102796, 'num_objects': 3675, 'num_object_clones': 0, 'num_object_copies': 14700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3675, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3676, 'num_write_kb': 29405, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1392}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:34.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:42:34.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3599", 'reported_seq': 3638, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:28.386503+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:42:28.386503+0000', 'last_peered': '2024-09-16T04:42:28.386503+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:42:28.386503+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3599, 'log_dups_size': 0, 'ondisk_log_size': 3599, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29483008, 'num_objects': 3599, 'num_object_clones': 0, 'num_object_copies': 14396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3599, 'num_write_kb': 28792, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1391}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:34.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:42:34.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3705", 'reported_seq': 3765, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:29.086888+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:42:29.086888+0000', 'last_peered': '2024-09-16T04:42:29.086888+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:42:29.086888+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3705, 'log_dups_size': 0, 'ondisk_log_size': 3705, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30299404, 'num_objects': 3699, 'num_object_clones': 0, 'num_object_copies': 14796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3703, 'num_write_kb': 29604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1473}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:34.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:42:34.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3673", 'reported_seq': 5469, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:29.145415+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:42:29.145415+0000', 'last_peered': '2024-09-16T04:42:29.145415+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:42:29.145415+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3673, 'log_dups_size': 0, 'ondisk_log_size': 3673, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30045452, 'num_objects': 3668, 'num_object_clones': 0, 'num_object_copies': 14672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1370, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3668, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3671, 'num_write_kb': 29356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 870, 'num_bytes_recovered': 7110656, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2298}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:34.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:42:34.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3723", 'reported_seq': 3775, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:28.290275+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:42:28.290275+0000', 'last_peered': '2024-09-16T04:42:28.290275+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:42:28.290275+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3723, 'log_dups_size': 0, 'ondisk_log_size': 3723, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30427672, 'num_objects': 3715, 'num_object_clones': 0, 'num_object_copies': 14860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3720, 'num_write_kb': 29737, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1375}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:34.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:42:34.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3621", 'reported_seq': 3694, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:28.810574+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:42:28.810574+0000', 'last_peered': '2024-09-16T04:42:28.810574+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:42:28.810574+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3621, 'log_dups_size': 0, 'ondisk_log_size': 3621, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29644044, 'num_objects': 3619, 'num_object_clones': 0, 'num_object_copies': 14476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3619, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3619, 'num_write_kb': 28950, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1419}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:34.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:42:34.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3638", 'reported_seq': 3691, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:28.414944+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:42:28.414944+0000', 'last_peered': '2024-09-16T04:42:28.414944+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:42:28.414944+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3638, 'log_dups_size': 0, 'ondisk_log_size': 3638, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29757330, 'num_objects': 3633, 'num_object_clones': 0, 'num_object_copies': 14532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3635, 'num_write_kb': 29068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1444}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:34.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:42:34.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3541", 'reported_seq': 3576, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:28.369184+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:42:28.369184+0000', 'last_peered': '2024-09-16T04:42:28.369184+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:42:28.369184+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3541, 'log_dups_size': 0, 'ondisk_log_size': 3541, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28998278, 'num_objects': 3540, 'num_object_clones': 0, 'num_object_copies': 14160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3540, 'num_write_kb': 28319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1412}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:34.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:42:34.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3628", 'reported_seq': 3696, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:28.826863+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:42:28.826863+0000', 'last_peered': '2024-09-16T04:42:28.826863+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:42:28.826863+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3628, 'log_dups_size': 0, 'ondisk_log_size': 3628, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29710982, 'num_objects': 3627, 'num_object_clones': 0, 'num_object_copies': 14508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3627, 'num_write_kb': 29015, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1402}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:34.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:42:34.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3678", 'reported_seq': 3727, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:28.381782+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:42:28.381782+0000', 'last_peered': '2024-09-16T04:42:28.381782+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:42:28.381782+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3678, 'log_dups_size': 0, 'ondisk_log_size': 3678, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30094604, 'num_objects': 3674, 'num_object_clones': 0, 'num_object_copies': 14696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3674, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3676, 'num_write_kb': 29404, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1440}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:34.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:42:34.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3526", 'reported_seq': 3600, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:28.674534+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:42:28.674534+0000', 'last_peered': '2024-09-16T04:42:28.674534+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:42:28.674534+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3526, 'log_dups_size': 0, 'ondisk_log_size': 3526, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28857612, 'num_objects': 3523, 'num_object_clones': 0, 'num_object_copies': 14092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3524, 'num_write_kb': 28189, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1391}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:34.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:42:34.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3622", 'reported_seq': 3682, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:29.139674+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:42:29.139674+0000', 'last_peered': '2024-09-16T04:42:29.139674+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:42:29.139674+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3622, 'log_dups_size': 0, 'ondisk_log_size': 3622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29627660, 'num_objects': 3617, 'num_object_clones': 0, 'num_object_copies': 14468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3617, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3620, 'num_write_kb': 28948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1378}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:34.347 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:42:35.027 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:35.028 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34619/239636 objects degraded (14.446%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":167,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":59909,"data_bytes":490735272,"bytes_used":1570975744,"bytes_avail":384976080896,"bytes_total":386547056640,"degraded_objects":34619,"degraded_total":239636,"degraded_ratio":0.14446493849004324,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":137183,"recovering_keys_per_sec":0,"num_objects_recovered":201,"num_bytes_recovered":1646592,"num_keys_recovered":0,"write_bytes_sec":3574287,"read_op_per_sec":0,"write_op_per_sec":436},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:42:35.057 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:42:35.058 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:35.274 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 101 20 60351 60331 4.66522 2.47656 0.045597 0.0334789 2024-09-16T04:42:35.618 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:35.619 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:35.647 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-09-16T04:42:34.587715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":490735272,"num_objects":59909,"num_object_clones":0,"num_object_copies":239636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59936,"num_write_kb":479370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":7651328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59963,"ondisk_log_size":59963,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1534156,"kb_used_data":813772,"kb_used_omap":14,"kb_used_meta":720305,"kb_avail":375953204,"statfs":{"total":386547056640,"available":384976080896,"internally_reserved":0,"allocated":833302528,"data_stored":832619288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":737592881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,19,9,10,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42901504,"num_objects":5237,"num_object_clones":0,"num_object_copies":20948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5237,"num_write_kb":41896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":1646592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002812"},"pg_stats":[{"pgid":"1.f","version":"31'3717","reported_seq":3755,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.462248+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:33.462248+0000","last_peered":"2024-09-16T04:42:33.462248+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:33.462248+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30449664,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":29736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3761","reported_seq":3821,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:34.086287+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:34.086287+0000","last_peered":"2024-09-16T04:42:34.086287+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:34.086287+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3761,"log_dups_size":0,"ondisk_log_size":3761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30774540,"num_objects":3757,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3759,"num_write_kb":30068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1546}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3750","reported_seq":3821,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.847760+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:33.847760+0000","last_peered":"2024-09-16T04:42:33.847760+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:33.847760+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3750,"log_dups_size":0,"ondisk_log_size":3750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30694022,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3749,"num_write_kb":29982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1520}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3805","reported_seq":3866,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:34.117043+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:34.117043+0000","last_peered":"2024-09-16T04:42:34.117043+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:34.117043+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3805,"log_dups_size":0,"ondisk_log_size":3805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31134988,"num_objects":3801,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3803,"num_write_kb":30413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3784","reported_seq":3835,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.457561+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:33.457561+0000","last_peered":"2024-09-16T04:42:33.457561+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:33.457561+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3784,"log_dups_size":0,"ondisk_log_size":3784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30971148,"num_objects":3781,"num_object_clones":0,"num_object_copies":15124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3782,"num_write_kb":30253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3700","reported_seq":3739,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.437293+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:33.437293+0000","last_peered":"2024-09-16T04:42:33.437293+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:33.437293+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3700,"log_dups_size":0,"ondisk_log_size":3700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30310400,"num_objects":3700,"num_object_clones":0,"num_object_copies":14800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3700,"num_write_kb":29600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1492}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3806","reported_seq":3866,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:34.114131+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:34.114131+0000","last_peered":"2024-09-16T04:42:34.114131+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:34.114131+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3806,"log_dups_size":0,"ondisk_log_size":3806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31126796,"num_objects":3800,"num_object_clones":0,"num_object_copies":15200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3804,"num_write_kb":30412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1574}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3776","reported_seq":5703,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:34.118285+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:34.118285+0000","last_peered":"2024-09-16T04:42:34.118285+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:34.118285+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3776,"log_dups_size":0,"ondisk_log_size":3776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30889228,"num_objects":3771,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3774,"num_write_kb":30180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":936,"num_bytes_recovered":7651328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2467}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3832","reported_seq":3884,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.356785+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:33.356785+0000","last_peered":"2024-09-16T04:42:33.356785+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:33.356785+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3832,"log_dups_size":0,"ondisk_log_size":3832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31320600,"num_objects":3824,"num_object_clones":0,"num_object_copies":15296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3829,"num_write_kb":30609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1484}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3732","reported_seq":3806,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.813137+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:33.813137+0000","last_peered":"2024-09-16T04:42:33.813137+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:33.813137+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3732,"log_dups_size":0,"ondisk_log_size":3732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30553356,"num_objects":3730,"num_object_clones":0,"num_object_copies":14920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3730,"num_write_kb":29838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1530}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3747","reported_seq":3800,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.431650+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:33.431650+0000","last_peered":"2024-09-16T04:42:33.431650+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:33.431650+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3747,"log_dups_size":0,"ondisk_log_size":3747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30650258,"num_objects":3742,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3744,"num_write_kb":29940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3641","reported_seq":3676,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.323737+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:33.323737+0000","last_peered":"2024-09-16T04:42:33.323737+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:33.323737+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3641,"log_dups_size":0,"ondisk_log_size":3641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29817478,"num_objects":3640,"num_object_clones":0,"num_object_copies":14560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3640,"num_write_kb":29119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1512}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3742","reported_seq":3811,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.807545+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:33.807545+0000","last_peered":"2024-09-16T04:42:33.807545+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:33.807545+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3742,"log_dups_size":0,"ondisk_log_size":3742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30644870,"num_objects":3741,"num_object_clones":0,"num_object_copies":14964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3741,"num_write_kb":29927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1516}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3799","reported_seq":3848,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.448086+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:33.448086+0000","last_peered":"2024-09-16T04:42:33.448086+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:33.448086+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3799,"log_dups_size":0,"ondisk_log_size":3799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31085836,"num_objects":3795,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3797,"num_write_kb":30372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1561}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3644","reported_seq":3718,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.856866+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:33.856866+0000","last_peered":"2024-09-16T04:42:33.856866+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:33.856866+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3644,"log_dups_size":0,"ondisk_log_size":3644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29824268,"num_objects":3641,"num_object_clones":0,"num_object_copies":14564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3642,"num_write_kb":29133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1509}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3727","reported_seq":3788,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.964875+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:33.964875+0000","last_peered":"2024-09-16T04:42:33.964875+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:33.964875+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3727,"log_dups_size":0,"ondisk_log_size":3727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30487820,"num_objects":3722,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3725,"num_write_kb":29788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1483}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":490735272,"num_objects":59909,"num_object_clones":0,"num_object_copies":239636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59936,"num_write_kb":479370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":7651328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":835424256,"data_stored":835424256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59963,"ondisk_log_size":59963,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437444,"kb_used_data":239156,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93934396,"statfs":{"total":96636764160,"available":96188821504,"internally_reserved":0,"allocated":244895744,"data_stored":244724934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":203026127},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,15,2,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":428728,"kb_used_data":239272,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93943112,"statfs":{"total":96636764160,"available":96197746688,"internally_reserved":0,"allocated":245014528,"data_stored":244843718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051598,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":226424,"kb_used_data":96168,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94145416,"statfs":{"total":96636764160,"available":96404905984,"internally_reserved":0,"allocated":98476032,"data_stored":98305222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133361359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":441560,"kb_used_data":239176,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93930280,"statfs":{"total":96636764160,"available":96184606720,"internally_reserved":0,"allocated":244916224,"data_stored":244745414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":207220430},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":245772288,"data_stored":245772288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99291136,"data_stored":99291136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":245403648,"data_stored":245403648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":244957184,"data_stored":244957184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:36.274 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 102 20 60691 60671 4.64553 2.65625 0.040406 0.0336186 2024-09-16T04:42:37.274 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 103 20 61050 61030 4.62766 2.80469 0.0772749 0.0337507 2024-09-16T04:42:38.274 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 104 20 61402 61382 4.6096 2.75 0.0403639 0.0338803 2024-09-16T04:42:38.649 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:39.235 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:39.235 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:39.257 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-09-16T04:42:38.588776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":496772776,"num_objects":60646,"num_object_clones":0,"num_object_copies":242584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60673,"num_write_kb":485266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":7651328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60700,"ondisk_log_size":60700,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1574256,"kb_used_data":827504,"kb_used_omap":14,"kb_used_meta":746673,"kb_avail":375913104,"statfs":{"total":386547056640,"available":384935018496,"internally_reserved":0,"allocated":847364096,"data_stored":846680856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":764593713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,7,8,14,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34603008,"num_objects":4224,"num_object_clones":0,"num_object_copies":16896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4224,"num_write_kb":33792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003002"},"pg_stats":[{"pgid":"1.f","version":"31'3811","reported_seq":3849,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.397960+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:38.397960+0000","last_peered":"2024-09-16T04:42:38.397960+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:38.397960+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3811,"log_dups_size":0,"ondisk_log_size":3811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31219712,"num_objects":3811,"num_object_clones":0,"num_object_copies":15244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3811,"num_write_kb":30488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3761","reported_seq":3821,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:34.086287+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:34.086287+0000","last_peered":"2024-09-16T04:42:34.086287+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:34.086287+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3761,"log_dups_size":0,"ondisk_log_size":3761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30774540,"num_objects":3757,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3759,"num_write_kb":30068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1546}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3750","reported_seq":3821,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.847760+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:33.847760+0000","last_peered":"2024-09-16T04:42:33.847760+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:33.847760+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3750,"log_dups_size":0,"ondisk_log_size":3750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30694022,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3749,"num_write_kb":29982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1520}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3805","reported_seq":3866,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:34.117043+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:34.117043+0000","last_peered":"2024-09-16T04:42:34.117043+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:34.117043+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3805,"log_dups_size":0,"ondisk_log_size":3805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31134988,"num_objects":3801,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3803,"num_write_kb":30413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3894","reported_seq":3945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.493636+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:38.493636+0000","last_peered":"2024-09-16T04:42:38.493636+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:38.493636+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31872268,"num_objects":3891,"num_object_clones":0,"num_object_copies":15564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3892,"num_write_kb":31133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3809","reported_seq":3848,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.463446+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:38.463446+0000","last_peered":"2024-09-16T04:42:38.463446+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:38.463446+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31203328,"num_objects":3809,"num_object_clones":0,"num_object_copies":15236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3809,"num_write_kb":30472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3806","reported_seq":3866,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:34.114131+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:34.114131+0000","last_peered":"2024-09-16T04:42:34.114131+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:34.114131+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3806,"log_dups_size":0,"ondisk_log_size":3806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31126796,"num_objects":3800,"num_object_clones":0,"num_object_copies":15200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3804,"num_write_kb":30412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1574}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3776","reported_seq":5703,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:34.118285+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:34.118285+0000","last_peered":"2024-09-16T04:42:34.118285+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:34.118285+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3776,"log_dups_size":0,"ondisk_log_size":3776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30889228,"num_objects":3771,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3774,"num_write_kb":30180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":936,"num_bytes_recovered":7651328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2467}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3939","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.354041+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:38.354041+0000","last_peered":"2024-09-16T04:42:38.354041+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:38.354041+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32197144,"num_objects":3931,"num_object_clones":0,"num_object_copies":15724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3936,"num_write_kb":31465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1591}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3732","reported_seq":3806,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.813137+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:33.813137+0000","last_peered":"2024-09-16T04:42:33.813137+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:33.813137+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3732,"log_dups_size":0,"ondisk_log_size":3732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30553356,"num_objects":3730,"num_object_clones":0,"num_object_copies":14920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3730,"num_write_kb":29838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1530}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3847","reported_seq":3900,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.493115+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:38.493115+0000","last_peered":"2024-09-16T04:42:38.493115+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:38.493115+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3847,"log_dups_size":0,"ondisk_log_size":3847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31469458,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3844,"num_write_kb":30740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1653}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3757","reported_seq":3791,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.501853+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:38.501853+0000","last_peered":"2024-09-16T04:42:38.501853+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:38.501853+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3757,"log_dups_size":0,"ondisk_log_size":3757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30767750,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3756,"num_write_kb":30047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1628}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3742","reported_seq":3811,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.807545+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:33.807545+0000","last_peered":"2024-09-16T04:42:33.807545+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:33.807545+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3742,"log_dups_size":0,"ondisk_log_size":3742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30644870,"num_objects":3741,"num_object_clones":0,"num_object_copies":14964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3741,"num_write_kb":29927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1516}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3900","reported_seq":3949,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.412345+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:38.412345+0000","last_peered":"2024-09-16T04:42:38.412345+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:38.412345+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31913228,"num_objects":3896,"num_object_clones":0,"num_object_copies":15584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3898,"num_write_kb":31180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1662}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3644","reported_seq":3718,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.856866+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:33.856866+0000","last_peered":"2024-09-16T04:42:33.856866+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:33.856866+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3644,"log_dups_size":0,"ondisk_log_size":3644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29824268,"num_objects":3641,"num_object_clones":0,"num_object_copies":14564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3642,"num_write_kb":29133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1509}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3727","reported_seq":3788,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.964875+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:33.964875+0000","last_peered":"2024-09-16T04:42:33.964875+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:33.964875+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3727,"log_dups_size":0,"ondisk_log_size":3727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30487820,"num_objects":3722,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3725,"num_write_kb":29788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1483}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":496772776,"num_objects":60646,"num_object_clones":0,"num_object_copies":242584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60673,"num_write_kb":485266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":7651328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":849498112,"data_stored":849498112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60700,"ondisk_log_size":60700,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466396,"kb_used_data":245900,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93905444,"statfs":{"total":96636764160,"available":96159174656,"internally_reserved":0,"allocated":251801600,"data_stored":251630790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":225767119},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":428728,"kb_used_data":239272,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93943112,"statfs":{"total":96636764160,"available":96197746688,"internally_reserved":0,"allocated":245014528,"data_stored":244843718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":237572,"kb_used_data":103156,"kb_used_omap":4,"kb_used_meta":134395,"kb_avail":94134268,"statfs":{"total":96636764160,"available":96393490432,"internally_reserved":0,"allocated":105631744,"data_stored":105460934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":137621199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":441560,"kb_used_data":239176,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93930280,"statfs":{"total":96636764160,"available":96184606720,"internally_reserved":0,"allocated":244916224,"data_stored":244745414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":207220430},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":245772288,"data_stored":245772288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106496000,"data_stored":106496000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":245403648,"data_stored":245403648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":251826176,"data_stored":251826176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:39.259 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:39.274 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 105 20 61711 61691 4.58869 2.41406 0.0487772 0.0340378 2024-09-16T04:42:39.825 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:39.826 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:39.848 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-09-16T04:42:38.588776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":496772776,"num_objects":60646,"num_object_clones":0,"num_object_copies":242584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60673,"num_write_kb":485266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":7651328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60700,"ondisk_log_size":60700,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1574256,"kb_used_data":827504,"kb_used_omap":14,"kb_used_meta":746673,"kb_avail":375913104,"statfs":{"total":386547056640,"available":384935018496,"internally_reserved":0,"allocated":847364096,"data_stored":846680856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":764593713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,7,8,14,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34603008,"num_objects":4224,"num_object_clones":0,"num_object_copies":16896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4224,"num_write_kb":33792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003002"},"pg_stats":[{"pgid":"1.f","version":"31'3811","reported_seq":3849,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.397960+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:38.397960+0000","last_peered":"2024-09-16T04:42:38.397960+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:38.397960+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3811,"log_dups_size":0,"ondisk_log_size":3811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31219712,"num_objects":3811,"num_object_clones":0,"num_object_copies":15244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3811,"num_write_kb":30488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3761","reported_seq":3821,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:34.086287+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:34.086287+0000","last_peered":"2024-09-16T04:42:34.086287+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:34.086287+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3761,"log_dups_size":0,"ondisk_log_size":3761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30774540,"num_objects":3757,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3759,"num_write_kb":30068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1546}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3750","reported_seq":3821,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.847760+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:33.847760+0000","last_peered":"2024-09-16T04:42:33.847760+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:33.847760+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3750,"log_dups_size":0,"ondisk_log_size":3750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30694022,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3749,"num_write_kb":29982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1520}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3805","reported_seq":3866,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:34.117043+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:34.117043+0000","last_peered":"2024-09-16T04:42:34.117043+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:34.117043+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3805,"log_dups_size":0,"ondisk_log_size":3805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31134988,"num_objects":3801,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3803,"num_write_kb":30413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3894","reported_seq":3945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.493636+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:38.493636+0000","last_peered":"2024-09-16T04:42:38.493636+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:38.493636+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31872268,"num_objects":3891,"num_object_clones":0,"num_object_copies":15564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3892,"num_write_kb":31133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3809","reported_seq":3848,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.463446+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:38.463446+0000","last_peered":"2024-09-16T04:42:38.463446+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:38.463446+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31203328,"num_objects":3809,"num_object_clones":0,"num_object_copies":15236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3809,"num_write_kb":30472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3806","reported_seq":3866,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:34.114131+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:34.114131+0000","last_peered":"2024-09-16T04:42:34.114131+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:34.114131+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3806,"log_dups_size":0,"ondisk_log_size":3806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31126796,"num_objects":3800,"num_object_clones":0,"num_object_copies":15200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3804,"num_write_kb":30412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1574}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3776","reported_seq":5703,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:34.118285+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:34.118285+0000","last_peered":"2024-09-16T04:42:34.118285+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:34.118285+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3776,"log_dups_size":0,"ondisk_log_size":3776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30889228,"num_objects":3771,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3774,"num_write_kb":30180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":936,"num_bytes_recovered":7651328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2467}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3939","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.354041+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:38.354041+0000","last_peered":"2024-09-16T04:42:38.354041+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:38.354041+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32197144,"num_objects":3931,"num_object_clones":0,"num_object_copies":15724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3936,"num_write_kb":31465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1591}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3732","reported_seq":3806,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.813137+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:33.813137+0000","last_peered":"2024-09-16T04:42:33.813137+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:33.813137+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3732,"log_dups_size":0,"ondisk_log_size":3732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30553356,"num_objects":3730,"num_object_clones":0,"num_object_copies":14920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3730,"num_write_kb":29838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1530}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3847","reported_seq":3900,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.493115+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:38.493115+0000","last_peered":"2024-09-16T04:42:38.493115+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:38.493115+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3847,"log_dups_size":0,"ondisk_log_size":3847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31469458,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3844,"num_write_kb":30740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1653}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3757","reported_seq":3791,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.501853+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:38.501853+0000","last_peered":"2024-09-16T04:42:38.501853+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:38.501853+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3757,"log_dups_size":0,"ondisk_log_size":3757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30767750,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3756,"num_write_kb":30047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1628}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3742","reported_seq":3811,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.807545+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:33.807545+0000","last_peered":"2024-09-16T04:42:33.807545+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:33.807545+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3742,"log_dups_size":0,"ondisk_log_size":3742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30644870,"num_objects":3741,"num_object_clones":0,"num_object_copies":14964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3741,"num_write_kb":29927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1516}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3900","reported_seq":3949,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.412345+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:38.412345+0000","last_peered":"2024-09-16T04:42:38.412345+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:38.412345+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31913228,"num_objects":3896,"num_object_clones":0,"num_object_copies":15584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3898,"num_write_kb":31180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1662}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3644","reported_seq":3718,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.856866+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:33.856866+0000","last_peered":"2024-09-16T04:42:33.856866+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:33.856866+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3644,"log_dups_size":0,"ondisk_log_size":3644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29824268,"num_objects":3641,"num_object_clones":0,"num_object_copies":14564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3642,"num_write_kb":29133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1509}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3727","reported_seq":3788,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:33.964875+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:33.964875+0000","last_peered":"2024-09-16T04:42:33.964875+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:33.964875+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3727,"log_dups_size":0,"ondisk_log_size":3727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30487820,"num_objects":3722,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3725,"num_write_kb":29788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1483}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":496772776,"num_objects":60646,"num_object_clones":0,"num_object_copies":242584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60673,"num_write_kb":485266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":7651328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":849498112,"data_stored":849498112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60700,"ondisk_log_size":60700,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466396,"kb_used_data":245900,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93905444,"statfs":{"total":96636764160,"available":96159174656,"internally_reserved":0,"allocated":251801600,"data_stored":251630790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":225767119},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":428728,"kb_used_data":239272,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93943112,"statfs":{"total":96636764160,"available":96197746688,"internally_reserved":0,"allocated":245014528,"data_stored":244843718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":237572,"kb_used_data":103156,"kb_used_omap":4,"kb_used_meta":134395,"kb_avail":94134268,"statfs":{"total":96636764160,"available":96393490432,"internally_reserved":0,"allocated":105631744,"data_stored":105460934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":137621199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":441560,"kb_used_data":239176,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93930280,"statfs":{"total":96636764160,"available":96184606720,"internally_reserved":0,"allocated":244916224,"data_stored":244745414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":207220430},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":245772288,"data_stored":245772288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106496000,"data_stored":106496000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":245403648,"data_stored":245403648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":251826176,"data_stored":251826176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:39.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:42:39.849 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3811", 'reported_seq': 3849, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:38.397960+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:42:38.397960+0000', 'last_peered': '2024-09-16T04:42:38.397960+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:42:38.397960+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3811, 'log_dups_size': 0, 'ondisk_log_size': 3811, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31219712, 'num_objects': 3811, 'num_object_clones': 0, 'num_object_copies': 15244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3811, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3811, 'num_write_kb': 30488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1592}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:39.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:42:39.849 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3761", 'reported_seq': 3821, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:34.086287+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:42:34.086287+0000', 'last_peered': '2024-09-16T04:42:34.086287+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:42:34.086287+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3761, 'log_dups_size': 0, 'ondisk_log_size': 3761, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30774540, 'num_objects': 3757, 'num_object_clones': 0, 'num_object_copies': 15028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3759, 'num_write_kb': 30068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1546}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:39.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:42:39.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3750", 'reported_seq': 3821, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:33.847760+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:42:33.847760+0000', 'last_peered': '2024-09-16T04:42:33.847760+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:42:33.847760+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3750, 'log_dups_size': 0, 'ondisk_log_size': 3750, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30694022, 'num_objects': 3747, 'num_object_clones': 0, 'num_object_copies': 14988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3747, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3749, 'num_write_kb': 29982, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1520}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:39.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:42:39.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3805", 'reported_seq': 3866, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:34.117043+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:42:34.117043+0000', 'last_peered': '2024-09-16T04:42:34.117043+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:42:34.117043+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3805, 'log_dups_size': 0, 'ondisk_log_size': 3805, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31134988, 'num_objects': 3801, 'num_object_clones': 0, 'num_object_copies': 15204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3801, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3803, 'num_write_kb': 30413, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1572}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:39.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:42:39.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3894", 'reported_seq': 3945, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:38.493636+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:42:38.493636+0000', 'last_peered': '2024-09-16T04:42:38.493636+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:42:38.493636+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3894, 'log_dups_size': 0, 'ondisk_log_size': 3894, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31872268, 'num_objects': 3891, 'num_object_clones': 0, 'num_object_copies': 15564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3892, 'num_write_kb': 31133, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1608}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:39.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:42:39.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3809", 'reported_seq': 3848, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:38.463446+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:42:38.463446+0000', 'last_peered': '2024-09-16T04:42:38.463446+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:42:38.463446+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3809, 'log_dups_size': 0, 'ondisk_log_size': 3809, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31203328, 'num_objects': 3809, 'num_object_clones': 0, 'num_object_copies': 15236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3809, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3809, 'num_write_kb': 30472, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1601}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:39.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:42:39.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3806", 'reported_seq': 3866, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:34.114131+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:42:34.114131+0000', 'last_peered': '2024-09-16T04:42:34.114131+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:42:34.114131+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3806, 'log_dups_size': 0, 'ondisk_log_size': 3806, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31126796, 'num_objects': 3800, 'num_object_clones': 0, 'num_object_copies': 15200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3800, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3804, 'num_write_kb': 30412, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1574}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:39.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:42:39.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3776", 'reported_seq': 5703, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:34.118285+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:42:34.118285+0000', 'last_peered': '2024-09-16T04:42:34.118285+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:42:34.118285+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3776, 'log_dups_size': 0, 'ondisk_log_size': 3776, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30889228, 'num_objects': 3771, 'num_object_clones': 0, 'num_object_copies': 15084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3771, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3774, 'num_write_kb': 30180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 936, 'num_bytes_recovered': 7651328, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2467}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:39.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:42:39.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3939", 'reported_seq': 3991, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:38.354041+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:42:38.354041+0000', 'last_peered': '2024-09-16T04:42:38.354041+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:42:38.354041+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3939, 'log_dups_size': 0, 'ondisk_log_size': 3939, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32197144, 'num_objects': 3931, 'num_object_clones': 0, 'num_object_copies': 15724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3936, 'num_write_kb': 31465, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1591}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:39.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:42:39.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3732", 'reported_seq': 3806, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:33.813137+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:42:33.813137+0000', 'last_peered': '2024-09-16T04:42:33.813137+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:42:33.813137+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3732, 'log_dups_size': 0, 'ondisk_log_size': 3732, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30553356, 'num_objects': 3730, 'num_object_clones': 0, 'num_object_copies': 14920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3730, 'num_write_kb': 29838, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1530}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:39.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:42:39.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3847", 'reported_seq': 3900, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:38.493115+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:42:38.493115+0000', 'last_peered': '2024-09-16T04:42:38.493115+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:42:38.493115+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3847, 'log_dups_size': 0, 'ondisk_log_size': 3847, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31469458, 'num_objects': 3842, 'num_object_clones': 0, 'num_object_copies': 15368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3844, 'num_write_kb': 30740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1653}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:39.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:42:39.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3757", 'reported_seq': 3791, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:38.501853+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:42:38.501853+0000', 'last_peered': '2024-09-16T04:42:38.501853+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:42:38.501853+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3757, 'log_dups_size': 0, 'ondisk_log_size': 3757, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30767750, 'num_objects': 3756, 'num_object_clones': 0, 'num_object_copies': 15024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3756, 'num_write_kb': 30047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1628}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:39.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:42:39.852 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3742", 'reported_seq': 3811, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:33.807545+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:42:33.807545+0000', 'last_peered': '2024-09-16T04:42:33.807545+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:42:33.807545+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3742, 'log_dups_size': 0, 'ondisk_log_size': 3742, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30644870, 'num_objects': 3741, 'num_object_clones': 0, 'num_object_copies': 14964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3741, 'num_write_kb': 29927, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1516}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:39.852 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:42:39.852 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3900", 'reported_seq': 3949, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:38.412345+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:42:38.412345+0000', 'last_peered': '2024-09-16T04:42:38.412345+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:42:38.412345+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3900, 'log_dups_size': 0, 'ondisk_log_size': 3900, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31913228, 'num_objects': 3896, 'num_object_clones': 0, 'num_object_copies': 15584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3896, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3898, 'num_write_kb': 31180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1662}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:39.852 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:42:39.852 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3644", 'reported_seq': 3718, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:33.856866+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:42:33.856866+0000', 'last_peered': '2024-09-16T04:42:33.856866+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:42:33.856866+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3644, 'log_dups_size': 0, 'ondisk_log_size': 3644, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29824268, 'num_objects': 3641, 'num_object_clones': 0, 'num_object_copies': 14564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3641, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3642, 'num_write_kb': 29133, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1509}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:39.852 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:42:39.852 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3727", 'reported_seq': 3788, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:33.964875+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:42:33.964875+0000', 'last_peered': '2024-09-16T04:42:33.964875+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:42:33.964875+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3727, 'log_dups_size': 0, 'ondisk_log_size': 3727, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30487820, 'num_objects': 3722, 'num_object_clones': 0, 'num_object_copies': 14888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3725, 'num_write_kb': 29788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1483}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:39.852 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:42:40.275 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 106 20 61993 61973 4.56619 2.20312 0.0444356 0.0341746 2024-09-16T04:42:40.514 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:40.514 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34619/242584 objects degraded (14.271%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":172,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":60646,"data_bytes":496772776,"bytes_used":1612038144,"bytes_avail":384935018496,"bytes_total":386547056640,"degraded_objects":34619,"degraded_total":242584,"degraded_ratio":0.14270932955182536,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":87359,"recovering_keys_per_sec":0,"num_objects_recovered":128,"num_bytes_recovered":1048576,"num_keys_recovered":0,"write_bytes_sec":2882862,"read_op_per_sec":0,"write_op_per_sec":351},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:42:40.540 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:42:40.541 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:41.159 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:41.160 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:41.184 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-16T04:42:40.589314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":504538792,"num_objects":61594,"num_object_clones":0,"num_object_copies":246376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61621,"num_write_kb":492850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1027,"num_bytes_recovered":8192000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61648,"ondisk_log_size":61648,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1636140,"kb_used_data":840812,"kb_used_omap":14,"kb_used_meta":795249,"kb_avail":375851220,"statfs":{"total":386547056640,"available":384871649280,"internally_reserved":0,"allocated":860991488,"data_stored":860308248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":814335537},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,6,6,12,10,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36429824,"num_objects":4447,"num_object_clones":0,"num_object_copies":17788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4447,"num_write_kb":35576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":1589248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003116"},"pg_stats":[{"pgid":"1.f","version":"31'3811","reported_seq":3849,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.397960+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:38.397960+0000","last_peered":"2024-09-16T04:42:38.397960+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:38.397960+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3811,"log_dups_size":0,"ondisk_log_size":3811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31219712,"num_objects":3811,"num_object_clones":0,"num_object_copies":15244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3811,"num_write_kb":30488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3887","reported_seq":3947,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:39.220398+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:39.220398+0000","last_peered":"2024-09-16T04:42:39.220398+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:39.220398+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3887,"log_dups_size":0,"ondisk_log_size":3887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31806732,"num_objects":3883,"num_object_clones":0,"num_object_copies":15532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3885,"num_write_kb":31076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1672}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3852","reported_seq":3923,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.772120+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:38.772120+0000","last_peered":"2024-09-16T04:42:38.772120+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:38.772120+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3852,"log_dups_size":0,"ondisk_log_size":3852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31529606,"num_objects":3849,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3851,"num_write_kb":30798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1622}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3896","reported_seq":3957,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:39.228994+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:39.228994+0000","last_peered":"2024-09-16T04:42:39.228994+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:39.228994+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3896,"log_dups_size":0,"ondisk_log_size":3896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31880460,"num_objects":3892,"num_object_clones":0,"num_object_copies":15568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3894,"num_write_kb":31141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3894","reported_seq":3945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.493636+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:38.493636+0000","last_peered":"2024-09-16T04:42:38.493636+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:38.493636+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31872268,"num_objects":3891,"num_object_clones":0,"num_object_copies":15564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3892,"num_write_kb":31133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3809","reported_seq":3848,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.463446+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:38.463446+0000","last_peered":"2024-09-16T04:42:38.463446+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:38.463446+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31203328,"num_objects":3809,"num_object_clones":0,"num_object_copies":15236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3809,"num_write_kb":30472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3899","reported_seq":3959,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:39.200273+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:39.200273+0000","last_peered":"2024-09-16T04:42:39.200273+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:39.200273+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3899,"log_dups_size":0,"ondisk_log_size":3899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31888652,"num_objects":3893,"num_object_clones":0,"num_object_copies":15572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3897,"num_write_kb":31156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1667}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3877","reported_seq":5937,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:39.157007+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:39.157007+0000","last_peered":"2024-09-16T04:42:39.157007+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:39.157007+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3877,"log_dups_size":0,"ondisk_log_size":3877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31716620,"num_objects":3872,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3875,"num_write_kb":30988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1002,"num_bytes_recovered":8192000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2634}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3939","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.354041+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:38.354041+0000","last_peered":"2024-09-16T04:42:38.354041+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:38.354041+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32197144,"num_objects":3931,"num_object_clones":0,"num_object_copies":15724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3936,"num_write_kb":31465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1591}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3859","reported_seq":3933,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.807510+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:38.807510+0000","last_peered":"2024-09-16T04:42:38.807510+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:38.807510+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3859,"log_dups_size":0,"ondisk_log_size":3859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31593740,"num_objects":3857,"num_object_clones":0,"num_object_copies":15428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3857,"num_write_kb":30854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1657}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3847","reported_seq":3900,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.493115+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:38.493115+0000","last_peered":"2024-09-16T04:42:38.493115+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:38.493115+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3847,"log_dups_size":0,"ondisk_log_size":3847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31469458,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3844,"num_write_kb":30740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1653}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3757","reported_seq":3791,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.501853+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:38.501853+0000","last_peered":"2024-09-16T04:42:38.501853+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:38.501853+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3757,"log_dups_size":0,"ondisk_log_size":3757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30767750,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3756,"num_write_kb":30047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1628}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3850","reported_seq":3919,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.907992+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:38.907992+0000","last_peered":"2024-09-16T04:42:38.907992+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:38.907992+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31529606,"num_objects":3849,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3849,"num_write_kb":30791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1624}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3900","reported_seq":3949,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.412345+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:38.412345+0000","last_peered":"2024-09-16T04:42:38.412345+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:38.412345+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31913228,"num_objects":3896,"num_object_clones":0,"num_object_copies":15584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3898,"num_write_kb":31180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1662}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3746","reported_seq":3821,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:38.812112+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:38.812112+0000","last_peered":"2024-09-16T04:42:38.812112+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:38.812112+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3746,"log_dups_size":0,"ondisk_log_size":3746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30659852,"num_objects":3743,"num_object_clones":0,"num_object_copies":14972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3744,"num_write_kb":29949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1611}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3825","reported_seq":3886,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:39.109461+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:39.109461+0000","last_peered":"2024-09-16T04:42:39.109461+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:39.109461+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3825,"log_dups_size":0,"ondisk_log_size":3825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31290636,"num_objects":3820,"num_object_clones":0,"num_object_copies":15280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3823,"num_write_kb":30572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1581}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":504538792,"num_objects":61594,"num_object_clones":0,"num_object_copies":246376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61621,"num_write_kb":492850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1027,"num_bytes_recovered":8192000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":863342592,"data_stored":863342592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61648,"ondisk_log_size":61648,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466396,"kb_used_data":245900,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93905444,"statfs":{"total":96636764160,"available":96159174656,"internally_reserved":0,"allocated":251801600,"data_stored":251630790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":225767119},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461684,"kb_used_data":245860,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93910156,"statfs":{"total":96636764160,"available":96163999744,"internally_reserved":0,"allocated":251760640,"data_stored":251589830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":237572,"kb_used_data":103156,"kb_used_omap":4,"kb_used_meta":134395,"kb_avail":94134268,"statfs":{"total":96636764160,"available":96393490432,"internally_reserved":0,"allocated":105631744,"data_stored":105460934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":137621199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470488,"kb_used_data":245896,"kb_used_omap":4,"kb_used_meta":224571,"kb_avail":93901352,"statfs":{"total":96636764160,"available":96154984448,"internally_reserved":0,"allocated":251797504,"data_stored":251626694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":229961422},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,0,8,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252715008,"data_stored":252715008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106496000,"data_stored":106496000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":252305408,"data_stored":252305408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":251826176,"data_stored":251826176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:41.275 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 107 20 62307 62287 4.54644 2.45312 0.056218 0.0343486 2024-09-16T04:42:42.275 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 108 20 62661 62641 4.52996 2.76562 0.0552441 0.0344828 2024-09-16T04:42:43.275 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 109 20 63015 62995 4.51377 2.76562 0.041787 0.034606 2024-09-16T04:42:44.185 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:44.275 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 110 20 63358 63338 4.4971 2.67969 0.0530671 0.0347299 2024-09-16T04:42:44.771 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:44.771 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:44.805 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-09-16T04:42:44.590314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":517973672,"num_objects":63234,"num_object_clones":0,"num_object_copies":252936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63261,"num_write_kb":505970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1105,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63288,"ondisk_log_size":63288,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1630528,"kb_used_data":868480,"kb_used_omap":14,"kb_used_meta":761969,"kb_avail":375856832,"statfs":{"total":386547056640,"available":384877395968,"internally_reserved":0,"allocated":889323520,"data_stored":888640280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":780256817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,5,14,11,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41410560,"num_objects":5055,"num_object_clones":0,"num_object_copies":20220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5055,"num_write_kb":40440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003071"},"pg_stats":[{"pgid":"1.f","version":"31'3925","reported_seq":3964,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.573612+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:43.573612+0000","last_peered":"2024-09-16T04:42:43.573612+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:43.573612+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32153600,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3982","reported_seq":4043,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.202258+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:44.202258+0000","last_peered":"2024-09-16T04:42:44.202258+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:44.202258+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3982,"log_dups_size":0,"ondisk_log_size":3982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32584972,"num_objects":3978,"num_object_clones":0,"num_object_copies":15912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3980,"num_write_kb":31836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1767}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3960","reported_seq":4031,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.841380+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:43.841380+0000","last_peered":"2024-09-16T04:42:43.841380+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:43.841380+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3960,"log_dups_size":0,"ondisk_log_size":3960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32414342,"num_objects":3957,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3959,"num_write_kb":31662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1730}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3999","reported_seq":4060,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.266775+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:44.266775+0000","last_peered":"2024-09-16T04:42:44.266775+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:44.266775+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3999,"log_dups_size":0,"ondisk_log_size":3999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32724236,"num_objects":3995,"num_object_clones":0,"num_object_copies":15980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3997,"num_write_kb":31965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1766}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3983","reported_seq":4034,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.474905+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:43.474905+0000","last_peered":"2024-09-16T04:42:43.474905+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:43.474905+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32601356,"num_objects":3980,"num_object_clones":0,"num_object_copies":15920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3981,"num_write_kb":31845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3925","reported_seq":3964,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.518478+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:43.518478+0000","last_peered":"2024-09-16T04:42:43.518478+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:43.518478+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32153600,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3998","reported_seq":4059,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.260361+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:44.260361+0000","last_peered":"2024-09-16T04:42:44.260361+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:44.260361+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699660,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3996,"num_write_kb":31948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1766}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3980","reported_seq":6195,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.262562+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:44.262562+0000","last_peered":"2024-09-16T04:42:44.262562+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:44.262562+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3980,"log_dups_size":0,"ondisk_log_size":3980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32560396,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3978,"num_write_kb":31812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4044","reported_seq":4096,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.540867+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:43.540867+0000","last_peered":"2024-09-16T04:42:43.540867+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:43.540867+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4044,"log_dups_size":0,"ondisk_log_size":4044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33057304,"num_objects":4036,"num_object_clones":0,"num_object_copies":16144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4041,"num_write_kb":32305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3972","reported_seq":4046,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.909086+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:43.909086+0000","last_peered":"2024-09-16T04:42:43.909086+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:43.909086+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32519436,"num_objects":3970,"num_object_clones":0,"num_object_copies":15880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3970,"num_write_kb":31758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3948","reported_seq":4001,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.311735+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:43.311735+0000","last_peered":"2024-09-16T04:42:43.311735+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:43.311735+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3948,"log_dups_size":0,"ondisk_log_size":3948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32296850,"num_objects":3943,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3945,"num_write_kb":31548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3853","reported_seq":3888,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.563843+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:43.563843+0000","last_peered":"2024-09-16T04:42:43.563843+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:43.563843+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3853,"log_dups_size":0,"ondisk_log_size":3853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31554182,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3852,"num_write_kb":30815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1724}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3926","reported_seq":3996,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.848567+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:43.848567+0000","last_peered":"2024-09-16T04:42:43.848567+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:43.848567+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3926,"log_dups_size":0,"ondisk_log_size":3926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32152198,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1700}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4014","reported_seq":4063,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.515049+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:43.515049+0000","last_peered":"2024-09-16T04:42:43.515049+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:43.515049+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4014,"log_dups_size":0,"ondisk_log_size":4014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32847116,"num_objects":4010,"num_object_clones":0,"num_object_copies":16040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4012,"num_write_kb":32092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1776}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3845","reported_seq":3920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.877450+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:43.877450+0000","last_peered":"2024-09-16T04:42:43.877450+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:43.877450+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31470860,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3843,"num_write_kb":30741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1710}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3934","reported_seq":3995,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.148835+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:44.148835+0000","last_peered":"2024-09-16T04:42:44.148835+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:44.148835+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3934,"log_dups_size":0,"ondisk_log_size":3934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32183564,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3932,"num_write_kb":31444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1690}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":517973672,"num_objects":63234,"num_object_clones":0,"num_object_copies":252936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63261,"num_write_kb":505970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1105,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":890679296,"data_stored":890679296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63288,"ondisk_log_size":63288,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460412,"kb_used_data":252396,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93911428,"statfs":{"total":96636764160,"available":96165302272,"internally_reserved":0,"allocated":258453504,"data_stored":258282694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":212987599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3,9,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472260,"kb_used_data":252276,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93899580,"statfs":{"total":96636764160,"available":96153169920,"internally_reserved":0,"allocated":258330624,"data_stored":258159814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051600,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232112,"kb_used_data":110176,"kb_used_omap":4,"kb_used_meta":121915,"kb_avail":94139728,"statfs":{"total":96636764160,"available":96399081472,"internally_reserved":0,"allocated":112820224,"data_stored":112649414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":124841679},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465744,"kb_used_data":253632,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93906096,"statfs":{"total":96636764160,"available":96159842304,"internally_reserved":0,"allocated":259719168,"data_stored":259548358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":217181902},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,6,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":259526656,"data_stored":259526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113577984,"data_stored":113577984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":259055616,"data_stored":259055616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258519040,"data_stored":258519040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:44.807 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:45.275 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 111 20 63673 63653 4.47876 2.46094 0.054769 0.034876 2024-09-16T04:42:45.396 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:45.397 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:45.427 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-09-16T04:42:44.590314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":517973672,"num_objects":63234,"num_object_clones":0,"num_object_copies":252936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63261,"num_write_kb":505970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1105,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63288,"ondisk_log_size":63288,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1630528,"kb_used_data":868480,"kb_used_omap":14,"kb_used_meta":761969,"kb_avail":375856832,"statfs":{"total":386547056640,"available":384877395968,"internally_reserved":0,"allocated":889323520,"data_stored":888640280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":780256817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,5,14,11,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41410560,"num_objects":5055,"num_object_clones":0,"num_object_copies":20220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5055,"num_write_kb":40440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003071"},"pg_stats":[{"pgid":"1.f","version":"31'3925","reported_seq":3964,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.573612+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:43.573612+0000","last_peered":"2024-09-16T04:42:43.573612+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:43.573612+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32153600,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3982","reported_seq":4043,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.202258+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:44.202258+0000","last_peered":"2024-09-16T04:42:44.202258+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:44.202258+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3982,"log_dups_size":0,"ondisk_log_size":3982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32584972,"num_objects":3978,"num_object_clones":0,"num_object_copies":15912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3980,"num_write_kb":31836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1767}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3960","reported_seq":4031,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.841380+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:43.841380+0000","last_peered":"2024-09-16T04:42:43.841380+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:43.841380+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3960,"log_dups_size":0,"ondisk_log_size":3960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32414342,"num_objects":3957,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3959,"num_write_kb":31662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1730}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3999","reported_seq":4060,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.266775+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:44.266775+0000","last_peered":"2024-09-16T04:42:44.266775+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:44.266775+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3999,"log_dups_size":0,"ondisk_log_size":3999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32724236,"num_objects":3995,"num_object_clones":0,"num_object_copies":15980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3997,"num_write_kb":31965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1766}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3983","reported_seq":4034,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.474905+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:43.474905+0000","last_peered":"2024-09-16T04:42:43.474905+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:43.474905+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32601356,"num_objects":3980,"num_object_clones":0,"num_object_copies":15920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3981,"num_write_kb":31845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3925","reported_seq":3964,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.518478+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:43.518478+0000","last_peered":"2024-09-16T04:42:43.518478+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:43.518478+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32153600,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3998","reported_seq":4059,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.260361+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:44.260361+0000","last_peered":"2024-09-16T04:42:44.260361+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:44.260361+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699660,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3996,"num_write_kb":31948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1766}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3980","reported_seq":6195,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.262562+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:44.262562+0000","last_peered":"2024-09-16T04:42:44.262562+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:44.262562+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3980,"log_dups_size":0,"ondisk_log_size":3980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32560396,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3978,"num_write_kb":31812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4044","reported_seq":4096,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.540867+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:43.540867+0000","last_peered":"2024-09-16T04:42:43.540867+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:43.540867+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4044,"log_dups_size":0,"ondisk_log_size":4044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33057304,"num_objects":4036,"num_object_clones":0,"num_object_copies":16144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4041,"num_write_kb":32305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3972","reported_seq":4046,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.909086+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:43.909086+0000","last_peered":"2024-09-16T04:42:43.909086+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:43.909086+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32519436,"num_objects":3970,"num_object_clones":0,"num_object_copies":15880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3970,"num_write_kb":31758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3948","reported_seq":4001,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.311735+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:43.311735+0000","last_peered":"2024-09-16T04:42:43.311735+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:43.311735+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3948,"log_dups_size":0,"ondisk_log_size":3948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32296850,"num_objects":3943,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3945,"num_write_kb":31548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3853","reported_seq":3888,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.563843+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:43.563843+0000","last_peered":"2024-09-16T04:42:43.563843+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:43.563843+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3853,"log_dups_size":0,"ondisk_log_size":3853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31554182,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3852,"num_write_kb":30815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1724}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3926","reported_seq":3996,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.848567+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:43.848567+0000","last_peered":"2024-09-16T04:42:43.848567+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:43.848567+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3926,"log_dups_size":0,"ondisk_log_size":3926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32152198,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1700}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4014","reported_seq":4063,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.515049+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:43.515049+0000","last_peered":"2024-09-16T04:42:43.515049+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:43.515049+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4014,"log_dups_size":0,"ondisk_log_size":4014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32847116,"num_objects":4010,"num_object_clones":0,"num_object_copies":16040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4012,"num_write_kb":32092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1776}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3845","reported_seq":3920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.877450+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:43.877450+0000","last_peered":"2024-09-16T04:42:43.877450+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:43.877450+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31470860,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3843,"num_write_kb":30741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1710}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3934","reported_seq":3995,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.148835+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:44.148835+0000","last_peered":"2024-09-16T04:42:44.148835+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:44.148835+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3934,"log_dups_size":0,"ondisk_log_size":3934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32183564,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3932,"num_write_kb":31444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1690}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":517973672,"num_objects":63234,"num_object_clones":0,"num_object_copies":252936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63261,"num_write_kb":505970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1105,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":890679296,"data_stored":890679296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63288,"ondisk_log_size":63288,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460412,"kb_used_data":252396,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93911428,"statfs":{"total":96636764160,"available":96165302272,"internally_reserved":0,"allocated":258453504,"data_stored":258282694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":212987599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3,9,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472260,"kb_used_data":252276,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93899580,"statfs":{"total":96636764160,"available":96153169920,"internally_reserved":0,"allocated":258330624,"data_stored":258159814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051600,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232112,"kb_used_data":110176,"kb_used_omap":4,"kb_used_meta":121915,"kb_avail":94139728,"statfs":{"total":96636764160,"available":96399081472,"internally_reserved":0,"allocated":112820224,"data_stored":112649414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":124841679},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465744,"kb_used_data":253632,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93906096,"statfs":{"total":96636764160,"available":96159842304,"internally_reserved":0,"allocated":259719168,"data_stored":259548358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":217181902},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,6,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":259526656,"data_stored":259526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113577984,"data_stored":113577984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":259055616,"data_stored":259055616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258519040,"data_stored":258519040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:45.428 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:42:45.428 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3925", 'reported_seq': 3964, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:43.573612+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:42:43.573612+0000', 'last_peered': '2024-09-16T04:42:43.573612+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:42:43.573612+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3925, 'log_dups_size': 0, 'ondisk_log_size': 3925, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32153600, 'num_objects': 3925, 'num_object_clones': 0, 'num_object_copies': 15700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3925, 'num_write_kb': 31400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1706}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:45.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:42:45.429 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3982", 'reported_seq': 4043, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:44.202258+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:42:44.202258+0000', 'last_peered': '2024-09-16T04:42:44.202258+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:42:44.202258+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3982, 'log_dups_size': 0, 'ondisk_log_size': 3982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32584972, 'num_objects': 3978, 'num_object_clones': 0, 'num_object_copies': 15912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3980, 'num_write_kb': 31836, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1767}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:45.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:42:45.429 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3960", 'reported_seq': 4031, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:43.841380+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:42:43.841380+0000', 'last_peered': '2024-09-16T04:42:43.841380+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:42:43.841380+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3960, 'log_dups_size': 0, 'ondisk_log_size': 3960, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32414342, 'num_objects': 3957, 'num_object_clones': 0, 'num_object_copies': 15828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3959, 'num_write_kb': 31662, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1730}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:45.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:42:45.429 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3999", 'reported_seq': 4060, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:44.266775+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:42:44.266775+0000', 'last_peered': '2024-09-16T04:42:44.266775+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:42:44.266775+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3999, 'log_dups_size': 0, 'ondisk_log_size': 3999, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32724236, 'num_objects': 3995, 'num_object_clones': 0, 'num_object_copies': 15980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3997, 'num_write_kb': 31965, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1766}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:45.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:42:45.429 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3983", 'reported_seq': 4034, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:43.474905+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:42:43.474905+0000', 'last_peered': '2024-09-16T04:42:43.474905+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:42:43.474905+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3983, 'log_dups_size': 0, 'ondisk_log_size': 3983, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32601356, 'num_objects': 3980, 'num_object_clones': 0, 'num_object_copies': 15920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3981, 'num_write_kb': 31845, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1697}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:45.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:42:45.429 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3925", 'reported_seq': 3964, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:43.518478+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:42:43.518478+0000', 'last_peered': '2024-09-16T04:42:43.518478+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:42:43.518478+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3925, 'log_dups_size': 0, 'ondisk_log_size': 3925, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32153600, 'num_objects': 3925, 'num_object_clones': 0, 'num_object_copies': 15700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3925, 'num_write_kb': 31400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1717}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:45.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:42:45.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3998", 'reported_seq': 4059, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:44.260361+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:42:44.260361+0000', 'last_peered': '2024-09-16T04:42:44.260361+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:42:44.260361+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32699660, 'num_objects': 3992, 'num_object_clones': 0, 'num_object_copies': 15968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3996, 'num_write_kb': 31948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1766}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:45.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:42:45.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3980", 'reported_seq': 6195, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:44.262562+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:42:44.262562+0000', 'last_peered': '2024-09-16T04:42:44.262562+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:42:44.262562+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3980, 'log_dups_size': 0, 'ondisk_log_size': 3980, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32560396, 'num_objects': 3975, 'num_object_clones': 0, 'num_object_copies': 15900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3978, 'num_write_kb': 31812, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1080, 'num_bytes_recovered': 8830976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2815}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:45.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:42:45.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4044", 'reported_seq': 4096, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:43.540867+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:42:43.540867+0000', 'last_peered': '2024-09-16T04:42:43.540867+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:42:43.540867+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4044, 'log_dups_size': 0, 'ondisk_log_size': 4044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33057304, 'num_objects': 4036, 'num_object_clones': 0, 'num_object_copies': 16144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4036, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4041, 'num_write_kb': 32305, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1696}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:45.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:42:45.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3972", 'reported_seq': 4046, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:43.909086+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:42:43.909086+0000', 'last_peered': '2024-09-16T04:42:43.909086+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:42:43.909086+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3972, 'log_dups_size': 0, 'ondisk_log_size': 3972, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32519436, 'num_objects': 3970, 'num_object_clones': 0, 'num_object_copies': 15880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3970, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3970, 'num_write_kb': 31758, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1770}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:45.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:42:45.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3948", 'reported_seq': 4001, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:43.311735+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:42:43.311735+0000', 'last_peered': '2024-09-16T04:42:43.311735+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:42:43.311735+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3948, 'log_dups_size': 0, 'ondisk_log_size': 3948, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32296850, 'num_objects': 3943, 'num_object_clones': 0, 'num_object_copies': 15772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3945, 'num_write_kb': 31548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1754}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:45.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:42:45.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3853", 'reported_seq': 3888, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:43.563843+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:42:43.563843+0000', 'last_peered': '2024-09-16T04:42:43.563843+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:42:43.563843+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3853, 'log_dups_size': 0, 'ondisk_log_size': 3853, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31554182, 'num_objects': 3852, 'num_object_clones': 0, 'num_object_copies': 15408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3852, 'num_write_kb': 30815, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1724}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:45.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:42:45.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3926", 'reported_seq': 3996, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:43.848567+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:42:43.848567+0000', 'last_peered': '2024-09-16T04:42:43.848567+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:42:43.848567+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3926, 'log_dups_size': 0, 'ondisk_log_size': 3926, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32152198, 'num_objects': 3925, 'num_object_clones': 0, 'num_object_copies': 15700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3925, 'num_write_kb': 31399, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1700}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:45.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:42:45.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4014", 'reported_seq': 4063, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:43.515049+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:42:43.515049+0000', 'last_peered': '2024-09-16T04:42:43.515049+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:42:43.515049+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4014, 'log_dups_size': 0, 'ondisk_log_size': 4014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32847116, 'num_objects': 4010, 'num_object_clones': 0, 'num_object_copies': 16040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4012, 'num_write_kb': 32092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1776}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:45.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:42:45.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3845", 'reported_seq': 3920, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:43.877450+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:42:43.877450+0000', 'last_peered': '2024-09-16T04:42:43.877450+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:42:43.877450+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3845, 'log_dups_size': 0, 'ondisk_log_size': 3845, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31470860, 'num_objects': 3842, 'num_object_clones': 0, 'num_object_copies': 15368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3843, 'num_write_kb': 30741, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1710}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:45.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:42:45.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3934", 'reported_seq': 3995, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:44.148835+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:42:44.148835+0000', 'last_peered': '2024-09-16T04:42:44.148835+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:42:44.148835+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3934, 'log_dups_size': 0, 'ondisk_log_size': 3934, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32183564, 'num_objects': 3929, 'num_object_clones': 0, 'num_object_copies': 15716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3932, 'num_write_kb': 31444, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1690}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:45.433 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:42:46.175 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:46.175 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34475/252936 objects degraded (13.630%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":178,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":63234,"data_bytes":517973672,"bytes_used":1669660672,"bytes_avail":384877395968,"bytes_total":386547056640,"degraded_objects":34475,"degraded_total":252936,"degraded_ratio":0.13629930100895088,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":143323,"recovering_keys_per_sec":0,"num_objects_recovered":210,"num_bytes_recovered":1720320,"num_keys_recovered":0,"write_bytes_sec":3449997,"read_op_per_sec":0,"write_op_per_sec":421},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:42:46.203 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:42:46.204 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:46.275 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 112 20 63981 63961 4.46024 2.40625 0.0590536 0.0350173 2024-09-16T04:42:46.759 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:46.760 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:46.781 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-09-16T04:42:46.591637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":517973672,"num_objects":63234,"num_object_clones":0,"num_object_copies":252936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63261,"num_write_kb":505970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1105,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63288,"ondisk_log_size":63288,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1641548,"kb_used_data":875340,"kb_used_omap":14,"kb_used_meta":766129,"kb_avail":375845812,"statfs":{"total":386547056640,"available":384866111488,"internally_reserved":0,"allocated":896348160,"data_stored":895664920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":784516657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,7,10,14,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27238400,"num_objects":3325,"num_object_clones":0,"num_object_copies":13300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3325,"num_write_kb":26600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003922"},"pg_stats":[{"pgid":"1.f","version":"31'3925","reported_seq":3964,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.573612+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:43.573612+0000","last_peered":"2024-09-16T04:42:43.573612+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:43.573612+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32153600,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3982","reported_seq":4043,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.202258+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:44.202258+0000","last_peered":"2024-09-16T04:42:44.202258+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:44.202258+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3982,"log_dups_size":0,"ondisk_log_size":3982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32584972,"num_objects":3978,"num_object_clones":0,"num_object_copies":15912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3980,"num_write_kb":31836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1767}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3960","reported_seq":4031,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.841380+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:43.841380+0000","last_peered":"2024-09-16T04:42:43.841380+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:43.841380+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3960,"log_dups_size":0,"ondisk_log_size":3960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32414342,"num_objects":3957,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3959,"num_write_kb":31662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1730}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3999","reported_seq":4060,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.266775+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:44.266775+0000","last_peered":"2024-09-16T04:42:44.266775+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:44.266775+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3999,"log_dups_size":0,"ondisk_log_size":3999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32724236,"num_objects":3995,"num_object_clones":0,"num_object_copies":15980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3997,"num_write_kb":31965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1766}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3983","reported_seq":4034,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.474905+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:43.474905+0000","last_peered":"2024-09-16T04:42:43.474905+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:43.474905+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32601356,"num_objects":3980,"num_object_clones":0,"num_object_copies":15920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3981,"num_write_kb":31845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3925","reported_seq":3964,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.518478+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:43.518478+0000","last_peered":"2024-09-16T04:42:43.518478+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:43.518478+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32153600,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3998","reported_seq":4059,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.260361+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:44.260361+0000","last_peered":"2024-09-16T04:42:44.260361+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:44.260361+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699660,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3996,"num_write_kb":31948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1766}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3980","reported_seq":6195,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.262562+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:44.262562+0000","last_peered":"2024-09-16T04:42:44.262562+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:44.262562+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3980,"log_dups_size":0,"ondisk_log_size":3980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32560396,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3978,"num_write_kb":31812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4044","reported_seq":4096,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.540867+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:43.540867+0000","last_peered":"2024-09-16T04:42:43.540867+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:43.540867+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4044,"log_dups_size":0,"ondisk_log_size":4044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33057304,"num_objects":4036,"num_object_clones":0,"num_object_copies":16144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4041,"num_write_kb":32305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3972","reported_seq":4046,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.909086+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:43.909086+0000","last_peered":"2024-09-16T04:42:43.909086+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:43.909086+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32519436,"num_objects":3970,"num_object_clones":0,"num_object_copies":15880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3970,"num_write_kb":31758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3948","reported_seq":4001,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.311735+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:43.311735+0000","last_peered":"2024-09-16T04:42:43.311735+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:43.311735+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3948,"log_dups_size":0,"ondisk_log_size":3948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32296850,"num_objects":3943,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3945,"num_write_kb":31548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3853","reported_seq":3888,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.563843+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:43.563843+0000","last_peered":"2024-09-16T04:42:43.563843+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:43.563843+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3853,"log_dups_size":0,"ondisk_log_size":3853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31554182,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3852,"num_write_kb":30815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1724}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3926","reported_seq":3996,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.848567+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:43.848567+0000","last_peered":"2024-09-16T04:42:43.848567+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:43.848567+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3926,"log_dups_size":0,"ondisk_log_size":3926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32152198,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1700}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4014","reported_seq":4063,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.515049+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:43.515049+0000","last_peered":"2024-09-16T04:42:43.515049+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:43.515049+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4014,"log_dups_size":0,"ondisk_log_size":4014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32847116,"num_objects":4010,"num_object_clones":0,"num_object_copies":16040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4012,"num_write_kb":32092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1776}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3845","reported_seq":3920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.877450+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:43.877450+0000","last_peered":"2024-09-16T04:42:43.877450+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:43.877450+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31470860,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3843,"num_write_kb":30741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1710}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3934","reported_seq":3995,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.148835+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:44.148835+0000","last_peered":"2024-09-16T04:42:44.148835+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:44.148835+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3934,"log_dups_size":0,"ondisk_log_size":3934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32183564,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3932,"num_write_kb":31444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1690}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":517973672,"num_objects":63234,"num_object_clones":0,"num_object_copies":252936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63261,"num_write_kb":505970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1105,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":897843200,"data_stored":897843200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63288,"ondisk_log_size":63288,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460412,"kb_used_data":252396,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93911428,"statfs":{"total":96636764160,"available":96165302272,"internally_reserved":0,"allocated":258453504,"data_stored":258282694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":212987599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3,9,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472260,"kb_used_data":252276,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93899580,"statfs":{"total":96636764160,"available":96153169920,"internally_reserved":0,"allocated":258330624,"data_stored":258159814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243132,"kb_used_data":117036,"kb_used_omap":4,"kb_used_meta":126075,"kb_avail":94128708,"statfs":{"total":96636764160,"available":96387796992,"internally_reserved":0,"allocated":119844864,"data_stored":119674054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":129101519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465744,"kb_used_data":253632,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93906096,"statfs":{"total":96636764160,"available":96159842304,"internally_reserved":0,"allocated":259719168,"data_stored":259548358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":217181902},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,6,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":259526656,"data_stored":259526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120741888,"data_stored":120741888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":259055616,"data_stored":259055616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258519040,"data_stored":258519040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:47.275 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 113 20 64305 64285 4.44318 2.53125 0.0753821 0.0351539 2024-09-16T04:42:48.275 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 114 20 64657 64637 4.42832 2.75 0.0624944 0.0352702 2024-09-16T04:42:49.276 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 115 20 65004 64984 4.41339 2.71094 0.0490638 0.0353908 2024-09-16T04:42:49.783 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:50.276 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 116 20 65327 65307 4.3971 2.52344 0.0757776 0.0355185 2024-09-16T04:42:50.336 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:50.336 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:50.367 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-16T04:42:48.592125+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":517973672,"num_objects":63234,"num_object_clones":0,"num_object_copies":252936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63261,"num_write_kb":505970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1105,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63288,"ondisk_log_size":63288,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1641548,"kb_used_data":875340,"kb_used_omap":14,"kb_used_meta":766129,"kb_avail":375845812,"statfs":{"total":386547056640,"available":384866111488,"internally_reserved":0,"allocated":896348160,"data_stored":895664920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":784516657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,7,10,14,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27238400,"num_objects":3325,"num_object_clones":0,"num_object_copies":13300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3325,"num_write_kb":26600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003836"},"pg_stats":[{"pgid":"1.f","version":"31'3925","reported_seq":3964,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.573612+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:43.573612+0000","last_peered":"2024-09-16T04:42:43.573612+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:43.573612+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32153600,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3982","reported_seq":4043,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.202258+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:44.202258+0000","last_peered":"2024-09-16T04:42:44.202258+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:44.202258+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3982,"log_dups_size":0,"ondisk_log_size":3982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32584972,"num_objects":3978,"num_object_clones":0,"num_object_copies":15912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3980,"num_write_kb":31836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1767}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3960","reported_seq":4031,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.841380+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:43.841380+0000","last_peered":"2024-09-16T04:42:43.841380+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:43.841380+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3960,"log_dups_size":0,"ondisk_log_size":3960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32414342,"num_objects":3957,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3959,"num_write_kb":31662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1730}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3999","reported_seq":4060,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.266775+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:44.266775+0000","last_peered":"2024-09-16T04:42:44.266775+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:44.266775+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3999,"log_dups_size":0,"ondisk_log_size":3999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32724236,"num_objects":3995,"num_object_clones":0,"num_object_copies":15980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3997,"num_write_kb":31965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1766}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3983","reported_seq":4034,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.474905+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:43.474905+0000","last_peered":"2024-09-16T04:42:43.474905+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:43.474905+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32601356,"num_objects":3980,"num_object_clones":0,"num_object_copies":15920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3981,"num_write_kb":31845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3925","reported_seq":3964,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.518478+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:43.518478+0000","last_peered":"2024-09-16T04:42:43.518478+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:43.518478+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32153600,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3998","reported_seq":4059,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.260361+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:44.260361+0000","last_peered":"2024-09-16T04:42:44.260361+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:44.260361+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699660,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3996,"num_write_kb":31948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1766}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3980","reported_seq":6195,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.262562+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:44.262562+0000","last_peered":"2024-09-16T04:42:44.262562+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:44.262562+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3980,"log_dups_size":0,"ondisk_log_size":3980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32560396,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3978,"num_write_kb":31812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4044","reported_seq":4096,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.540867+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:43.540867+0000","last_peered":"2024-09-16T04:42:43.540867+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:43.540867+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4044,"log_dups_size":0,"ondisk_log_size":4044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33057304,"num_objects":4036,"num_object_clones":0,"num_object_copies":16144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4041,"num_write_kb":32305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3972","reported_seq":4046,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.909086+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:43.909086+0000","last_peered":"2024-09-16T04:42:43.909086+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:43.909086+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32519436,"num_objects":3970,"num_object_clones":0,"num_object_copies":15880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3970,"num_write_kb":31758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3948","reported_seq":4001,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.311735+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:43.311735+0000","last_peered":"2024-09-16T04:42:43.311735+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:43.311735+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3948,"log_dups_size":0,"ondisk_log_size":3948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32296850,"num_objects":3943,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3945,"num_write_kb":31548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3853","reported_seq":3888,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.563843+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:43.563843+0000","last_peered":"2024-09-16T04:42:43.563843+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:43.563843+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3853,"log_dups_size":0,"ondisk_log_size":3853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31554182,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3852,"num_write_kb":30815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1724}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3926","reported_seq":3996,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.848567+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:43.848567+0000","last_peered":"2024-09-16T04:42:43.848567+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:43.848567+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3926,"log_dups_size":0,"ondisk_log_size":3926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32152198,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1700}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4014","reported_seq":4063,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.515049+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:43.515049+0000","last_peered":"2024-09-16T04:42:43.515049+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:43.515049+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4014,"log_dups_size":0,"ondisk_log_size":4014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32847116,"num_objects":4010,"num_object_clones":0,"num_object_copies":16040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4012,"num_write_kb":32092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1776}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3845","reported_seq":3920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:43.877450+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:43.877450+0000","last_peered":"2024-09-16T04:42:43.877450+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:43.877450+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31470860,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3843,"num_write_kb":30741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1710}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3934","reported_seq":3995,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:44.148835+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:44.148835+0000","last_peered":"2024-09-16T04:42:44.148835+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:44.148835+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3934,"log_dups_size":0,"ondisk_log_size":3934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32183564,"num_objects":3929,"num_object_clones":0,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3932,"num_write_kb":31444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1690}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":517973672,"num_objects":63234,"num_object_clones":0,"num_object_copies":252936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63261,"num_write_kb":505970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1105,"num_bytes_recovered":8830976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":897843200,"data_stored":897843200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63288,"ondisk_log_size":63288,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460412,"kb_used_data":252396,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93911428,"statfs":{"total":96636764160,"available":96165302272,"internally_reserved":0,"allocated":258453504,"data_stored":258282694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":212987599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3,9,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472260,"kb_used_data":252276,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93899580,"statfs":{"total":96636764160,"available":96153169920,"internally_reserved":0,"allocated":258330624,"data_stored":258159814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243132,"kb_used_data":117036,"kb_used_omap":4,"kb_used_meta":126075,"kb_avail":94128708,"statfs":{"total":96636764160,"available":96387796992,"internally_reserved":0,"allocated":119844864,"data_stored":119674054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":129101519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465744,"kb_used_data":253632,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93906096,"statfs":{"total":96636764160,"available":96159842304,"internally_reserved":0,"allocated":259719168,"data_stored":259548358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":217181902},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,6,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":259526656,"data_stored":259526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120741888,"data_stored":120741888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":259055616,"data_stored":259055616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258519040,"data_stored":258519040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:50.369 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:50.930 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:50.931 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:50.955 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-09-16T04:42:50.592609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":531531432,"num_objects":64889,"num_object_clones":0,"num_object_copies":259556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64916,"num_write_kb":519210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1161,"num_bytes_recovered":9289728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64943,"ondisk_log_size":64943,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1672768,"kb_used_data":894080,"kb_used_omap":14,"kb_used_meta":778609,"kb_avail":375814592,"statfs":{"total":386547056640,"available":384834142208,"internally_reserved":0,"allocated":915537920,"data_stored":914854680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":797296177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,8,6,13,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34758656,"num_objects":4243,"num_object_clones":0,"num_object_copies":16972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4243,"num_write_kb":33944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1638400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003832"},"pg_stats":[{"pgid":"1.f","version":"31'4038","reported_seq":4076,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.600947+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:48.600947+0000","last_peered":"2024-09-16T04:42:48.600947+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:48.600947+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4038,"log_dups_size":0,"ondisk_log_size":4038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33079296,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4038,"num_write_kb":32304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1819}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4078","reported_seq":4137,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:49.233482+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:49.233482+0000","last_peered":"2024-09-16T04:42:49.233482+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:49.233482+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33371404,"num_objects":4074,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4076,"num_write_kb":32604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1863}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4062","reported_seq":4133,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.886092+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:48.886092+0000","last_peered":"2024-09-16T04:42:48.886092+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:48.886092+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4062,"log_dups_size":0,"ondisk_log_size":4062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33249926,"num_objects":4059,"num_object_clones":0,"num_object_copies":16236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4061,"num_write_kb":32478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1832}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4079","reported_seq":4141,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:49.277581+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:49.277581+0000","last_peered":"2024-09-16T04:42:49.277581+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:49.277581+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4079,"log_dups_size":0,"ondisk_log_size":4079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33379596,"num_objects":4075,"num_object_clones":0,"num_object_copies":16300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4077,"num_write_kb":32605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1846}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4091","reported_seq":4143,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.599379+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:48.599379+0000","last_peered":"2024-09-16T04:42:48.599379+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:48.599379+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33486092,"num_objects":4088,"num_object_clones":0,"num_object_copies":16352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4089,"num_write_kb":32709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4018","reported_seq":4057,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.451219+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:48.451219+0000","last_peered":"2024-09-16T04:42:48.451219+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:48.451219+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32915456,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4018,"num_write_kb":32144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4104","reported_seq":4165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:49.151177+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:49.151177+0000","last_peered":"2024-09-16T04:42:49.151177+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:49.151177+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4104,"log_dups_size":0,"ondisk_log_size":4104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33568012,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4102,"num_write_kb":32796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1872}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4097","reported_seq":6426,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:49.249261+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:49.249261+0000","last_peered":"2024-09-16T04:42:49.249261+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:49.249261+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4097,"log_dups_size":0,"ondisk_log_size":4097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33518860,"num_objects":4092,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4095,"num_write_kb":32748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":9289728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2988}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4154","reported_seq":4203,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.588092+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:48.588092+0000","last_peered":"2024-09-16T04:42:48.588092+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:48.588092+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4154,"log_dups_size":0,"ondisk_log_size":4154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33958424,"num_objects":4146,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4151,"num_write_kb":33185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4066","reported_seq":4140,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.844621+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:48.844621+0000","last_peered":"2024-09-16T04:42:48.844621+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:48.844621+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4066,"log_dups_size":0,"ondisk_log_size":4066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33289484,"num_objects":4064,"num_object_clones":0,"num_object_copies":16256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":32510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1864}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4050","reported_seq":4103,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.431574+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:48.431574+0000","last_peered":"2024-09-16T04:42:48.431574+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:48.431574+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4050,"log_dups_size":0,"ondisk_log_size":4050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33132434,"num_objects":4045,"num_object_clones":0,"num_object_copies":16180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4047,"num_write_kb":32364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1856}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3973","reported_seq":4007,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.602134+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:48.602134+0000","last_peered":"2024-09-16T04:42:48.602134+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:48.602134+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3973,"log_dups_size":0,"ondisk_log_size":3973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32537222,"num_objects":3972,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3972,"num_write_kb":31775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1844}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4032","reported_seq":4102,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.861393+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:48.861393+0000","last_peered":"2024-09-16T04:42:48.861393+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:48.861393+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33020550,"num_objects":4031,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4031,"num_write_kb":32247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1806}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4109","reported_seq":4158,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.544124+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:48.544124+0000","last_peered":"2024-09-16T04:42:48.544124+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:48.544124+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33625356,"num_objects":4105,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1871}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3947","reported_seq":4022,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.903867+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:48.903867+0000","last_peered":"2024-09-16T04:42:48.903867+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:48.903867+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3947,"log_dups_size":0,"ondisk_log_size":3947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32306444,"num_objects":3944,"num_object_clones":0,"num_object_copies":15776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3945,"num_write_kb":31557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1812}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4045","reported_seq":4104,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:49.236711+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:49.236711+0000","last_peered":"2024-09-16T04:42:49.236711+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:49.236711+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4045,"log_dups_size":0,"ondisk_log_size":4045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33092876,"num_objects":4040,"num_object_clones":0,"num_object_copies":16160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4043,"num_write_kb":32332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1801}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":531531432,"num_objects":64889,"num_object_clones":0,"num_object_copies":259556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64916,"num_write_kb":519210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1161,"num_bytes_recovered":9289728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":918175744,"data_stored":918175744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64943,"ondisk_log_size":64943,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471172,"kb_used_data":258996,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93900668,"statfs":{"total":96636764160,"available":96154284032,"internally_reserved":0,"allocated":265211904,"data_stored":265041094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217247439},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483180,"kb_used_data":259036,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":93888660,"statfs":{"total":96636764160,"available":96141987840,"internally_reserved":0,"allocated":265252864,"data_stored":265082054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":229505477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243132,"kb_used_data":117036,"kb_used_omap":4,"kb_used_meta":126075,"kb_avail":94128708,"statfs":{"total":96636764160,"available":96387796992,"internally_reserved":0,"allocated":119844864,"data_stored":119674054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":129101519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475284,"kb_used_data":259012,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93896556,"statfs":{"total":96636764160,"available":96150073344,"internally_reserved":0,"allocated":265228288,"data_stored":265057478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":221441742},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,3,2,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266289152,"data_stored":266289152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120741888,"data_stored":120741888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":265801728,"data_stored":265801728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265342976,"data_stored":265342976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:50.956 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:42:50.956 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4038", 'reported_seq': 4076, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:48.600947+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:42:48.600947+0000', 'last_peered': '2024-09-16T04:42:48.600947+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:42:48.600947+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4038, 'log_dups_size': 0, 'ondisk_log_size': 4038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33079296, 'num_objects': 4038, 'num_object_clones': 0, 'num_object_copies': 16152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4038, 'num_write_kb': 32304, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1819}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:50.956 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:42:50.956 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4078", 'reported_seq': 4137, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:49.233482+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:42:49.233482+0000', 'last_peered': '2024-09-16T04:42:49.233482+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:42:49.233482+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4078, 'log_dups_size': 0, 'ondisk_log_size': 4078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33371404, 'num_objects': 4074, 'num_object_clones': 0, 'num_object_copies': 16296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4074, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4076, 'num_write_kb': 32604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1863}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:50.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:42:50.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4062", 'reported_seq': 4133, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:48.886092+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:42:48.886092+0000', 'last_peered': '2024-09-16T04:42:48.886092+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:42:48.886092+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4062, 'log_dups_size': 0, 'ondisk_log_size': 4062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33249926, 'num_objects': 4059, 'num_object_clones': 0, 'num_object_copies': 16236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4061, 'num_write_kb': 32478, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1832}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:50.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:42:50.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4079", 'reported_seq': 4141, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:49.277581+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:42:49.277581+0000', 'last_peered': '2024-09-16T04:42:49.277581+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:42:49.277581+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4079, 'log_dups_size': 0, 'ondisk_log_size': 4079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33379596, 'num_objects': 4075, 'num_object_clones': 0, 'num_object_copies': 16300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4077, 'num_write_kb': 32605, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1846}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:50.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:42:50.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4091", 'reported_seq': 4143, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:48.599379+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:42:48.599379+0000', 'last_peered': '2024-09-16T04:42:48.599379+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:42:48.599379+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4091, 'log_dups_size': 0, 'ondisk_log_size': 4091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33486092, 'num_objects': 4088, 'num_object_clones': 0, 'num_object_copies': 16352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4088, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4089, 'num_write_kb': 32709, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1805}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:50.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:42:50.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4018", 'reported_seq': 4057, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:48.451219+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:42:48.451219+0000', 'last_peered': '2024-09-16T04:42:48.451219+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:42:48.451219+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4018, 'log_dups_size': 0, 'ondisk_log_size': 4018, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32915456, 'num_objects': 4018, 'num_object_clones': 0, 'num_object_copies': 16072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4018, 'num_write_kb': 32144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1810}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:50.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:42:50.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4104", 'reported_seq': 4165, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:49.151177+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:42:49.151177+0000', 'last_peered': '2024-09-16T04:42:49.151177+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:42:49.151177+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4104, 'log_dups_size': 0, 'ondisk_log_size': 4104, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33568012, 'num_objects': 4098, 'num_object_clones': 0, 'num_object_copies': 16392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4098, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4102, 'num_write_kb': 32796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1872}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:50.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:42:50.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4097", 'reported_seq': 6426, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:49.249261+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:42:49.249261+0000', 'last_peered': '2024-09-16T04:42:49.249261+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:42:49.249261+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4097, 'log_dups_size': 0, 'ondisk_log_size': 4097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33518860, 'num_objects': 4092, 'num_object_clones': 0, 'num_object_copies': 16368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4092, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4095, 'num_write_kb': 32748, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1136, 'num_bytes_recovered': 9289728, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2988}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:50.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:42:50.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4154", 'reported_seq': 4203, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:48.588092+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:42:48.588092+0000', 'last_peered': '2024-09-16T04:42:48.588092+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:42:48.588092+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4154, 'log_dups_size': 0, 'ondisk_log_size': 4154, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33958424, 'num_objects': 4146, 'num_object_clones': 0, 'num_object_copies': 16584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4151, 'num_write_kb': 33185, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1806}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:50.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:42:50.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4066", 'reported_seq': 4140, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:48.844621+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:42:48.844621+0000', 'last_peered': '2024-09-16T04:42:48.844621+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:42:48.844621+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4066, 'log_dups_size': 0, 'ondisk_log_size': 4066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33289484, 'num_objects': 4064, 'num_object_clones': 0, 'num_object_copies': 16256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4064, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4064, 'num_write_kb': 32510, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1864}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:50.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:42:50.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4050", 'reported_seq': 4103, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:48.431574+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:42:48.431574+0000', 'last_peered': '2024-09-16T04:42:48.431574+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:42:48.431574+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4050, 'log_dups_size': 0, 'ondisk_log_size': 4050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33132434, 'num_objects': 4045, 'num_object_clones': 0, 'num_object_copies': 16180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4045, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4047, 'num_write_kb': 32364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1856}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:50.959 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:42:50.959 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3973", 'reported_seq': 4007, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:48.602134+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:42:48.602134+0000', 'last_peered': '2024-09-16T04:42:48.602134+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:42:48.602134+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3973, 'log_dups_size': 0, 'ondisk_log_size': 3973, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32537222, 'num_objects': 3972, 'num_object_clones': 0, 'num_object_copies': 15888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3972, 'num_write_kb': 31775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1844}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:50.959 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:42:50.959 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4032", 'reported_seq': 4102, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:48.861393+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:42:48.861393+0000', 'last_peered': '2024-09-16T04:42:48.861393+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:42:48.861393+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4032, 'log_dups_size': 0, 'ondisk_log_size': 4032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33020550, 'num_objects': 4031, 'num_object_clones': 0, 'num_object_copies': 16124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4031, 'num_write_kb': 32247, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1806}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:50.959 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:42:50.959 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4109", 'reported_seq': 4158, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:48.544124+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:42:48.544124+0000', 'last_peered': '2024-09-16T04:42:48.544124+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:42:48.544124+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4109, 'log_dups_size': 0, 'ondisk_log_size': 4109, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33625356, 'num_objects': 4105, 'num_object_clones': 0, 'num_object_copies': 16420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4105, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4107, 'num_write_kb': 32852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1871}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:50.959 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:42:50.959 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3947", 'reported_seq': 4022, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:48.903867+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:42:48.903867+0000', 'last_peered': '2024-09-16T04:42:48.903867+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:42:48.903867+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 3947, 'log_dups_size': 0, 'ondisk_log_size': 3947, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32306444, 'num_objects': 3944, 'num_object_clones': 0, 'num_object_copies': 15776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3944, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3945, 'num_write_kb': 31557, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1812}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:50.959 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:42:50.959 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4045", 'reported_seq': 4104, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:49.236711+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:42:49.236711+0000', 'last_peered': '2024-09-16T04:42:49.236711+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:42:49.236711+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4045, 'log_dups_size': 0, 'ondisk_log_size': 4045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33092876, 'num_objects': 4040, 'num_object_clones': 0, 'num_object_copies': 16160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4040, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4043, 'num_write_kb': 32332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1801}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:50.960 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:42:51.278 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 117 20 65638 65618 4.38029 2.42969 0.0991681 0.0356597 2024-09-16T04:42:51.697 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:51.697 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34419/259556 objects degraded (13.261%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":184,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":64889,"data_bytes":531531432,"bytes_used":1712914432,"bytes_avail":384834142208,"bytes_total":386547056640,"degraded_objects":34419,"degraded_total":259556,"degraded_ratio":0.13260722156297677,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":136489,"recovering_keys_per_sec":0,"num_objects_recovered":200,"num_bytes_recovered":1638400,"num_keys_recovered":0,"write_bytes_sec":2895629,"read_op_per_sec":0,"write_op_per_sec":353},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:42:51.721 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:42:51.721 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:52.276 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 118 20 65937 65917 4.36296 2.33594 0.0532246 0.035791 2024-09-16T04:42:52.317 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:52.317 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:52.340 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-09-16T04:42:50.592609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":531531432,"num_objects":64889,"num_object_clones":0,"num_object_copies":259556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64916,"num_write_kb":519210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1161,"num_bytes_recovered":9289728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64943,"ondisk_log_size":64943,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1672768,"kb_used_data":894080,"kb_used_omap":14,"kb_used_meta":778609,"kb_avail":375814592,"statfs":{"total":386547056640,"available":384834142208,"internally_reserved":0,"allocated":915537920,"data_stored":914854680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":797296177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,8,6,13,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34758656,"num_objects":4243,"num_object_clones":0,"num_object_copies":16972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4243,"num_write_kb":33944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1638400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003832"},"pg_stats":[{"pgid":"1.f","version":"31'4038","reported_seq":4076,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.600947+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:48.600947+0000","last_peered":"2024-09-16T04:42:48.600947+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:48.600947+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4038,"log_dups_size":0,"ondisk_log_size":4038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33079296,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4038,"num_write_kb":32304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1819}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4078","reported_seq":4137,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:49.233482+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:49.233482+0000","last_peered":"2024-09-16T04:42:49.233482+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:49.233482+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33371404,"num_objects":4074,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4076,"num_write_kb":32604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1863}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4062","reported_seq":4133,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.886092+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:48.886092+0000","last_peered":"2024-09-16T04:42:48.886092+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:48.886092+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4062,"log_dups_size":0,"ondisk_log_size":4062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33249926,"num_objects":4059,"num_object_clones":0,"num_object_copies":16236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4061,"num_write_kb":32478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1832}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4079","reported_seq":4141,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:49.277581+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:49.277581+0000","last_peered":"2024-09-16T04:42:49.277581+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:49.277581+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4079,"log_dups_size":0,"ondisk_log_size":4079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33379596,"num_objects":4075,"num_object_clones":0,"num_object_copies":16300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4077,"num_write_kb":32605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1846}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4091","reported_seq":4143,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.599379+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:48.599379+0000","last_peered":"2024-09-16T04:42:48.599379+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:48.599379+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33486092,"num_objects":4088,"num_object_clones":0,"num_object_copies":16352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4089,"num_write_kb":32709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4018","reported_seq":4057,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.451219+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:48.451219+0000","last_peered":"2024-09-16T04:42:48.451219+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:48.451219+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32915456,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4018,"num_write_kb":32144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4104","reported_seq":4165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:49.151177+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:49.151177+0000","last_peered":"2024-09-16T04:42:49.151177+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:49.151177+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4104,"log_dups_size":0,"ondisk_log_size":4104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33568012,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4102,"num_write_kb":32796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1872}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4097","reported_seq":6426,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:49.249261+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:49.249261+0000","last_peered":"2024-09-16T04:42:49.249261+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:49.249261+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4097,"log_dups_size":0,"ondisk_log_size":4097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33518860,"num_objects":4092,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4095,"num_write_kb":32748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":9289728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2988}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4154","reported_seq":4203,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.588092+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:48.588092+0000","last_peered":"2024-09-16T04:42:48.588092+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:48.588092+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4154,"log_dups_size":0,"ondisk_log_size":4154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33958424,"num_objects":4146,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4151,"num_write_kb":33185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4066","reported_seq":4140,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.844621+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:48.844621+0000","last_peered":"2024-09-16T04:42:48.844621+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:48.844621+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4066,"log_dups_size":0,"ondisk_log_size":4066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33289484,"num_objects":4064,"num_object_clones":0,"num_object_copies":16256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":32510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1864}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4050","reported_seq":4103,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.431574+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:48.431574+0000","last_peered":"2024-09-16T04:42:48.431574+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:48.431574+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4050,"log_dups_size":0,"ondisk_log_size":4050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33132434,"num_objects":4045,"num_object_clones":0,"num_object_copies":16180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4047,"num_write_kb":32364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1856}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3973","reported_seq":4007,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.602134+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:48.602134+0000","last_peered":"2024-09-16T04:42:48.602134+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:48.602134+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3973,"log_dups_size":0,"ondisk_log_size":3973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32537222,"num_objects":3972,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3972,"num_write_kb":31775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1844}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4032","reported_seq":4102,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.861393+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:48.861393+0000","last_peered":"2024-09-16T04:42:48.861393+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:48.861393+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33020550,"num_objects":4031,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4031,"num_write_kb":32247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1806}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4109","reported_seq":4158,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.544124+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:48.544124+0000","last_peered":"2024-09-16T04:42:48.544124+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:48.544124+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33625356,"num_objects":4105,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1871}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3947","reported_seq":4022,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:48.903867+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:48.903867+0000","last_peered":"2024-09-16T04:42:48.903867+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:48.903867+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":3947,"log_dups_size":0,"ondisk_log_size":3947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32306444,"num_objects":3944,"num_object_clones":0,"num_object_copies":15776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3945,"num_write_kb":31557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1812}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4045","reported_seq":4104,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:49.236711+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:49.236711+0000","last_peered":"2024-09-16T04:42:49.236711+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:49.236711+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4045,"log_dups_size":0,"ondisk_log_size":4045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33092876,"num_objects":4040,"num_object_clones":0,"num_object_copies":16160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4043,"num_write_kb":32332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1801}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":531531432,"num_objects":64889,"num_object_clones":0,"num_object_copies":259556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64916,"num_write_kb":519210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1161,"num_bytes_recovered":9289728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":918175744,"data_stored":918175744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64943,"ondisk_log_size":64943,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471172,"kb_used_data":258996,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93900668,"statfs":{"total":96636764160,"available":96154284032,"internally_reserved":0,"allocated":265211904,"data_stored":265041094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":217247439},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483180,"kb_used_data":259036,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":93888660,"statfs":{"total":96636764160,"available":96141987840,"internally_reserved":0,"allocated":265252864,"data_stored":265082054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":229505477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243132,"kb_used_data":117036,"kb_used_omap":4,"kb_used_meta":126075,"kb_avail":94128708,"statfs":{"total":96636764160,"available":96387796992,"internally_reserved":0,"allocated":119844864,"data_stored":119674054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":129101519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475284,"kb_used_data":259012,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93896556,"statfs":{"total":96636764160,"available":96150073344,"internally_reserved":0,"allocated":265228288,"data_stored":265057478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":221441742},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,3,2,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266289152,"data_stored":266289152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120741888,"data_stored":120741888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":265801728,"data_stored":265801728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265342976,"data_stored":265342976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:53.276 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 119 20 66278 66258 4.34869 2.66406 0.0292033 0.0359178 2024-09-16T04:42:54.276 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:2024-09-16T04:42:54.275800+0000 min lat: 0.00269411 max lat: 1.73107 avg lat: 0.0360397 2024-09-16T04:42:54.276 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T04:42:54.276 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 120 20 66610 66590 4.33406 2.59375 0.0816587 0.0360397 2024-09-16T04:42:55.276 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 121 20 66947 66927 4.32 2.63281 0.0521664 0.0361522 2024-09-16T04:42:55.342 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:55.913 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:55.914 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:55.941 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-09-16T04:42:54.593898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":544712360,"num_objects":66498,"num_object_clones":0,"num_object_copies":265992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66525,"num_write_kb":532082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1233,"num_bytes_recovered":9863168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66552,"ondisk_log_size":66552,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1699260,"kb_used_data":920572,"kb_used_omap":14,"kb_used_meta":778609,"kb_avail":375788100,"statfs":{"total":386547056640,"available":384807014400,"internally_reserved":0,"allocated":942665728,"data_stored":941982488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":797296177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,1,6,7,17,6,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40173568,"num_objects":4904,"num_object_clones":0,"num_object_copies":19616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4904,"num_write_kb":39232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004061"},"pg_stats":[{"pgid":"1.f","version":"31'4149","reported_seq":4188,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.597139+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:53.597139+0000","last_peered":"2024-09-16T04:42:53.597139+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:53.597139+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33988608,"num_objects":4149,"num_object_clones":0,"num_object_copies":16596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4149,"num_write_kb":33192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1930}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4188","reported_seq":4247,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:54.241502+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:54.241502+0000","last_peered":"2024-09-16T04:42:54.241502+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:54.241502+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4188,"log_dups_size":0,"ondisk_log_size":4188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34272524,"num_objects":4184,"num_object_clones":0,"num_object_copies":16736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4186,"num_write_kb":33484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4157","reported_seq":4228,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.728481+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:53.728481+0000","last_peered":"2024-09-16T04:42:53.728481+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:53.728481+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4157,"log_dups_size":0,"ondisk_log_size":4157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34028166,"num_objects":4154,"num_object_clones":0,"num_object_copies":16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4156,"num_write_kb":33238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1927}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4166","reported_seq":4227,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:54.288972+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:54.288972+0000","last_peered":"2024-09-16T04:42:54.288972+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:54.288972+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4166,"log_dups_size":0,"ondisk_log_size":4166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34092300,"num_objects":4162,"num_object_clones":0,"num_object_copies":16648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4164,"num_write_kb":33301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1933}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4184","reported_seq":4236,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.594259+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:53.594259+0000","last_peered":"2024-09-16T04:42:53.594259+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:53.594259+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4184,"log_dups_size":0,"ondisk_log_size":4184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34247948,"num_objects":4181,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4182,"num_write_kb":33453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4117","reported_seq":4155,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.596370+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:53.596370+0000","last_peered":"2024-09-16T04:42:53.596370+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:53.596370+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4117,"log_dups_size":0,"ondisk_log_size":4117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33726464,"num_objects":4117,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4117,"num_write_kb":32936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4204","reported_seq":4265,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:54.029989+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:54.029989+0000","last_peered":"2024-09-16T04:42:54.029989+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:54.029989+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4204,"log_dups_size":0,"ondisk_log_size":4204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34387212,"num_objects":4198,"num_object_clones":0,"num_object_copies":16792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4202,"num_write_kb":33596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1972}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4182","reported_seq":6655,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:54.316976+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:54.316976+0000","last_peered":"2024-09-16T04:42:54.316976+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:54.316976+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34215180,"num_objects":4177,"num_object_clones":0,"num_object_copies":16708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4180,"num_write_kb":33428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":9863168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3143}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4257","reported_seq":4310,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.554029+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:53.554029+0000","last_peered":"2024-09-16T04:42:53.554029+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:53.554029+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4257,"log_dups_size":0,"ondisk_log_size":4257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34802200,"num_objects":4249,"num_object_clones":0,"num_object_copies":16996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4254,"num_write_kb":34009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4173","reported_seq":4243,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.753869+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:53.753869+0000","last_peered":"2024-09-16T04:42:53.753869+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:53.753869+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34166028,"num_objects":4171,"num_object_clones":0,"num_object_copies":16684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4171,"num_write_kb":33366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1971}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4137","reported_seq":4189,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.625939+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:53.625939+0000","last_peered":"2024-09-16T04:42:53.625939+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:53.625939+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4137,"log_dups_size":0,"ondisk_log_size":4137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33845138,"num_objects":4132,"num_object_clones":0,"num_object_copies":16528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4134,"num_write_kb":33060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1943}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4097","reported_seq":4132,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.606440+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:53.606440+0000","last_peered":"2024-09-16T04:42:53.606440+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:53.606440+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4097,"log_dups_size":0,"ondisk_log_size":4097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33553030,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4096,"num_write_kb":32767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1968}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4135","reported_seq":4205,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.975059+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:53.975059+0000","last_peered":"2024-09-16T04:42:53.975059+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:53.975059+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4135,"log_dups_size":0,"ondisk_log_size":4135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33864326,"num_objects":4134,"num_object_clones":0,"num_object_copies":16536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4134,"num_write_kb":33071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1909}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4203","reported_seq":4252,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.279766+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:53.279766+0000","last_peered":"2024-09-16T04:42:53.279766+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:53.279766+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4203,"log_dups_size":0,"ondisk_log_size":4203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34395404,"num_objects":4199,"num_object_clones":0,"num_object_copies":16796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4201,"num_write_kb":33604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1965}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4065","reported_seq":4140,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.705698+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:53.705698+0000","last_peered":"2024-09-16T04:42:53.705698+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:53.705698+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4065,"log_dups_size":0,"ondisk_log_size":4065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33273100,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4063,"num_write_kb":32501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1930}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4138","reported_seq":4199,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:54.264367+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:54.264367+0000","last_peered":"2024-09-16T04:42:54.264367+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:54.264367+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4138,"log_dups_size":0,"ondisk_log_size":4138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33854732,"num_objects":4133,"num_object_clones":0,"num_object_copies":16532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4136,"num_write_kb":33076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1894}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":544712360,"num_objects":66498,"num_object_clones":0,"num_object_copies":265992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66525,"num_write_kb":532082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1233,"num_bytes_recovered":9863168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":945082368,"data_stored":945082368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66552,"ondisk_log_size":66552,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481800,"kb_used_data":265464,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93890040,"statfs":{"total":96636764160,"available":96143400960,"internally_reserved":0,"allocated":271835136,"data_stored":271664326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":221507279},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,3,0,5,1,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477240,"kb_used_data":265576,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93894600,"statfs":{"total":96636764160,"available":96148070400,"internally_reserved":0,"allocated":271949824,"data_stored":271779014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051602,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":254188,"kb_used_data":123932,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94117652,"statfs":{"total":96636764160,"available":96376475648,"internally_reserved":0,"allocated":126906368,"data_stored":126735558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133361359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486032,"kb_used_data":265600,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93885808,"statfs":{"total":96636764160,"available":96139067392,"internally_reserved":0,"allocated":271974400,"data_stored":271803590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":225701582},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":272916480,"data_stored":272916480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":127791104,"data_stored":127791104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":272400384,"data_stored":272400384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271974400,"data_stored":271974400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:55.942 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:56.277 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 122 20 67251 67231 4.30404 2.375 0.0502195 0.0362828 2024-09-16T04:42:56.523 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:56.524 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:56.549 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-09-16T04:42:54.593898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":544712360,"num_objects":66498,"num_object_clones":0,"num_object_copies":265992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66525,"num_write_kb":532082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1233,"num_bytes_recovered":9863168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66552,"ondisk_log_size":66552,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1699260,"kb_used_data":920572,"kb_used_omap":14,"kb_used_meta":778609,"kb_avail":375788100,"statfs":{"total":386547056640,"available":384807014400,"internally_reserved":0,"allocated":942665728,"data_stored":941982488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":797296177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,1,6,7,17,6,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40173568,"num_objects":4904,"num_object_clones":0,"num_object_copies":19616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4904,"num_write_kb":39232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004061"},"pg_stats":[{"pgid":"1.f","version":"31'4149","reported_seq":4188,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.597139+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:53.597139+0000","last_peered":"2024-09-16T04:42:53.597139+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:53.597139+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33988608,"num_objects":4149,"num_object_clones":0,"num_object_copies":16596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4149,"num_write_kb":33192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1930}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4188","reported_seq":4247,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:54.241502+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:54.241502+0000","last_peered":"2024-09-16T04:42:54.241502+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:54.241502+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4188,"log_dups_size":0,"ondisk_log_size":4188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34272524,"num_objects":4184,"num_object_clones":0,"num_object_copies":16736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4186,"num_write_kb":33484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4157","reported_seq":4228,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.728481+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:53.728481+0000","last_peered":"2024-09-16T04:42:53.728481+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:53.728481+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4157,"log_dups_size":0,"ondisk_log_size":4157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34028166,"num_objects":4154,"num_object_clones":0,"num_object_copies":16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4156,"num_write_kb":33238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1927}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4166","reported_seq":4227,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:54.288972+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:54.288972+0000","last_peered":"2024-09-16T04:42:54.288972+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:54.288972+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4166,"log_dups_size":0,"ondisk_log_size":4166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34092300,"num_objects":4162,"num_object_clones":0,"num_object_copies":16648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4164,"num_write_kb":33301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1933}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4184","reported_seq":4236,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.594259+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:53.594259+0000","last_peered":"2024-09-16T04:42:53.594259+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:53.594259+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4184,"log_dups_size":0,"ondisk_log_size":4184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34247948,"num_objects":4181,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4182,"num_write_kb":33453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4117","reported_seq":4155,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.596370+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:53.596370+0000","last_peered":"2024-09-16T04:42:53.596370+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:53.596370+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4117,"log_dups_size":0,"ondisk_log_size":4117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33726464,"num_objects":4117,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4117,"num_write_kb":32936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4204","reported_seq":4265,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:54.029989+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:54.029989+0000","last_peered":"2024-09-16T04:42:54.029989+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:54.029989+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4204,"log_dups_size":0,"ondisk_log_size":4204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34387212,"num_objects":4198,"num_object_clones":0,"num_object_copies":16792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4202,"num_write_kb":33596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1972}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4182","reported_seq":6655,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:54.316976+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:54.316976+0000","last_peered":"2024-09-16T04:42:54.316976+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:54.316976+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34215180,"num_objects":4177,"num_object_clones":0,"num_object_copies":16708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4180,"num_write_kb":33428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":9863168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3143}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4257","reported_seq":4310,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.554029+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:53.554029+0000","last_peered":"2024-09-16T04:42:53.554029+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:53.554029+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4257,"log_dups_size":0,"ondisk_log_size":4257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34802200,"num_objects":4249,"num_object_clones":0,"num_object_copies":16996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4254,"num_write_kb":34009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4173","reported_seq":4243,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.753869+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:53.753869+0000","last_peered":"2024-09-16T04:42:53.753869+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:53.753869+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34166028,"num_objects":4171,"num_object_clones":0,"num_object_copies":16684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4171,"num_write_kb":33366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1971}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4137","reported_seq":4189,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.625939+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:53.625939+0000","last_peered":"2024-09-16T04:42:53.625939+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:53.625939+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4137,"log_dups_size":0,"ondisk_log_size":4137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33845138,"num_objects":4132,"num_object_clones":0,"num_object_copies":16528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4134,"num_write_kb":33060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1943}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4097","reported_seq":4132,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.606440+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:53.606440+0000","last_peered":"2024-09-16T04:42:53.606440+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:53.606440+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4097,"log_dups_size":0,"ondisk_log_size":4097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33553030,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4096,"num_write_kb":32767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1968}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4135","reported_seq":4205,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.975059+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:53.975059+0000","last_peered":"2024-09-16T04:42:53.975059+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:53.975059+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4135,"log_dups_size":0,"ondisk_log_size":4135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33864326,"num_objects":4134,"num_object_clones":0,"num_object_copies":16536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4134,"num_write_kb":33071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1909}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4203","reported_seq":4252,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.279766+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:53.279766+0000","last_peered":"2024-09-16T04:42:53.279766+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:53.279766+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4203,"log_dups_size":0,"ondisk_log_size":4203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34395404,"num_objects":4199,"num_object_clones":0,"num_object_copies":16796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4201,"num_write_kb":33604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1965}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4065","reported_seq":4140,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.705698+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:53.705698+0000","last_peered":"2024-09-16T04:42:53.705698+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:53.705698+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4065,"log_dups_size":0,"ondisk_log_size":4065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33273100,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4063,"num_write_kb":32501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1930}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4138","reported_seq":4199,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:54.264367+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:54.264367+0000","last_peered":"2024-09-16T04:42:54.264367+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:54.264367+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4138,"log_dups_size":0,"ondisk_log_size":4138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33854732,"num_objects":4133,"num_object_clones":0,"num_object_copies":16532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4136,"num_write_kb":33076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1894}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":544712360,"num_objects":66498,"num_object_clones":0,"num_object_copies":265992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66525,"num_write_kb":532082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1233,"num_bytes_recovered":9863168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":945082368,"data_stored":945082368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66552,"ondisk_log_size":66552,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481800,"kb_used_data":265464,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93890040,"statfs":{"total":96636764160,"available":96143400960,"internally_reserved":0,"allocated":271835136,"data_stored":271664326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":221507279},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,3,0,5,1,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477240,"kb_used_data":265576,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93894600,"statfs":{"total":96636764160,"available":96148070400,"internally_reserved":0,"allocated":271949824,"data_stored":271779014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051602,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":254188,"kb_used_data":123932,"kb_used_omap":4,"kb_used_meta":130235,"kb_avail":94117652,"statfs":{"total":96636764160,"available":96376475648,"internally_reserved":0,"allocated":126906368,"data_stored":126735558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133361359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486032,"kb_used_data":265600,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93885808,"statfs":{"total":96636764160,"available":96139067392,"internally_reserved":0,"allocated":271974400,"data_stored":271803590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":225701582},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":272916480,"data_stored":272916480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":127791104,"data_stored":127791104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":272400384,"data_stored":272400384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271974400,"data_stored":271974400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:56.550 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:42:56.550 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4149", 'reported_seq': 4188, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:53.597139+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:42:53.597139+0000', 'last_peered': '2024-09-16T04:42:53.597139+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:42:53.597139+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4149, 'log_dups_size': 0, 'ondisk_log_size': 4149, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33988608, 'num_objects': 4149, 'num_object_clones': 0, 'num_object_copies': 16596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4149, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4149, 'num_write_kb': 33192, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1930}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:56.551 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:42:56.551 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4188", 'reported_seq': 4247, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:54.241502+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:42:54.241502+0000', 'last_peered': '2024-09-16T04:42:54.241502+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:42:54.241502+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4188, 'log_dups_size': 0, 'ondisk_log_size': 4188, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34272524, 'num_objects': 4184, 'num_object_clones': 0, 'num_object_copies': 16736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4186, 'num_write_kb': 33484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1973}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:56.551 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:42:56.551 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4157", 'reported_seq': 4228, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:53.728481+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:42:53.728481+0000', 'last_peered': '2024-09-16T04:42:53.728481+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:42:53.728481+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4157, 'log_dups_size': 0, 'ondisk_log_size': 4157, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34028166, 'num_objects': 4154, 'num_object_clones': 0, 'num_object_copies': 16616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4154, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4156, 'num_write_kb': 33238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1927}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:56.551 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:42:56.551 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4166", 'reported_seq': 4227, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:54.288972+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:42:54.288972+0000', 'last_peered': '2024-09-16T04:42:54.288972+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:42:54.288972+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4166, 'log_dups_size': 0, 'ondisk_log_size': 4166, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34092300, 'num_objects': 4162, 'num_object_clones': 0, 'num_object_copies': 16648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4162, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4164, 'num_write_kb': 33301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1933}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:56.551 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:42:56.551 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4184", 'reported_seq': 4236, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:53.594259+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:42:53.594259+0000', 'last_peered': '2024-09-16T04:42:53.594259+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:42:53.594259+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4184, 'log_dups_size': 0, 'ondisk_log_size': 4184, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34247948, 'num_objects': 4181, 'num_object_clones': 0, 'num_object_copies': 16724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4182, 'num_write_kb': 33453, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1898}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:56.551 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:42:56.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4117", 'reported_seq': 4155, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:53.596370+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:42:53.596370+0000', 'last_peered': '2024-09-16T04:42:53.596370+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:42:53.596370+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4117, 'log_dups_size': 0, 'ondisk_log_size': 4117, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33726464, 'num_objects': 4117, 'num_object_clones': 0, 'num_object_copies': 16468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4117, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4117, 'num_write_kb': 32936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1909}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:56.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:42:56.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4204", 'reported_seq': 4265, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:54.029989+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:42:54.029989+0000', 'last_peered': '2024-09-16T04:42:54.029989+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:42:54.029989+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4204, 'log_dups_size': 0, 'ondisk_log_size': 4204, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34387212, 'num_objects': 4198, 'num_object_clones': 0, 'num_object_copies': 16792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4198, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4202, 'num_write_kb': 33596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1972}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:56.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:42:56.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4182", 'reported_seq': 6655, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:54.316976+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:42:54.316976+0000', 'last_peered': '2024-09-16T04:42:54.316976+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:42:54.316976+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4182, 'log_dups_size': 0, 'ondisk_log_size': 4182, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34215180, 'num_objects': 4177, 'num_object_clones': 0, 'num_object_copies': 16708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1034, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4180, 'num_write_kb': 33428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1208, 'num_bytes_recovered': 9863168, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3143}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:56.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:42:56.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4257", 'reported_seq': 4310, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:53.554029+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:42:53.554029+0000', 'last_peered': '2024-09-16T04:42:53.554029+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:42:53.554029+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4257, 'log_dups_size': 0, 'ondisk_log_size': 4257, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34802200, 'num_objects': 4249, 'num_object_clones': 0, 'num_object_copies': 16996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4254, 'num_write_kb': 34009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1909}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:56.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:42:56.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4173", 'reported_seq': 4243, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:53.753869+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:42:53.753869+0000', 'last_peered': '2024-09-16T04:42:53.753869+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:42:53.753869+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4173, 'log_dups_size': 0, 'ondisk_log_size': 4173, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34166028, 'num_objects': 4171, 'num_object_clones': 0, 'num_object_copies': 16684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4171, 'num_write_kb': 33366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1971}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:56.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:42:56.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4137", 'reported_seq': 4189, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:53.625939+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:42:53.625939+0000', 'last_peered': '2024-09-16T04:42:53.625939+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:42:53.625939+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4137, 'log_dups_size': 0, 'ondisk_log_size': 4137, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33845138, 'num_objects': 4132, 'num_object_clones': 0, 'num_object_copies': 16528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4132, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4134, 'num_write_kb': 33060, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1943}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:56.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:42:56.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4097", 'reported_seq': 4132, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:53.606440+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:42:53.606440+0000', 'last_peered': '2024-09-16T04:42:53.606440+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:42:53.606440+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4097, 'log_dups_size': 0, 'ondisk_log_size': 4097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33553030, 'num_objects': 4096, 'num_object_clones': 0, 'num_object_copies': 16384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4096, 'num_write_kb': 32767, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1968}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:56.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:42:56.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4135", 'reported_seq': 4205, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:53.975059+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:42:53.975059+0000', 'last_peered': '2024-09-16T04:42:53.975059+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:42:53.975059+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4135, 'log_dups_size': 0, 'ondisk_log_size': 4135, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33864326, 'num_objects': 4134, 'num_object_clones': 0, 'num_object_copies': 16536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4134, 'num_write_kb': 33071, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1909}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:56.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:42:56.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4203", 'reported_seq': 4252, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:53.279766+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:42:53.279766+0000', 'last_peered': '2024-09-16T04:42:53.279766+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:42:53.279766+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4203, 'log_dups_size': 0, 'ondisk_log_size': 4203, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34395404, 'num_objects': 4199, 'num_object_clones': 0, 'num_object_copies': 16796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4201, 'num_write_kb': 33604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1965}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:42:56.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:42:56.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4065", 'reported_seq': 4140, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:53.705698+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:42:53.705698+0000', 'last_peered': '2024-09-16T04:42:53.705698+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:42:53.705698+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4065, 'log_dups_size': 0, 'ondisk_log_size': 4065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33273100, 'num_objects': 4062, 'num_object_clones': 0, 'num_object_copies': 16248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4063, 'num_write_kb': 32501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1930}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:42:56.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:42:56.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4138", 'reported_seq': 4199, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:54.264367+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:42:54.264367+0000', 'last_peered': '2024-09-16T04:42:54.264367+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:42:54.264367+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4138, 'log_dups_size': 0, 'ondisk_log_size': 4138, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33854732, 'num_objects': 4133, 'num_object_clones': 0, 'num_object_copies': 16532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4136, 'num_write_kb': 33076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1894}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:42:56.554 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:42:57.258 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:57.258 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34349/265992 objects degraded (12.914%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":189,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":66498,"data_bytes":544712360,"bytes_used":1752563712,"bytes_avail":384794492928,"bytes_total":386547056640,"degraded_objects":34349,"degraded_total":265992,"degraded_ratio":0.12913546272068332,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":85983,"recovering_keys_per_sec":0,"num_objects_recovered":128,"num_bytes_recovered":1032192,"num_keys_recovered":0,"write_bytes_sec":2227380,"read_op_per_sec":0,"write_op_per_sec":271},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:42:57.277 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 123 20 67548 67528 4.28791 2.32031 0.0718256 0.0364249 2024-09-16T04:42:57.279 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:42:57.279 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:42:57.921 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-09-16T04:42:56.594858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":544712360,"num_objects":66498,"num_object_clones":0,"num_object_copies":265992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66525,"num_write_kb":532082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1233,"num_bytes_recovered":9863168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66552,"ondisk_log_size":66552,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1711488,"kb_used_data":927232,"kb_used_omap":14,"kb_used_meta":784177,"kb_avail":375775872,"statfs":{"total":386547056640,"available":384794492928,"internally_reserved":0,"allocated":949485568,"data_stored":948802328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":802997809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,2,6,7,15,8,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26738688,"num_objects":3264,"num_object_clones":0,"num_object_copies":13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3264,"num_write_kb":26112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":1032192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004543"},"pg_stats":[{"pgid":"1.f","version":"31'4149","reported_seq":4188,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.597139+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:53.597139+0000","last_peered":"2024-09-16T04:42:53.597139+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:53.597139+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33988608,"num_objects":4149,"num_object_clones":0,"num_object_copies":16596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4149,"num_write_kb":33192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1930}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4188","reported_seq":4247,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:54.241502+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:54.241502+0000","last_peered":"2024-09-16T04:42:54.241502+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:54.241502+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4188,"log_dups_size":0,"ondisk_log_size":4188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34272524,"num_objects":4184,"num_object_clones":0,"num_object_copies":16736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4186,"num_write_kb":33484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4157","reported_seq":4228,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.728481+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:53.728481+0000","last_peered":"2024-09-16T04:42:53.728481+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:53.728481+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4157,"log_dups_size":0,"ondisk_log_size":4157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34028166,"num_objects":4154,"num_object_clones":0,"num_object_copies":16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4156,"num_write_kb":33238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1927}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4166","reported_seq":4227,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:54.288972+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:54.288972+0000","last_peered":"2024-09-16T04:42:54.288972+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:54.288972+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4166,"log_dups_size":0,"ondisk_log_size":4166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34092300,"num_objects":4162,"num_object_clones":0,"num_object_copies":16648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4164,"num_write_kb":33301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1933}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4184","reported_seq":4236,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.594259+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:53.594259+0000","last_peered":"2024-09-16T04:42:53.594259+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:53.594259+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4184,"log_dups_size":0,"ondisk_log_size":4184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34247948,"num_objects":4181,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4182,"num_write_kb":33453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4117","reported_seq":4155,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.596370+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:53.596370+0000","last_peered":"2024-09-16T04:42:53.596370+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:53.596370+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4117,"log_dups_size":0,"ondisk_log_size":4117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33726464,"num_objects":4117,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4117,"num_write_kb":32936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4204","reported_seq":4265,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:54.029989+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:54.029989+0000","last_peered":"2024-09-16T04:42:54.029989+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:54.029989+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4204,"log_dups_size":0,"ondisk_log_size":4204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34387212,"num_objects":4198,"num_object_clones":0,"num_object_copies":16792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4202,"num_write_kb":33596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1972}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4182","reported_seq":6655,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:54.316976+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:54.316976+0000","last_peered":"2024-09-16T04:42:54.316976+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:54.316976+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34215180,"num_objects":4177,"num_object_clones":0,"num_object_copies":16708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4180,"num_write_kb":33428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":9863168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3143}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4257","reported_seq":4310,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.554029+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:53.554029+0000","last_peered":"2024-09-16T04:42:53.554029+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:53.554029+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4257,"log_dups_size":0,"ondisk_log_size":4257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34802200,"num_objects":4249,"num_object_clones":0,"num_object_copies":16996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4254,"num_write_kb":34009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4173","reported_seq":4243,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.753869+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:53.753869+0000","last_peered":"2024-09-16T04:42:53.753869+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:53.753869+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34166028,"num_objects":4171,"num_object_clones":0,"num_object_copies":16684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4171,"num_write_kb":33366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1971}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4137","reported_seq":4189,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.625939+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:53.625939+0000","last_peered":"2024-09-16T04:42:53.625939+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:53.625939+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4137,"log_dups_size":0,"ondisk_log_size":4137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33845138,"num_objects":4132,"num_object_clones":0,"num_object_copies":16528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4134,"num_write_kb":33060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1943}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4097","reported_seq":4132,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.606440+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:53.606440+0000","last_peered":"2024-09-16T04:42:53.606440+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:53.606440+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4097,"log_dups_size":0,"ondisk_log_size":4097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33553030,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4096,"num_write_kb":32767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1968}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4135","reported_seq":4205,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.975059+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:53.975059+0000","last_peered":"2024-09-16T04:42:53.975059+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:53.975059+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4135,"log_dups_size":0,"ondisk_log_size":4135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33864326,"num_objects":4134,"num_object_clones":0,"num_object_copies":16536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4134,"num_write_kb":33071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1909}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4203","reported_seq":4252,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.279766+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:53.279766+0000","last_peered":"2024-09-16T04:42:53.279766+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:53.279766+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4203,"log_dups_size":0,"ondisk_log_size":4203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34395404,"num_objects":4199,"num_object_clones":0,"num_object_copies":16796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4201,"num_write_kb":33604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1965}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4065","reported_seq":4140,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:53.705698+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:53.705698+0000","last_peered":"2024-09-16T04:42:53.705698+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:53.705698+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4065,"log_dups_size":0,"ondisk_log_size":4065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33273100,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4063,"num_write_kb":32501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1930}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4138","reported_seq":4199,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:54.264367+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:54.264367+0000","last_peered":"2024-09-16T04:42:54.264367+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:54.264367+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4138,"log_dups_size":0,"ondisk_log_size":4138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33854732,"num_objects":4133,"num_object_clones":0,"num_object_copies":16532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4136,"num_write_kb":33076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1894}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":544712360,"num_objects":66498,"num_object_clones":0,"num_object_copies":265992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66525,"num_write_kb":532082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1233,"num_bytes_recovered":9863168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":952033280,"data_stored":952033280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66552,"ondisk_log_size":66552,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481800,"kb_used_data":265464,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93890040,"statfs":{"total":96636764160,"available":96143400960,"internally_reserved":0,"allocated":271835136,"data_stored":271664326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":221507279},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,3,0,5,1,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477240,"kb_used_data":265576,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93894600,"statfs":{"total":96636764160,"available":96148070400,"internally_reserved":0,"allocated":271949824,"data_stored":271779014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051603,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":266416,"kb_used_data":130592,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94105424,"statfs":{"total":96636764160,"available":96363954176,"internally_reserved":0,"allocated":133726208,"data_stored":133555398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":139062991},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486032,"kb_used_data":265600,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93885808,"statfs":{"total":96636764160,"available":96139067392,"internally_reserved":0,"allocated":271974400,"data_stored":271803590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":225701582},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":272916480,"data_stored":272916480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":134742016,"data_stored":134742016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":272400384,"data_stored":272400384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271974400,"data_stored":271974400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:58.281 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 124 19 67864 67845 4.2733 2.47656 0.0584368 0.036552 2024-09-16T04:42:59.277 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 125 20 68200 68180 4.26005 2.61719 0.0481337 0.0366646 2024-09-16T04:43:00.279 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 126 20 68538 68518 4.24717 2.64062 0.0732824 0.0367745 2024-09-16T04:43:00.922 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:01.281 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 127 20 68863 68843 4.23372 2.53906 0.0601778 0.0368894 2024-09-16T04:43:01.516 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:01.517 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:01.540 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-09-16T04:43:00.596000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":558007976,"num_objects":68121,"num_object_clones":0,"num_object_copies":272484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68148,"num_write_kb":545066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":10362880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68175,"ondisk_log_size":68175,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1726476,"kb_used_data":946380,"kb_used_omap":14,"kb_used_meta":780017,"kb_avail":375760884,"statfs":{"total":386547056640,"available":384779145216,"internally_reserved":0,"allocated":969093120,"data_stored":968409880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":798737969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,6,8,16,21,6,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40034304,"num_objects":4887,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4887,"num_write_kb":39096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003874"},"pg_stats":[{"pgid":"1.f","version":"31'4254","reported_seq":4291,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.667377+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:58.667377+0000","last_peered":"2024-09-16T04:42:58.667377+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:58.667377+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4254,"log_dups_size":0,"ondisk_log_size":4254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34848768,"num_objects":4254,"num_object_clones":0,"num_object_copies":17016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4254,"num_write_kb":34032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2035}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4303","reported_seq":4365,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.357776+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:59.357776+0000","last_peered":"2024-09-16T04:42:59.357776+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:59.357776+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4303,"log_dups_size":0,"ondisk_log_size":4303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35214604,"num_objects":4299,"num_object_clones":0,"num_object_copies":17196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4301,"num_write_kb":34404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4270","reported_seq":4341,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.916329+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:58.916329+0000","last_peered":"2024-09-16T04:42:58.916329+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:58.916329+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34953862,"num_objects":4267,"num_object_clones":0,"num_object_copies":17068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4269,"num_write_kb":34142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4284","reported_seq":4346,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.302874+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:59.302874+0000","last_peered":"2024-09-16T04:42:59.302874+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:59.302874+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4284,"log_dups_size":0,"ondisk_log_size":4284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35058956,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4282,"num_write_kb":34245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2051}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4273","reported_seq":4325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.535942+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:58.535942+0000","last_peered":"2024-09-16T04:42:58.535942+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:58.535942+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34977036,"num_objects":4270,"num_object_clones":0,"num_object_copies":17080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4271,"num_write_kb":34165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4197","reported_seq":4237,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.629438+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:58.629438+0000","last_peered":"2024-09-16T04:42:58.629438+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:58.629438+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34381824,"num_objects":4197,"num_object_clones":0,"num_object_copies":16788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4197,"num_write_kb":33576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4321","reported_seq":4382,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.236639+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:59.236639+0000","last_peered":"2024-09-16T04:42:59.236639+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:59.236639+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4321,"log_dups_size":0,"ondisk_log_size":4321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35345676,"num_objects":4315,"num_object_clones":0,"num_object_copies":17260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4319,"num_write_kb":34532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2089}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4279","reported_seq":6874,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.335709+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:59.335709+0000","last_peered":"2024-09-16T04:42:59.335709+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:59.335709+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4279,"log_dups_size":0,"ondisk_log_size":4279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35009804,"num_objects":4274,"num_object_clones":0,"num_object_copies":17096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4277,"num_write_kb":34204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1269,"num_bytes_recovered":10362880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3301}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4349","reported_seq":4402,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.617746+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:58.617746+0000","last_peered":"2024-09-16T04:42:58.617746+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:58.617746+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35555864,"num_objects":4341,"num_object_clones":0,"num_object_copies":17364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4346,"num_write_kb":34745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2001}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4265","reported_seq":4339,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.887996+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:58.887996+0000","last_peered":"2024-09-16T04:42:58.887996+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:58.887996+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34919692,"num_objects":4263,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4263,"num_write_kb":34102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2063}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4219","reported_seq":4273,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.527140+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:58.527140+0000","last_peered":"2024-09-16T04:42:58.527140+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:58.527140+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4219,"log_dups_size":0,"ondisk_log_size":4219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34516882,"num_objects":4214,"num_object_clones":0,"num_object_copies":16856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4216,"num_write_kb":33716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2025}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4212","reported_seq":4248,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.667272+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:58.667272+0000","last_peered":"2024-09-16T04:42:58.667272+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:58.667272+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4212,"log_dups_size":0,"ondisk_log_size":4212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34495110,"num_objects":4211,"num_object_clones":0,"num_object_copies":16844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4211,"num_write_kb":33687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4242","reported_seq":4312,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.886171+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:58.886171+0000","last_peered":"2024-09-16T04:42:58.886171+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:58.886171+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34740870,"num_objects":4241,"num_object_clones":0,"num_object_copies":16964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":33927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2016}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4311","reported_seq":4360,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.599544+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:58.599544+0000","last_peered":"2024-09-16T04:42:58.599544+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:58.599544+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4311,"log_dups_size":0,"ondisk_log_size":4311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35280140,"num_objects":4307,"num_object_clones":0,"num_object_copies":17228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4309,"num_write_kb":34468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2073}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4163","reported_seq":4238,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.941877+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:58.941877+0000","last_peered":"2024-09-16T04:42:58.941877+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:58.941877+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4163,"log_dups_size":0,"ondisk_log_size":4163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34075916,"num_objects":4160,"num_object_clones":0,"num_object_copies":16640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4161,"num_write_kb":33285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2028}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4233","reported_seq":4295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.353202+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:59.353202+0000","last_peered":"2024-09-16T04:42:59.353202+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:59.353202+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34632972,"num_objects":4228,"num_object_clones":0,"num_object_copies":16912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4231,"num_write_kb":33836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1989}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":558007976,"num_objects":68121,"num_object_clones":0,"num_object_copies":272484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68148,"num_write_kb":545066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":10362880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":971718656,"data_stored":971718656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68175,"ondisk_log_size":68175,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492408,"kb_used_data":271912,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93879432,"statfs":{"total":96636764160,"available":96132538368,"internally_reserved":0,"allocated":278437888,"data_stored":278267078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":225767119},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,5,0,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487740,"kb_used_data":271916,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93884100,"statfs":{"total":96636764160,"available":96137318400,"internally_reserved":0,"allocated":278441984,"data_stored":278271174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,11,9,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051603,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":266416,"kb_used_data":130592,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94105424,"statfs":{"total":96636764160,"available":96363954176,"internally_reserved":0,"allocated":133726208,"data_stored":133555398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":139062991},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479912,"kb_used_data":271960,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93891928,"statfs":{"total":96636764160,"available":96145334272,"internally_reserved":0,"allocated":278487040,"data_stored":278316230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":212922062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":279470080,"data_stored":279470080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":134742016,"data_stored":134742016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278953984,"data_stored":278953984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":278552576,"data_stored":278552576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:01.542 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:02.112 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:02.113 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:02.138 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-09-16T04:43:00.596000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":558007976,"num_objects":68121,"num_object_clones":0,"num_object_copies":272484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68148,"num_write_kb":545066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":10362880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68175,"ondisk_log_size":68175,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1726476,"kb_used_data":946380,"kb_used_omap":14,"kb_used_meta":780017,"kb_avail":375760884,"statfs":{"total":386547056640,"available":384779145216,"internally_reserved":0,"allocated":969093120,"data_stored":968409880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":798737969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,6,8,16,21,6,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40034304,"num_objects":4887,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4887,"num_write_kb":39096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003874"},"pg_stats":[{"pgid":"1.f","version":"31'4254","reported_seq":4291,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.667377+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:58.667377+0000","last_peered":"2024-09-16T04:42:58.667377+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:58.667377+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4254,"log_dups_size":0,"ondisk_log_size":4254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34848768,"num_objects":4254,"num_object_clones":0,"num_object_copies":17016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4254,"num_write_kb":34032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2035}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4303","reported_seq":4365,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.357776+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:59.357776+0000","last_peered":"2024-09-16T04:42:59.357776+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:59.357776+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4303,"log_dups_size":0,"ondisk_log_size":4303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35214604,"num_objects":4299,"num_object_clones":0,"num_object_copies":17196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4301,"num_write_kb":34404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4270","reported_seq":4341,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.916329+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:58.916329+0000","last_peered":"2024-09-16T04:42:58.916329+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:58.916329+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34953862,"num_objects":4267,"num_object_clones":0,"num_object_copies":17068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4269,"num_write_kb":34142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4284","reported_seq":4346,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.302874+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:59.302874+0000","last_peered":"2024-09-16T04:42:59.302874+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:59.302874+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4284,"log_dups_size":0,"ondisk_log_size":4284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35058956,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4282,"num_write_kb":34245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2051}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4273","reported_seq":4325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.535942+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:58.535942+0000","last_peered":"2024-09-16T04:42:58.535942+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:58.535942+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34977036,"num_objects":4270,"num_object_clones":0,"num_object_copies":17080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4271,"num_write_kb":34165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4197","reported_seq":4237,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.629438+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:58.629438+0000","last_peered":"2024-09-16T04:42:58.629438+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:58.629438+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34381824,"num_objects":4197,"num_object_clones":0,"num_object_copies":16788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4197,"num_write_kb":33576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4321","reported_seq":4382,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.236639+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:59.236639+0000","last_peered":"2024-09-16T04:42:59.236639+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:59.236639+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4321,"log_dups_size":0,"ondisk_log_size":4321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35345676,"num_objects":4315,"num_object_clones":0,"num_object_copies":17260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4319,"num_write_kb":34532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2089}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4279","reported_seq":6874,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.335709+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:59.335709+0000","last_peered":"2024-09-16T04:42:59.335709+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:59.335709+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4279,"log_dups_size":0,"ondisk_log_size":4279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35009804,"num_objects":4274,"num_object_clones":0,"num_object_copies":17096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4277,"num_write_kb":34204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1269,"num_bytes_recovered":10362880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3301}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4349","reported_seq":4402,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.617746+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:58.617746+0000","last_peered":"2024-09-16T04:42:58.617746+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:58.617746+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35555864,"num_objects":4341,"num_object_clones":0,"num_object_copies":17364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4346,"num_write_kb":34745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2001}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4265","reported_seq":4339,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.887996+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:58.887996+0000","last_peered":"2024-09-16T04:42:58.887996+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:58.887996+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34919692,"num_objects":4263,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4263,"num_write_kb":34102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2063}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4219","reported_seq":4273,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.527140+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:58.527140+0000","last_peered":"2024-09-16T04:42:58.527140+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:58.527140+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4219,"log_dups_size":0,"ondisk_log_size":4219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34516882,"num_objects":4214,"num_object_clones":0,"num_object_copies":16856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4216,"num_write_kb":33716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2025}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4212","reported_seq":4248,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.667272+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:58.667272+0000","last_peered":"2024-09-16T04:42:58.667272+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:58.667272+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4212,"log_dups_size":0,"ondisk_log_size":4212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34495110,"num_objects":4211,"num_object_clones":0,"num_object_copies":16844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4211,"num_write_kb":33687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4242","reported_seq":4312,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.886171+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:58.886171+0000","last_peered":"2024-09-16T04:42:58.886171+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:58.886171+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34740870,"num_objects":4241,"num_object_clones":0,"num_object_copies":16964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":33927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2016}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4311","reported_seq":4360,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.599544+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:58.599544+0000","last_peered":"2024-09-16T04:42:58.599544+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:58.599544+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4311,"log_dups_size":0,"ondisk_log_size":4311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35280140,"num_objects":4307,"num_object_clones":0,"num_object_copies":17228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4309,"num_write_kb":34468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2073}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4163","reported_seq":4238,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.941877+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:58.941877+0000","last_peered":"2024-09-16T04:42:58.941877+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:58.941877+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4163,"log_dups_size":0,"ondisk_log_size":4163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34075916,"num_objects":4160,"num_object_clones":0,"num_object_copies":16640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4161,"num_write_kb":33285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2028}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4233","reported_seq":4295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.353202+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:59.353202+0000","last_peered":"2024-09-16T04:42:59.353202+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:59.353202+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34632972,"num_objects":4228,"num_object_clones":0,"num_object_copies":16912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4231,"num_write_kb":33836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1989}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":558007976,"num_objects":68121,"num_object_clones":0,"num_object_copies":272484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68148,"num_write_kb":545066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":10362880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":971718656,"data_stored":971718656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68175,"ondisk_log_size":68175,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492408,"kb_used_data":271912,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93879432,"statfs":{"total":96636764160,"available":96132538368,"internally_reserved":0,"allocated":278437888,"data_stored":278267078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":225767119},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,5,0,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487740,"kb_used_data":271916,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93884100,"statfs":{"total":96636764160,"available":96137318400,"internally_reserved":0,"allocated":278441984,"data_stored":278271174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,11,9,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051603,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":266416,"kb_used_data":130592,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94105424,"statfs":{"total":96636764160,"available":96363954176,"internally_reserved":0,"allocated":133726208,"data_stored":133555398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":139062991},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479912,"kb_used_data":271960,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93891928,"statfs":{"total":96636764160,"available":96145334272,"internally_reserved":0,"allocated":278487040,"data_stored":278316230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":212922062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":279470080,"data_stored":279470080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":134742016,"data_stored":134742016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278953984,"data_stored":278953984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":278552576,"data_stored":278552576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:02.139 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:43:02.139 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4254", 'reported_seq': 4291, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:58.667377+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:42:58.667377+0000', 'last_peered': '2024-09-16T04:42:58.667377+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:42:58.667377+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4254, 'log_dups_size': 0, 'ondisk_log_size': 4254, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34848768, 'num_objects': 4254, 'num_object_clones': 0, 'num_object_copies': 17016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4254, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4254, 'num_write_kb': 34032, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2035}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:02.139 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:43:02.139 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4303", 'reported_seq': 4365, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:59.357776+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:42:59.357776+0000', 'last_peered': '2024-09-16T04:42:59.357776+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:42:59.357776+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4303, 'log_dups_size': 0, 'ondisk_log_size': 4303, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35214604, 'num_objects': 4299, 'num_object_clones': 0, 'num_object_copies': 17196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4299, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4301, 'num_write_kb': 34404, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2088}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:02.139 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:43:02.140 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4270", 'reported_seq': 4341, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:58.916329+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:42:58.916329+0000', 'last_peered': '2024-09-16T04:42:58.916329+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:42:58.916329+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4270, 'log_dups_size': 0, 'ondisk_log_size': 4270, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34953862, 'num_objects': 4267, 'num_object_clones': 0, 'num_object_copies': 17068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4267, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4269, 'num_write_kb': 34142, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2040}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:02.140 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:43:02.140 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4284", 'reported_seq': 4346, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:59.302874+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:42:59.302874+0000', 'last_peered': '2024-09-16T04:42:59.302874+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:42:59.302874+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4284, 'log_dups_size': 0, 'ondisk_log_size': 4284, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35058956, 'num_objects': 4280, 'num_object_clones': 0, 'num_object_copies': 17120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4282, 'num_write_kb': 34245, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2051}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:02.140 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:43:02.140 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4273", 'reported_seq': 4325, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:58.535942+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:42:58.535942+0000', 'last_peered': '2024-09-16T04:42:58.535942+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:42:58.535942+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4273, 'log_dups_size': 0, 'ondisk_log_size': 4273, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34977036, 'num_objects': 4270, 'num_object_clones': 0, 'num_object_copies': 17080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4270, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4271, 'num_write_kb': 34165, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1987}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:02.140 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:43:02.140 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4197", 'reported_seq': 4237, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:58.629438+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:42:58.629438+0000', 'last_peered': '2024-09-16T04:42:58.629438+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:42:58.629438+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4197, 'log_dups_size': 0, 'ondisk_log_size': 4197, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34381824, 'num_objects': 4197, 'num_object_clones': 0, 'num_object_copies': 16788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4197, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4197, 'num_write_kb': 33576, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1989}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:02.140 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:43:02.140 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4321", 'reported_seq': 4382, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:59.236639+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:42:59.236639+0000', 'last_peered': '2024-09-16T04:42:59.236639+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:42:59.236639+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4321, 'log_dups_size': 0, 'ondisk_log_size': 4321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35345676, 'num_objects': 4315, 'num_object_clones': 0, 'num_object_copies': 17260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4319, 'num_write_kb': 34532, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2089}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:02.140 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:43:02.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4279", 'reported_seq': 6874, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:59.335709+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:42:59.335709+0000', 'last_peered': '2024-09-16T04:42:59.335709+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:42:59.335709+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4279, 'log_dups_size': 0, 'ondisk_log_size': 4279, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35009804, 'num_objects': 4274, 'num_object_clones': 0, 'num_object_copies': 17096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 973, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4277, 'num_write_kb': 34204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1269, 'num_bytes_recovered': 10362880, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:02.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:43:02.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4349", 'reported_seq': 4402, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:58.617746+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:42:58.617746+0000', 'last_peered': '2024-09-16T04:42:58.617746+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:42:58.617746+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4349, 'log_dups_size': 0, 'ondisk_log_size': 4349, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35555864, 'num_objects': 4341, 'num_object_clones': 0, 'num_object_copies': 17364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4346, 'num_write_kb': 34745, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2001}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:02.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:43:02.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4265", 'reported_seq': 4339, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:58.887996+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:42:58.887996+0000', 'last_peered': '2024-09-16T04:42:58.887996+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:42:58.887996+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4265, 'log_dups_size': 0, 'ondisk_log_size': 4265, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34919692, 'num_objects': 4263, 'num_object_clones': 0, 'num_object_copies': 17052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4263, 'num_write_kb': 34102, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2063}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:02.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:43:02.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4219", 'reported_seq': 4273, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:58.527140+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:42:58.527140+0000', 'last_peered': '2024-09-16T04:42:58.527140+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:42:58.527140+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4219, 'log_dups_size': 0, 'ondisk_log_size': 4219, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34516882, 'num_objects': 4214, 'num_object_clones': 0, 'num_object_copies': 16856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4216, 'num_write_kb': 33716, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2025}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:02.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:43:02.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4212", 'reported_seq': 4248, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:58.667272+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:42:58.667272+0000', 'last_peered': '2024-09-16T04:42:58.667272+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:42:58.667272+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4212, 'log_dups_size': 0, 'ondisk_log_size': 4212, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34495110, 'num_objects': 4211, 'num_object_clones': 0, 'num_object_copies': 16844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4211, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4211, 'num_write_kb': 33687, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:02.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:43:02.141 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4242", 'reported_seq': 4312, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:58.886171+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:42:58.886171+0000', 'last_peered': '2024-09-16T04:42:58.886171+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:42:58.886171+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4242, 'log_dups_size': 0, 'ondisk_log_size': 4242, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34740870, 'num_objects': 4241, 'num_object_clones': 0, 'num_object_copies': 16964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4241, 'num_write_kb': 33927, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2016}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:02.141 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:43:02.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4311", 'reported_seq': 4360, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:58.599544+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:42:58.599544+0000', 'last_peered': '2024-09-16T04:42:58.599544+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:42:58.599544+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4311, 'log_dups_size': 0, 'ondisk_log_size': 4311, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35280140, 'num_objects': 4307, 'num_object_clones': 0, 'num_object_copies': 17228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4309, 'num_write_kb': 34468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2073}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:02.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:43:02.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4163", 'reported_seq': 4238, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:58.941877+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:42:58.941877+0000', 'last_peered': '2024-09-16T04:42:58.941877+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:42:58.941877+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4163, 'log_dups_size': 0, 'ondisk_log_size': 4163, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34075916, 'num_objects': 4160, 'num_object_clones': 0, 'num_object_copies': 16640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4160, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4161, 'num_write_kb': 33285, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2028}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:02.142 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:43:02.142 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4233", 'reported_seq': 4295, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:42:59.353202+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:42:59.353202+0000', 'last_peered': '2024-09-16T04:42:59.353202+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:42:59.353202+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4233, 'log_dups_size': 0, 'ondisk_log_size': 4233, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34632972, 'num_objects': 4228, 'num_object_clones': 0, 'num_object_copies': 16912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4228, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4231, 'num_write_kb': 33836, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1989}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:02.142 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:43:02.278 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 128 20 69163 69143 4.21896 2.34375 0.0426989 0.0370231 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34288/272484 objects degraded (12.583%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":195,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":68121,"data_bytes":558007976,"bytes_used":1767911424,"bytes_avail":384779145216,"bytes_total":386547056640,"degraded_objects":34288,"degraded_total":272484,"degraded_ratio":0.12583491140764227,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":127617,"recovering_keys_per_sec":0,"num_objects_recovered":189,"num_bytes_recovered":1531904,"num_keys_recovered":0,"write_bytes_sec":3335115,"read_op_per_sec":0,"write_op_per_sec":407},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:43:02.872 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:43:02.873 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:03.278 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 129 20 69460 69440 4.20424 2.32031 0.0804962 0.037151 2024-09-16T04:43:03.473 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:03.473 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:03.495 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-16T04:43:02.596563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":558007976,"num_objects":68121,"num_object_clones":0,"num_object_copies":272484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68148,"num_write_kb":545066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":10362880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68175,"ondisk_log_size":68175,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1755520,"kb_used_data":953216,"kb_used_omap":14,"kb_used_meta":802225,"kb_avail":375731840,"statfs":{"total":386547056640,"available":384749404160,"internally_reserved":0,"allocated":976093184,"data_stored":975409944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":821478961},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,11,16,21,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26476544,"num_objects":3232,"num_object_clones":0,"num_object_copies":12928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3232,"num_write_kb":25856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":1073152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003953"},"pg_stats":[{"pgid":"1.f","version":"31'4254","reported_seq":4291,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.667377+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:42:58.667377+0000","last_peered":"2024-09-16T04:42:58.667377+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:42:58.667377+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4254,"log_dups_size":0,"ondisk_log_size":4254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34848768,"num_objects":4254,"num_object_clones":0,"num_object_copies":17016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4254,"num_write_kb":34032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2035}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4303","reported_seq":4365,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.357776+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:42:59.357776+0000","last_peered":"2024-09-16T04:42:59.357776+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:42:59.357776+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4303,"log_dups_size":0,"ondisk_log_size":4303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35214604,"num_objects":4299,"num_object_clones":0,"num_object_copies":17196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4301,"num_write_kb":34404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4270","reported_seq":4341,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.916329+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:42:58.916329+0000","last_peered":"2024-09-16T04:42:58.916329+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:42:58.916329+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34953862,"num_objects":4267,"num_object_clones":0,"num_object_copies":17068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4269,"num_write_kb":34142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4284","reported_seq":4346,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.302874+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:42:59.302874+0000","last_peered":"2024-09-16T04:42:59.302874+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:42:59.302874+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4284,"log_dups_size":0,"ondisk_log_size":4284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35058956,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4282,"num_write_kb":34245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2051}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4273","reported_seq":4325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.535942+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:42:58.535942+0000","last_peered":"2024-09-16T04:42:58.535942+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:42:58.535942+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34977036,"num_objects":4270,"num_object_clones":0,"num_object_copies":17080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4271,"num_write_kb":34165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4197","reported_seq":4237,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.629438+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:42:58.629438+0000","last_peered":"2024-09-16T04:42:58.629438+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:42:58.629438+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34381824,"num_objects":4197,"num_object_clones":0,"num_object_copies":16788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4197,"num_write_kb":33576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4321","reported_seq":4382,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.236639+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:42:59.236639+0000","last_peered":"2024-09-16T04:42:59.236639+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:42:59.236639+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4321,"log_dups_size":0,"ondisk_log_size":4321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35345676,"num_objects":4315,"num_object_clones":0,"num_object_copies":17260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4319,"num_write_kb":34532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2089}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4279","reported_seq":6874,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.335709+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:42:59.335709+0000","last_peered":"2024-09-16T04:42:59.335709+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:42:59.335709+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4279,"log_dups_size":0,"ondisk_log_size":4279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35009804,"num_objects":4274,"num_object_clones":0,"num_object_copies":17096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4277,"num_write_kb":34204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1269,"num_bytes_recovered":10362880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3301}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4349","reported_seq":4402,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.617746+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:42:58.617746+0000","last_peered":"2024-09-16T04:42:58.617746+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:42:58.617746+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35555864,"num_objects":4341,"num_object_clones":0,"num_object_copies":17364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4346,"num_write_kb":34745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2001}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4265","reported_seq":4339,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.887996+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:42:58.887996+0000","last_peered":"2024-09-16T04:42:58.887996+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:42:58.887996+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34919692,"num_objects":4263,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4263,"num_write_kb":34102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2063}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4219","reported_seq":4273,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.527140+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:42:58.527140+0000","last_peered":"2024-09-16T04:42:58.527140+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:42:58.527140+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4219,"log_dups_size":0,"ondisk_log_size":4219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34516882,"num_objects":4214,"num_object_clones":0,"num_object_copies":16856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4216,"num_write_kb":33716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2025}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4212","reported_seq":4248,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.667272+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:42:58.667272+0000","last_peered":"2024-09-16T04:42:58.667272+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:42:58.667272+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4212,"log_dups_size":0,"ondisk_log_size":4212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34495110,"num_objects":4211,"num_object_clones":0,"num_object_copies":16844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4211,"num_write_kb":33687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4242","reported_seq":4312,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.886171+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:42:58.886171+0000","last_peered":"2024-09-16T04:42:58.886171+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:42:58.886171+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34740870,"num_objects":4241,"num_object_clones":0,"num_object_copies":16964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":33927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2016}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4311","reported_seq":4360,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.599544+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:42:58.599544+0000","last_peered":"2024-09-16T04:42:58.599544+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:42:58.599544+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4311,"log_dups_size":0,"ondisk_log_size":4311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35280140,"num_objects":4307,"num_object_clones":0,"num_object_copies":17228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4309,"num_write_kb":34468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2073}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4163","reported_seq":4238,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:58.941877+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:42:58.941877+0000","last_peered":"2024-09-16T04:42:58.941877+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:42:58.941877+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4163,"log_dups_size":0,"ondisk_log_size":4163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34075916,"num_objects":4160,"num_object_clones":0,"num_object_copies":16640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4161,"num_write_kb":33285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2028}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4233","reported_seq":4295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.353202+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:42:59.353202+0000","last_peered":"2024-09-16T04:42:59.353202+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:42:59.353202+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34632972,"num_objects":4228,"num_object_clones":0,"num_object_copies":16912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4231,"num_write_kb":33836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1989}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":558007976,"num_objects":68121,"num_object_clones":0,"num_object_copies":272484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68148,"num_write_kb":545066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":10362880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":978628608,"data_stored":978628608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68175,"ondisk_log_size":68175,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492408,"kb_used_data":271912,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93879432,"statfs":{"total":96636764160,"available":96132538368,"internally_reserved":0,"allocated":278437888,"data_stored":278267078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":225767119},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,5,0,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487740,"kb_used_data":271916,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93884100,"statfs":{"total":96636764160,"available":96137318400,"internally_reserved":0,"allocated":278441984,"data_stored":278271174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,11,9,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051604,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":295460,"kb_used_data":137428,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94076380,"statfs":{"total":96636764160,"available":96334213120,"internally_reserved":0,"allocated":140726272,"data_stored":140555462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":161803983},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479912,"kb_used_data":271960,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93891928,"statfs":{"total":96636764160,"available":96145334272,"internally_reserved":0,"allocated":278487040,"data_stored":278316230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":212922062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":279470080,"data_stored":279470080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":141651968,"data_stored":141651968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278953984,"data_stored":278953984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":278552576,"data_stored":278552576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:04.279 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 130 20 69787 69767 4.19155 2.55469 0.109927 0.0372632 2024-09-16T04:43:05.279 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 131 20 70114 70094 4.17906 2.55469 0.081062 0.0373753 2024-09-16T04:43:06.279 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 132 20 70446 70426 4.16705 2.59375 0.0484951 0.0374814 2024-09-16T04:43:06.497 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:07.106 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:07.106 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:07.129 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-09-16T04:43:06.597587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":571164328,"num_objects":69727,"num_object_clones":0,"num_object_copies":278908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69754,"num_write_kb":557914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":10854400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69781,"ondisk_log_size":69781,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1781400,"kb_used_data":979096,"kb_used_omap":14,"kb_used_meta":802225,"kb_avail":375705960,"statfs":{"total":386547056640,"available":384722903040,"internally_reserved":0,"allocated":1002594304,"data_stored":1001911064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":821478961},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,3,5,20,1,15,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26451968,"num_objects":3229,"num_object_clones":0,"num_object_copies":12916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3229,"num_write_kb":25832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003688"},"pg_stats":[{"pgid":"1.f","version":"31'4351","reported_seq":4391,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.753837+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:03.753837+0000","last_peered":"2024-09-16T04:43:03.753837+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:03.753837+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4351,"log_dups_size":0,"ondisk_log_size":4351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35643392,"num_objects":4351,"num_object_clones":0,"num_object_copies":17404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4351,"num_write_kb":34808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4413","reported_seq":4475,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:04.327519+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:04.327519+0000","last_peered":"2024-09-16T04:43:04.327519+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:04.327519+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36115724,"num_objects":4409,"num_object_clones":0,"num_object_copies":17636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4411,"num_write_kb":35284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4386","reported_seq":4457,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.974641+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:03.974641+0000","last_peered":"2024-09-16T04:43:03.974641+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:03.974641+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4386,"log_dups_size":0,"ondisk_log_size":4386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35904134,"num_objects":4383,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4385,"num_write_kb":35070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2156}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4391","reported_seq":4453,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:04.303385+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:04.303385+0000","last_peered":"2024-09-16T04:43:04.303385+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:04.303385+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4391,"log_dups_size":0,"ondisk_log_size":4391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35935500,"num_objects":4387,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":35101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2158}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4380","reported_seq":4432,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.657187+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:03.657187+0000","last_peered":"2024-09-16T04:43:03.657187+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:03.657187+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35853580,"num_objects":4377,"num_object_clones":0,"num_object_copies":17508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4378,"num_write_kb":35021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4288","reported_seq":4328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.727009+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:03.727009+0000","last_peered":"2024-09-16T04:43:03.727009+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:03.727009+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":34304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4415","reported_seq":4476,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:04.262149+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:04.262149+0000","last_peered":"2024-09-16T04:43:04.262149+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:04.262149+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4415,"log_dups_size":0,"ondisk_log_size":4415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36115724,"num_objects":4409,"num_object_clones":0,"num_object_copies":17636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4413,"num_write_kb":35284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2183}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4385","reported_seq":7100,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:04.367225+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:04.367225+0000","last_peered":"2024-09-16T04:43:04.367225+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:04.367225+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35878156,"num_objects":4380,"num_object_clones":0,"num_object_copies":17520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4383,"num_write_kb":35052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1329,"num_bytes_recovered":10854400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3467}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4448","reported_seq":4501,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.725015+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:03.725015+0000","last_peered":"2024-09-16T04:43:03.725015+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:03.725015+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4448,"log_dups_size":0,"ondisk_log_size":4448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36366872,"num_objects":4440,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4445,"num_write_kb":35537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4362","reported_seq":4436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.964293+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:03.964293+0000","last_peered":"2024-09-16T04:43:03.964293+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:03.964293+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4362,"log_dups_size":0,"ondisk_log_size":4362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35714316,"num_objects":4360,"num_object_clones":0,"num_object_copies":17440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4360,"num_write_kb":34878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4326","reported_seq":4380,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.703470+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:03.703470+0000","last_peered":"2024-09-16T04:43:03.703470+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:03.703470+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35393426,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4323,"num_write_kb":34572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4296","reported_seq":4332,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.660125+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:03.660125+0000","last_peered":"2024-09-16T04:43:03.660125+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:03.660125+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4296,"log_dups_size":0,"ondisk_log_size":4296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35183238,"num_objects":4295,"num_object_clones":0,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":34359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2167}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4346","reported_seq":4415,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.962158+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:03.962158+0000","last_peered":"2024-09-16T04:43:03.962158+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:03.962158+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4346,"log_dups_size":0,"ondisk_log_size":4346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35592838,"num_objects":4345,"num_object_clones":0,"num_object_copies":17380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4345,"num_write_kb":34759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2120}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4402","reported_seq":4449,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.748912+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:03.748912+0000","last_peered":"2024-09-16T04:43:03.748912+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:03.748912+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4402,"log_dups_size":0,"ondisk_log_size":4402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36025612,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4400,"num_write_kb":35196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2164}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4269","reported_seq":4343,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.835621+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:03.835621+0000","last_peered":"2024-09-16T04:43:03.835621+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:03.835621+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4269,"log_dups_size":0,"ondisk_log_size":4269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34944268,"num_objects":4266,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4267,"num_write_kb":34133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2134}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4323","reported_seq":4385,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:04.373218+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:04.373218+0000","last_peered":"2024-09-16T04:43:04.373218+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:04.373218+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4323,"log_dups_size":0,"ondisk_log_size":4323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35370252,"num_objects":4318,"num_object_clones":0,"num_object_copies":17272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":34556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":571164328,"num_objects":69727,"num_object_clones":0,"num_object_copies":278908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69754,"num_write_kb":557914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":10854400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1005006848,"data_stored":1005006848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69781,"ondisk_log_size":69781,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486304,"kb_used_data":278288,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93885536,"statfs":{"total":96636764160,"available":96138788864,"internally_reserved":0,"allocated":284966912,"data_stored":284796102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":212987599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,6,0,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":498468,"kb_used_data":278484,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93873372,"statfs":{"total":96636764160,"available":96126332928,"internally_reserved":0,"allocated":285167616,"data_stored":284996806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306184,"kb_used_data":143992,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94065656,"statfs":{"total":96636764160,"available":96323231744,"internally_reserved":0,"allocated":147447808,"data_stored":147276998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":166063823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,6,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490444,"kb_used_data":278332,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93881396,"statfs":{"total":96636764160,"available":96134549504,"internally_reserved":0,"allocated":285011968,"data_stored":284841158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":217181902},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,0,0,6,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":286007296,"data_stored":286007296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148389888,"data_stored":148389888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":285483008,"data_stored":285483008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":285126656,"data_stored":285126656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:07.130 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:07.279 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 133 20 70748 70728 4.15346 2.35938 0.103022 0.0376011 2024-09-16T04:43:07.735 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:07.735 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:07.761 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-09-16T04:43:06.597587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":571164328,"num_objects":69727,"num_object_clones":0,"num_object_copies":278908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69754,"num_write_kb":557914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":10854400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69781,"ondisk_log_size":69781,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1781400,"kb_used_data":979096,"kb_used_omap":14,"kb_used_meta":802225,"kb_avail":375705960,"statfs":{"total":386547056640,"available":384722903040,"internally_reserved":0,"allocated":1002594304,"data_stored":1001911064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":821478961},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,3,5,20,1,15,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26451968,"num_objects":3229,"num_object_clones":0,"num_object_copies":12916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3229,"num_write_kb":25832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003688"},"pg_stats":[{"pgid":"1.f","version":"31'4351","reported_seq":4391,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.753837+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:03.753837+0000","last_peered":"2024-09-16T04:43:03.753837+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:03.753837+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4351,"log_dups_size":0,"ondisk_log_size":4351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35643392,"num_objects":4351,"num_object_clones":0,"num_object_copies":17404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4351,"num_write_kb":34808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4413","reported_seq":4475,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:04.327519+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:04.327519+0000","last_peered":"2024-09-16T04:43:04.327519+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:04.327519+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36115724,"num_objects":4409,"num_object_clones":0,"num_object_copies":17636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4411,"num_write_kb":35284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4386","reported_seq":4457,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.974641+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:03.974641+0000","last_peered":"2024-09-16T04:43:03.974641+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:03.974641+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4386,"log_dups_size":0,"ondisk_log_size":4386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35904134,"num_objects":4383,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4385,"num_write_kb":35070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2156}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4391","reported_seq":4453,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:04.303385+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:04.303385+0000","last_peered":"2024-09-16T04:43:04.303385+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:04.303385+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4391,"log_dups_size":0,"ondisk_log_size":4391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35935500,"num_objects":4387,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":35101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2158}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4380","reported_seq":4432,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.657187+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:03.657187+0000","last_peered":"2024-09-16T04:43:03.657187+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:03.657187+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35853580,"num_objects":4377,"num_object_clones":0,"num_object_copies":17508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4378,"num_write_kb":35021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4288","reported_seq":4328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.727009+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:03.727009+0000","last_peered":"2024-09-16T04:43:03.727009+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:03.727009+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":34304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4415","reported_seq":4476,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:04.262149+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:04.262149+0000","last_peered":"2024-09-16T04:43:04.262149+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:04.262149+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4415,"log_dups_size":0,"ondisk_log_size":4415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36115724,"num_objects":4409,"num_object_clones":0,"num_object_copies":17636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4413,"num_write_kb":35284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2183}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4385","reported_seq":7100,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:04.367225+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:04.367225+0000","last_peered":"2024-09-16T04:43:04.367225+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:04.367225+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35878156,"num_objects":4380,"num_object_clones":0,"num_object_copies":17520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4383,"num_write_kb":35052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1329,"num_bytes_recovered":10854400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3467}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4448","reported_seq":4501,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.725015+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:03.725015+0000","last_peered":"2024-09-16T04:43:03.725015+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:03.725015+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4448,"log_dups_size":0,"ondisk_log_size":4448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36366872,"num_objects":4440,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4445,"num_write_kb":35537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4362","reported_seq":4436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.964293+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:03.964293+0000","last_peered":"2024-09-16T04:43:03.964293+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:03.964293+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4362,"log_dups_size":0,"ondisk_log_size":4362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35714316,"num_objects":4360,"num_object_clones":0,"num_object_copies":17440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4360,"num_write_kb":34878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4326","reported_seq":4380,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.703470+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:03.703470+0000","last_peered":"2024-09-16T04:43:03.703470+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:03.703470+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35393426,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4323,"num_write_kb":34572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4296","reported_seq":4332,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.660125+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:03.660125+0000","last_peered":"2024-09-16T04:43:03.660125+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:03.660125+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4296,"log_dups_size":0,"ondisk_log_size":4296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35183238,"num_objects":4295,"num_object_clones":0,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":34359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2167}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4346","reported_seq":4415,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.962158+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:03.962158+0000","last_peered":"2024-09-16T04:43:03.962158+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:03.962158+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4346,"log_dups_size":0,"ondisk_log_size":4346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35592838,"num_objects":4345,"num_object_clones":0,"num_object_copies":17380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4345,"num_write_kb":34759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2120}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4402","reported_seq":4449,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.748912+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:03.748912+0000","last_peered":"2024-09-16T04:43:03.748912+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:03.748912+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4402,"log_dups_size":0,"ondisk_log_size":4402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36025612,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4400,"num_write_kb":35196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2164}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4269","reported_seq":4343,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.835621+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:03.835621+0000","last_peered":"2024-09-16T04:43:03.835621+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:03.835621+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4269,"log_dups_size":0,"ondisk_log_size":4269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34944268,"num_objects":4266,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4267,"num_write_kb":34133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2134}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4323","reported_seq":4385,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:04.373218+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:04.373218+0000","last_peered":"2024-09-16T04:43:04.373218+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:04.373218+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4323,"log_dups_size":0,"ondisk_log_size":4323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35370252,"num_objects":4318,"num_object_clones":0,"num_object_copies":17272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":34556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":571164328,"num_objects":69727,"num_object_clones":0,"num_object_copies":278908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69754,"num_write_kb":557914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":10854400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1005006848,"data_stored":1005006848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69781,"ondisk_log_size":69781,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486304,"kb_used_data":278288,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93885536,"statfs":{"total":96636764160,"available":96138788864,"internally_reserved":0,"allocated":284966912,"data_stored":284796102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":212987599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,6,0,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":498468,"kb_used_data":278484,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93873372,"statfs":{"total":96636764160,"available":96126332928,"internally_reserved":0,"allocated":285167616,"data_stored":284996806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306184,"kb_used_data":143992,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94065656,"statfs":{"total":96636764160,"available":96323231744,"internally_reserved":0,"allocated":147447808,"data_stored":147276998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":166063823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,6,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490444,"kb_used_data":278332,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93881396,"statfs":{"total":96636764160,"available":96134549504,"internally_reserved":0,"allocated":285011968,"data_stored":284841158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":217181902},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,0,0,6,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":286007296,"data_stored":286007296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148389888,"data_stored":148389888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":285483008,"data_stored":285483008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":285126656,"data_stored":285126656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:07.763 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:43:07.763 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4351", 'reported_seq': 4391, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:03.753837+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:43:03.753837+0000', 'last_peered': '2024-09-16T04:43:03.753837+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:43:03.753837+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4351, 'log_dups_size': 0, 'ondisk_log_size': 4351, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35643392, 'num_objects': 4351, 'num_object_clones': 0, 'num_object_copies': 17404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4351, 'num_write_kb': 34808, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2132}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:07.763 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:43:07.763 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4413", 'reported_seq': 4475, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:04.327519+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:43:04.327519+0000', 'last_peered': '2024-09-16T04:43:04.327519+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:43:04.327519+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4413, 'log_dups_size': 0, 'ondisk_log_size': 4413, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36115724, 'num_objects': 4409, 'num_object_clones': 0, 'num_object_copies': 17636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4411, 'num_write_kb': 35284, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2198}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:07.763 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:43:07.763 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4386", 'reported_seq': 4457, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:03.974641+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:43:03.974641+0000', 'last_peered': '2024-09-16T04:43:03.974641+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:43:03.974641+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4386, 'log_dups_size': 0, 'ondisk_log_size': 4386, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35904134, 'num_objects': 4383, 'num_object_clones': 0, 'num_object_copies': 17532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4385, 'num_write_kb': 35070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2156}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:07.763 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:43:07.763 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4391", 'reported_seq': 4453, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:04.303385+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:43:04.303385+0000', 'last_peered': '2024-09-16T04:43:04.303385+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:43:04.303385+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4391, 'log_dups_size': 0, 'ondisk_log_size': 4391, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35935500, 'num_objects': 4387, 'num_object_clones': 0, 'num_object_copies': 17548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4387, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4389, 'num_write_kb': 35101, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2158}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:07.763 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:43:07.763 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4380", 'reported_seq': 4432, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:03.657187+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:43:03.657187+0000', 'last_peered': '2024-09-16T04:43:03.657187+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:43:03.657187+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4380, 'log_dups_size': 0, 'ondisk_log_size': 4380, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35853580, 'num_objects': 4377, 'num_object_clones': 0, 'num_object_copies': 17508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4378, 'num_write_kb': 35021, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2094}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:07.764 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:43:07.764 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4288", 'reported_seq': 4328, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:03.727009+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:43:03.727009+0000', 'last_peered': '2024-09-16T04:43:03.727009+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:43:03.727009+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4288, 'log_dups_size': 0, 'ondisk_log_size': 4288, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35127296, 'num_objects': 4288, 'num_object_clones': 0, 'num_object_copies': 17152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4288, 'num_write_kb': 34304, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2080}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:07.764 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:43:07.764 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4415", 'reported_seq': 4476, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:04.262149+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:43:04.262149+0000', 'last_peered': '2024-09-16T04:43:04.262149+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:43:04.262149+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4415, 'log_dups_size': 0, 'ondisk_log_size': 4415, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36115724, 'num_objects': 4409, 'num_object_clones': 0, 'num_object_copies': 17636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4413, 'num_write_kb': 35284, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2183}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:07.764 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:43:07.764 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4385", 'reported_seq': 7100, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:04.367225+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:43:04.367225+0000', 'last_peered': '2024-09-16T04:43:04.367225+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:43:04.367225+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4385, 'log_dups_size': 0, 'ondisk_log_size': 4385, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35878156, 'num_objects': 4380, 'num_object_clones': 0, 'num_object_copies': 17520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 913, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4380, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4383, 'num_write_kb': 35052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1329, 'num_bytes_recovered': 10854400, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3467}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:07.764 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:43:07.764 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4448", 'reported_seq': 4501, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:03.725015+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:43:03.725015+0000', 'last_peered': '2024-09-16T04:43:03.725015+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:43:03.725015+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4448, 'log_dups_size': 0, 'ondisk_log_size': 4448, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36366872, 'num_objects': 4440, 'num_object_clones': 0, 'num_object_copies': 17760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4445, 'num_write_kb': 35537, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2100}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:07.764 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:43:07.764 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4362", 'reported_seq': 4436, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:03.964293+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:43:03.964293+0000', 'last_peered': '2024-09-16T04:43:03.964293+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:43:03.964293+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4362, 'log_dups_size': 0, 'ondisk_log_size': 4362, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35714316, 'num_objects': 4360, 'num_object_clones': 0, 'num_object_copies': 17440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4360, 'num_write_kb': 34878, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2160}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:07.765 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:43:07.765 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4326", 'reported_seq': 4380, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:03.703470+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:43:03.703470+0000', 'last_peered': '2024-09-16T04:43:03.703470+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:43:03.703470+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4326, 'log_dups_size': 0, 'ondisk_log_size': 4326, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35393426, 'num_objects': 4321, 'num_object_clones': 0, 'num_object_copies': 17284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4323, 'num_write_kb': 34572, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2132}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:07.765 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:43:07.765 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4296", 'reported_seq': 4332, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:03.660125+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:43:03.660125+0000', 'last_peered': '2024-09-16T04:43:03.660125+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:43:03.660125+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4296, 'log_dups_size': 0, 'ondisk_log_size': 4296, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35183238, 'num_objects': 4295, 'num_object_clones': 0, 'num_object_copies': 17180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4295, 'num_write_kb': 34359, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:07.765 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:43:07.765 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4346", 'reported_seq': 4415, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:03.962158+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:43:03.962158+0000', 'last_peered': '2024-09-16T04:43:03.962158+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:43:03.962158+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4346, 'log_dups_size': 0, 'ondisk_log_size': 4346, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35592838, 'num_objects': 4345, 'num_object_clones': 0, 'num_object_copies': 17380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4345, 'num_write_kb': 34759, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2120}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:07.765 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:43:07.765 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4402", 'reported_seq': 4449, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:03.748912+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:43:03.748912+0000', 'last_peered': '2024-09-16T04:43:03.748912+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:43:03.748912+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4402, 'log_dups_size': 0, 'ondisk_log_size': 4402, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36025612, 'num_objects': 4398, 'num_object_clones': 0, 'num_object_copies': 17592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4400, 'num_write_kb': 35196, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2164}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:07.765 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:43:07.765 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4269", 'reported_seq': 4343, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:03.835621+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:43:03.835621+0000', 'last_peered': '2024-09-16T04:43:03.835621+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:43:03.835621+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4269, 'log_dups_size': 0, 'ondisk_log_size': 4269, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34944268, 'num_objects': 4266, 'num_object_clones': 0, 'num_object_copies': 17064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4266, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4267, 'num_write_kb': 34133, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:07.765 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:43:07.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4323", 'reported_seq': 4385, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:04.373218+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:43:04.373218+0000', 'last_peered': '2024-09-16T04:43:04.373218+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:43:04.373218+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4323, 'log_dups_size': 0, 'ondisk_log_size': 4323, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35370252, 'num_objects': 4318, 'num_object_clones': 0, 'num_object_copies': 17272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4321, 'num_write_kb': 34556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2079}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:07.766 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:43:08.279 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 134 20 71036 71016 4.13925 2.25 0.0810111 0.0377264 2024-09-16T04:43:08.463 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:08.463 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34228/278908 objects degraded (12.272%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":200,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":69727,"data_bytes":571164328,"bytes_used":1824153600,"bytes_avail":384722903040,"bytes_total":386547056640,"degraded_objects":34228,"degraded_total":278908,"degraded_ratio":0.12272147087928637,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":82577,"recovering_keys_per_sec":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"write_bytes_sec":2203653,"read_op_per_sec":0,"write_op_per_sec":269},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:43:08.488 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:43:08.488 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:09.091 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:09.092 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:09.118 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-09-16T04:43:08.598904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":571164328,"num_objects":69727,"num_object_clones":0,"num_object_copies":278908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69754,"num_write_kb":557914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":10854400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69781,"ondisk_log_size":69781,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1781400,"kb_used_data":979096,"kb_used_omap":14,"kb_used_meta":802225,"kb_avail":375705960,"statfs":{"total":386547056640,"available":384722903040,"internally_reserved":0,"allocated":1002594304,"data_stored":1001911064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":821478961},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,3,5,20,1,15,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26451968,"num_objects":3229,"num_object_clones":0,"num_object_copies":12916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3229,"num_write_kb":25832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004046"},"pg_stats":[{"pgid":"1.f","version":"31'4351","reported_seq":4391,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.753837+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:03.753837+0000","last_peered":"2024-09-16T04:43:03.753837+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:03.753837+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4351,"log_dups_size":0,"ondisk_log_size":4351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35643392,"num_objects":4351,"num_object_clones":0,"num_object_copies":17404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4351,"num_write_kb":34808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4413","reported_seq":4475,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:04.327519+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:04.327519+0000","last_peered":"2024-09-16T04:43:04.327519+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:04.327519+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36115724,"num_objects":4409,"num_object_clones":0,"num_object_copies":17636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4411,"num_write_kb":35284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4386","reported_seq":4457,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.974641+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:03.974641+0000","last_peered":"2024-09-16T04:43:03.974641+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:03.974641+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4386,"log_dups_size":0,"ondisk_log_size":4386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35904134,"num_objects":4383,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4385,"num_write_kb":35070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2156}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4391","reported_seq":4453,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:04.303385+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:04.303385+0000","last_peered":"2024-09-16T04:43:04.303385+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:04.303385+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4391,"log_dups_size":0,"ondisk_log_size":4391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35935500,"num_objects":4387,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":35101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2158}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4380","reported_seq":4432,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.657187+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:03.657187+0000","last_peered":"2024-09-16T04:43:03.657187+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:03.657187+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35853580,"num_objects":4377,"num_object_clones":0,"num_object_copies":17508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4378,"num_write_kb":35021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4288","reported_seq":4328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.727009+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:03.727009+0000","last_peered":"2024-09-16T04:43:03.727009+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:03.727009+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":34304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4415","reported_seq":4476,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:04.262149+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:04.262149+0000","last_peered":"2024-09-16T04:43:04.262149+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:04.262149+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4415,"log_dups_size":0,"ondisk_log_size":4415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36115724,"num_objects":4409,"num_object_clones":0,"num_object_copies":17636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4413,"num_write_kb":35284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2183}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4385","reported_seq":7100,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:04.367225+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:04.367225+0000","last_peered":"2024-09-16T04:43:04.367225+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:04.367225+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35878156,"num_objects":4380,"num_object_clones":0,"num_object_copies":17520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4383,"num_write_kb":35052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1329,"num_bytes_recovered":10854400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3467}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4448","reported_seq":4501,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.725015+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:03.725015+0000","last_peered":"2024-09-16T04:43:03.725015+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:03.725015+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4448,"log_dups_size":0,"ondisk_log_size":4448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36366872,"num_objects":4440,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4445,"num_write_kb":35537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4362","reported_seq":4436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.964293+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:03.964293+0000","last_peered":"2024-09-16T04:43:03.964293+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:03.964293+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4362,"log_dups_size":0,"ondisk_log_size":4362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35714316,"num_objects":4360,"num_object_clones":0,"num_object_copies":17440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4360,"num_write_kb":34878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4326","reported_seq":4380,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.703470+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:03.703470+0000","last_peered":"2024-09-16T04:43:03.703470+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:03.703470+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35393426,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4323,"num_write_kb":34572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4296","reported_seq":4332,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.660125+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:03.660125+0000","last_peered":"2024-09-16T04:43:03.660125+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:03.660125+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4296,"log_dups_size":0,"ondisk_log_size":4296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35183238,"num_objects":4295,"num_object_clones":0,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":34359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2167}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4346","reported_seq":4415,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.962158+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:03.962158+0000","last_peered":"2024-09-16T04:43:03.962158+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:03.962158+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4346,"log_dups_size":0,"ondisk_log_size":4346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35592838,"num_objects":4345,"num_object_clones":0,"num_object_copies":17380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4345,"num_write_kb":34759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2120}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4402","reported_seq":4449,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.748912+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:03.748912+0000","last_peered":"2024-09-16T04:43:03.748912+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:03.748912+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4402,"log_dups_size":0,"ondisk_log_size":4402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36025612,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4400,"num_write_kb":35196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2164}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4269","reported_seq":4343,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:03.835621+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:03.835621+0000","last_peered":"2024-09-16T04:43:03.835621+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:03.835621+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4269,"log_dups_size":0,"ondisk_log_size":4269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34944268,"num_objects":4266,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4267,"num_write_kb":34133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2134}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4323","reported_seq":4385,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:04.373218+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:04.373218+0000","last_peered":"2024-09-16T04:43:04.373218+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:04.373218+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4323,"log_dups_size":0,"ondisk_log_size":4323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35370252,"num_objects":4318,"num_object_clones":0,"num_object_copies":17272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":34556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":571164328,"num_objects":69727,"num_object_clones":0,"num_object_copies":278908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69754,"num_write_kb":557914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":10854400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1005006848,"data_stored":1005006848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69781,"ondisk_log_size":69781,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486304,"kb_used_data":278288,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93885536,"statfs":{"total":96636764160,"available":96138788864,"internally_reserved":0,"allocated":284966912,"data_stored":284796102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":212987599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,6,0,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":498468,"kb_used_data":278484,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93873372,"statfs":{"total":96636764160,"available":96126332928,"internally_reserved":0,"allocated":285167616,"data_stored":284996806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306184,"kb_used_data":143992,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94065656,"statfs":{"total":96636764160,"available":96323231744,"internally_reserved":0,"allocated":147447808,"data_stored":147276998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":166063823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,6,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490444,"kb_used_data":278332,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93881396,"statfs":{"total":96636764160,"available":96134549504,"internally_reserved":0,"allocated":285011968,"data_stored":284841158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":217181902},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,0,0,6,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":286007296,"data_stored":286007296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148389888,"data_stored":148389888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":285483008,"data_stored":285483008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":285126656,"data_stored":285126656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:09.279 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 135 20 71330 71310 4.12561 2.29688 0.0585992 0.0378602 2024-09-16T04:43:10.279 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 136 20 71653 71633 4.11383 2.52344 0.0549657 0.0379657 2024-09-16T04:43:11.280 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 137 20 71982 71962 4.10253 2.57031 0.13308 0.0380734 2024-09-16T04:43:12.121 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:12.280 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 138 20 72295 72275 4.09052 2.44531 0.131649 0.0381793 2024-09-16T04:43:12.701 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:12.701 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:12.731 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-09-16T04:43:12.599999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":583829160,"num_objects":71273,"num_object_clones":0,"num_object_copies":285092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71300,"num_write_kb":570282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1425,"num_bytes_recovered":11436032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71327,"ondisk_log_size":71327,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1862180,"kb_used_data":1005728,"kb_used_omap":14,"kb_used_meta":856369,"kb_avail":375625180,"statfs":{"total":386547056640,"available":384640184320,"internally_reserved":0,"allocated":1029865472,"data_stored":1029182232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":876922417},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,6,7,19,18,20,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25821184,"num_objects":3152,"num_object_clones":0,"num_object_copies":12608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3152,"num_write_kb":25216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":1073152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003999"},"pg_stats":[{"pgid":"1.f","version":"31'4448","reported_seq":4486,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.757851+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:08.757851+0000","last_peered":"2024-09-16T04:43:08.757851+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:08.757851+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4448,"log_dups_size":0,"ondisk_log_size":4448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36438016,"num_objects":4448,"num_object_clones":0,"num_object_copies":17792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4448,"num_write_kb":35584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2229}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4522","reported_seq":4584,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:09.219397+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:09.219397+0000","last_peered":"2024-09-16T04:43:09.219397+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:09.219397+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37008652,"num_objects":4518,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4520,"num_write_kb":36156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2307}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4493","reported_seq":4564,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.952182+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:08.952182+0000","last_peered":"2024-09-16T04:43:08.952182+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:08.952182+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36780678,"num_objects":4490,"num_object_clones":0,"num_object_copies":17960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4492,"num_write_kb":35926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4490","reported_seq":4552,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:09.387932+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:09.387932+0000","last_peered":"2024-09-16T04:43:09.387932+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:09.387932+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36746508,"num_objects":4486,"num_object_clones":0,"num_object_copies":17944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4488,"num_write_kb":35893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4463","reported_seq":4515,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.770637+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:08.770637+0000","last_peered":"2024-09-16T04:43:08.770637+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:08.770637+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4463,"log_dups_size":0,"ondisk_log_size":4463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36533516,"num_objects":4460,"num_object_clones":0,"num_object_copies":17840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4461,"num_write_kb":35685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4367","reported_seq":4407,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.741991+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:08.741991+0000","last_peered":"2024-09-16T04:43:08.741991+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:08.741991+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4367,"log_dups_size":0,"ondisk_log_size":4367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35774464,"num_objects":4367,"num_object_clones":0,"num_object_copies":17468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4367,"num_write_kb":34936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2159}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4528","reported_seq":4589,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:09.356546+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:09.356546+0000","last_peered":"2024-09-16T04:43:09.356546+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:09.356546+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37041420,"num_objects":4522,"num_object_clones":0,"num_object_copies":18088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4526,"num_write_kb":36188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2296}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4471","reported_seq":7328,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:09.454415+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:09.454415+0000","last_peered":"2024-09-16T04:43:09.454415+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:09.454415+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4471,"log_dups_size":0,"ondisk_log_size":4471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36582668,"num_objects":4466,"num_object_clones":0,"num_object_copies":17864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4469,"num_write_kb":35740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1400,"num_bytes_recovered":11436032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4537","reported_seq":4590,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.765267+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:08.765267+0000","last_peered":"2024-09-16T04:43:08.765267+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:08.765267+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4537,"log_dups_size":0,"ondisk_log_size":4537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37095960,"num_objects":4529,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4534,"num_write_kb":36249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2189}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4463","reported_seq":4537,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.799350+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:08.799350+0000","last_peered":"2024-09-16T04:43:08.799350+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:08.799350+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4463,"log_dups_size":0,"ondisk_log_size":4463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36541708,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4461,"num_write_kb":35686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2261}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4412","reported_seq":4464,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.705220+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:08.705220+0000","last_peered":"2024-09-16T04:43:08.705220+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:08.705220+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36097938,"num_objects":4407,"num_object_clones":0,"num_object_copies":17628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4409,"num_write_kb":35260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2218}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4406","reported_seq":4440,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.777198+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:08.777198+0000","last_peered":"2024-09-16T04:43:08.777198+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:08.777198+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4406,"log_dups_size":0,"ondisk_log_size":4406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36084358,"num_objects":4405,"num_object_clones":0,"num_object_copies":17620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4405,"num_write_kb":35239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4433","reported_seq":4502,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.925027+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:08.925027+0000","last_peered":"2024-09-16T04:43:08.925027+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:08.925027+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4433,"log_dups_size":0,"ondisk_log_size":4433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36305542,"num_objects":4432,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4432,"num_write_kb":35455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2207}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4512","reported_seq":4561,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.679924+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:08.679924+0000","last_peered":"2024-09-16T04:43:08.679924+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:08.679924+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4512,"log_dups_size":0,"ondisk_log_size":4512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36926732,"num_objects":4508,"num_object_clones":0,"num_object_copies":18032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4510,"num_write_kb":36076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2274}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4373","reported_seq":4448,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.967068+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:08.967068+0000","last_peered":"2024-09-16T04:43:08.967068+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:08.967068+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35796236,"num_objects":4370,"num_object_clones":0,"num_object_copies":17480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4371,"num_write_kb":34965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2238}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4409","reported_seq":4471,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:09.305171+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:09.305171+0000","last_peered":"2024-09-16T04:43:09.305171+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:09.305171+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4409,"log_dups_size":0,"ondisk_log_size":4409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36074764,"num_objects":4404,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4407,"num_write_kb":35244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2165}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":583829160,"num_objects":71273,"num_object_clones":0,"num_object_copies":285092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71300,"num_write_kb":570282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1425,"num_bytes_recovered":11436032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1031000064,"data_stored":1031000064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71327,"ondisk_log_size":71327,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514784,"kb_used_data":284556,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93857056,"statfs":{"total":96636764160,"available":96109625344,"internally_reserved":0,"allocated":291385344,"data_stored":291214534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235728591},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,2,3,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510380,"kb_used_data":284828,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":93861460,"statfs":{"total":96636764160,"available":96114135040,"internally_reserved":0,"allocated":291663872,"data_stored":291493062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":230947269},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,5,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051606,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318016,"kb_used_data":151664,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94053824,"statfs":{"total":96636764160,"available":96311115776,"internally_reserved":0,"allocated":155303936,"data_stored":155133126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170323663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519000,"kb_used_data":284680,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93852840,"statfs":{"total":96636764160,"available":96105308160,"internally_reserved":0,"allocated":291512320,"data_stored":291341510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":239922894},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,8,7,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292392960,"data_stored":292392960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155103232,"data_stored":155103232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":291860480,"data_stored":291860480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":291643392,"data_stored":291643392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:12.732 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:13.280 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 139 20 72577 72557 4.07695 2.20312 0.0410477 0.0383069 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:13.339 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-09-16T04:43:12.599999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":583829160,"num_objects":71273,"num_object_clones":0,"num_object_copies":285092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71300,"num_write_kb":570282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1425,"num_bytes_recovered":11436032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71327,"ondisk_log_size":71327,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1862180,"kb_used_data":1005728,"kb_used_omap":14,"kb_used_meta":856369,"kb_avail":375625180,"statfs":{"total":386547056640,"available":384640184320,"internally_reserved":0,"allocated":1029865472,"data_stored":1029182232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":876922417},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,6,7,19,18,20,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25821184,"num_objects":3152,"num_object_clones":0,"num_object_copies":12608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3152,"num_write_kb":25216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":1073152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003999"},"pg_stats":[{"pgid":"1.f","version":"31'4448","reported_seq":4486,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.757851+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:08.757851+0000","last_peered":"2024-09-16T04:43:08.757851+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:08.757851+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4448,"log_dups_size":0,"ondisk_log_size":4448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36438016,"num_objects":4448,"num_object_clones":0,"num_object_copies":17792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4448,"num_write_kb":35584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2229}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4522","reported_seq":4584,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:09.219397+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:09.219397+0000","last_peered":"2024-09-16T04:43:09.219397+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:09.219397+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37008652,"num_objects":4518,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4520,"num_write_kb":36156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2307}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4493","reported_seq":4564,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.952182+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:08.952182+0000","last_peered":"2024-09-16T04:43:08.952182+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:08.952182+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36780678,"num_objects":4490,"num_object_clones":0,"num_object_copies":17960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4492,"num_write_kb":35926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4490","reported_seq":4552,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:09.387932+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:09.387932+0000","last_peered":"2024-09-16T04:43:09.387932+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:09.387932+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36746508,"num_objects":4486,"num_object_clones":0,"num_object_copies":17944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4488,"num_write_kb":35893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4463","reported_seq":4515,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.770637+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:08.770637+0000","last_peered":"2024-09-16T04:43:08.770637+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:08.770637+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4463,"log_dups_size":0,"ondisk_log_size":4463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36533516,"num_objects":4460,"num_object_clones":0,"num_object_copies":17840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4461,"num_write_kb":35685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4367","reported_seq":4407,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.741991+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:08.741991+0000","last_peered":"2024-09-16T04:43:08.741991+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:08.741991+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4367,"log_dups_size":0,"ondisk_log_size":4367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35774464,"num_objects":4367,"num_object_clones":0,"num_object_copies":17468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4367,"num_write_kb":34936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2159}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4528","reported_seq":4589,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:09.356546+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:09.356546+0000","last_peered":"2024-09-16T04:43:09.356546+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:09.356546+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37041420,"num_objects":4522,"num_object_clones":0,"num_object_copies":18088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4526,"num_write_kb":36188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2296}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4471","reported_seq":7328,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:09.454415+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:09.454415+0000","last_peered":"2024-09-16T04:43:09.454415+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:09.454415+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4471,"log_dups_size":0,"ondisk_log_size":4471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36582668,"num_objects":4466,"num_object_clones":0,"num_object_copies":17864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4469,"num_write_kb":35740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1400,"num_bytes_recovered":11436032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4537","reported_seq":4590,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.765267+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:08.765267+0000","last_peered":"2024-09-16T04:43:08.765267+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:08.765267+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4537,"log_dups_size":0,"ondisk_log_size":4537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37095960,"num_objects":4529,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4534,"num_write_kb":36249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2189}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4463","reported_seq":4537,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.799350+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:08.799350+0000","last_peered":"2024-09-16T04:43:08.799350+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:08.799350+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4463,"log_dups_size":0,"ondisk_log_size":4463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36541708,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4461,"num_write_kb":35686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2261}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4412","reported_seq":4464,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.705220+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:08.705220+0000","last_peered":"2024-09-16T04:43:08.705220+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:08.705220+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36097938,"num_objects":4407,"num_object_clones":0,"num_object_copies":17628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4409,"num_write_kb":35260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2218}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4406","reported_seq":4440,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.777198+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:08.777198+0000","last_peered":"2024-09-16T04:43:08.777198+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:08.777198+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4406,"log_dups_size":0,"ondisk_log_size":4406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36084358,"num_objects":4405,"num_object_clones":0,"num_object_copies":17620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4405,"num_write_kb":35239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4433","reported_seq":4502,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.925027+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:08.925027+0000","last_peered":"2024-09-16T04:43:08.925027+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:08.925027+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4433,"log_dups_size":0,"ondisk_log_size":4433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36305542,"num_objects":4432,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4432,"num_write_kb":35455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2207}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4512","reported_seq":4561,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.679924+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:08.679924+0000","last_peered":"2024-09-16T04:43:08.679924+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:08.679924+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4512,"log_dups_size":0,"ondisk_log_size":4512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36926732,"num_objects":4508,"num_object_clones":0,"num_object_copies":18032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4510,"num_write_kb":36076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2274}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4373","reported_seq":4448,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.967068+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:08.967068+0000","last_peered":"2024-09-16T04:43:08.967068+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:08.967068+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35796236,"num_objects":4370,"num_object_clones":0,"num_object_copies":17480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4371,"num_write_kb":34965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2238}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4409","reported_seq":4471,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:09.305171+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:09.305171+0000","last_peered":"2024-09-16T04:43:09.305171+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:09.305171+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4409,"log_dups_size":0,"ondisk_log_size":4409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36074764,"num_objects":4404,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4407,"num_write_kb":35244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2165}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":583829160,"num_objects":71273,"num_object_clones":0,"num_object_copies":285092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71300,"num_write_kb":570282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1425,"num_bytes_recovered":11436032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1031000064,"data_stored":1031000064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71327,"ondisk_log_size":71327,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514784,"kb_used_data":284556,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93857056,"statfs":{"total":96636764160,"available":96109625344,"internally_reserved":0,"allocated":291385344,"data_stored":291214534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235728591},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,2,3,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510380,"kb_used_data":284828,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":93861460,"statfs":{"total":96636764160,"available":96114135040,"internally_reserved":0,"allocated":291663872,"data_stored":291493062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":230947269},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,5,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051606,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318016,"kb_used_data":151664,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94053824,"statfs":{"total":96636764160,"available":96311115776,"internally_reserved":0,"allocated":155303936,"data_stored":155133126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170323663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519000,"kb_used_data":284680,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93852840,"statfs":{"total":96636764160,"available":96105308160,"internally_reserved":0,"allocated":291512320,"data_stored":291341510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":239922894},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,8,7,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292392960,"data_stored":292392960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155103232,"data_stored":155103232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":291860480,"data_stored":291860480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":291643392,"data_stored":291643392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:13.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:43:13.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4448", 'reported_seq': 4486, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:08.757851+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:43:08.757851+0000', 'last_peered': '2024-09-16T04:43:08.757851+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:43:08.757851+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4448, 'log_dups_size': 0, 'ondisk_log_size': 4448, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36438016, 'num_objects': 4448, 'num_object_clones': 0, 'num_object_copies': 17792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4448, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4448, 'num_write_kb': 35584, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2229}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:13.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:43:13.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4522", 'reported_seq': 4584, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:09.219397+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:43:09.219397+0000', 'last_peered': '2024-09-16T04:43:09.219397+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:43:09.219397+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4522, 'log_dups_size': 0, 'ondisk_log_size': 4522, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37008652, 'num_objects': 4518, 'num_object_clones': 0, 'num_object_copies': 18072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4520, 'num_write_kb': 36156, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2307}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:13.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:43:13.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4493", 'reported_seq': 4564, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:08.952182+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:43:08.952182+0000', 'last_peered': '2024-09-16T04:43:08.952182+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:43:08.952182+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4493, 'log_dups_size': 0, 'ondisk_log_size': 4493, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36780678, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 17960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4492, 'num_write_kb': 35926, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2263}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:13.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:43:13.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4490", 'reported_seq': 4552, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:09.387932+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:43:09.387932+0000', 'last_peered': '2024-09-16T04:43:09.387932+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:43:09.387932+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36746508, 'num_objects': 4486, 'num_object_clones': 0, 'num_object_copies': 17944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4486, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4488, 'num_write_kb': 35893, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2257}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:13.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:43:13.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4463", 'reported_seq': 4515, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:08.770637+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:43:08.770637+0000', 'last_peered': '2024-09-16T04:43:08.770637+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:43:08.770637+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4463, 'log_dups_size': 0, 'ondisk_log_size': 4463, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36533516, 'num_objects': 4460, 'num_object_clones': 0, 'num_object_copies': 17840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4461, 'num_write_kb': 35685, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2177}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:13.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:43:13.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4367", 'reported_seq': 4407, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:08.741991+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:43:08.741991+0000', 'last_peered': '2024-09-16T04:43:08.741991+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:43:08.741991+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4367, 'log_dups_size': 0, 'ondisk_log_size': 4367, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35774464, 'num_objects': 4367, 'num_object_clones': 0, 'num_object_copies': 17468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4367, 'num_write_kb': 34936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2159}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:13.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:43:13.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4528", 'reported_seq': 4589, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:09.356546+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:43:09.356546+0000', 'last_peered': '2024-09-16T04:43:09.356546+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:43:09.356546+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4528, 'log_dups_size': 0, 'ondisk_log_size': 4528, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37041420, 'num_objects': 4522, 'num_object_clones': 0, 'num_object_copies': 18088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4526, 'num_write_kb': 36188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2296}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:13.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:43:13.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4471", 'reported_seq': 7328, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:09.454415+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:43:09.454415+0000', 'last_peered': '2024-09-16T04:43:09.454415+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:43:09.454415+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4471, 'log_dups_size': 0, 'ondisk_log_size': 4471, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36582668, 'num_objects': 4466, 'num_object_clones': 0, 'num_object_copies': 17864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 842, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4469, 'num_write_kb': 35740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1400, 'num_bytes_recovered': 11436032, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3624}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:13.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:43:13.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4537", 'reported_seq': 4590, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:08.765267+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:43:08.765267+0000', 'last_peered': '2024-09-16T04:43:08.765267+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:43:08.765267+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4537, 'log_dups_size': 0, 'ondisk_log_size': 4537, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37095960, 'num_objects': 4529, 'num_object_clones': 0, 'num_object_copies': 18116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4534, 'num_write_kb': 36249, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2189}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:13.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:43:13.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4463", 'reported_seq': 4537, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:08.799350+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:43:08.799350+0000', 'last_peered': '2024-09-16T04:43:08.799350+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:43:08.799350+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4463, 'log_dups_size': 0, 'ondisk_log_size': 4463, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36541708, 'num_objects': 4461, 'num_object_clones': 0, 'num_object_copies': 17844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4461, 'num_write_kb': 35686, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2261}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:13.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:43:13.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4412", 'reported_seq': 4464, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:08.705220+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:43:08.705220+0000', 'last_peered': '2024-09-16T04:43:08.705220+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:43:08.705220+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4412, 'log_dups_size': 0, 'ondisk_log_size': 4412, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36097938, 'num_objects': 4407, 'num_object_clones': 0, 'num_object_copies': 17628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4409, 'num_write_kb': 35260, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2218}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:13.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:43:13.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4406", 'reported_seq': 4440, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:08.777198+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:43:08.777198+0000', 'last_peered': '2024-09-16T04:43:08.777198+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:43:08.777198+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4406, 'log_dups_size': 0, 'ondisk_log_size': 4406, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36084358, 'num_objects': 4405, 'num_object_clones': 0, 'num_object_copies': 17620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4405, 'num_write_kb': 35239, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:13.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:43:13.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4433", 'reported_seq': 4502, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:08.925027+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:43:08.925027+0000', 'last_peered': '2024-09-16T04:43:08.925027+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:43:08.925027+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4433, 'log_dups_size': 0, 'ondisk_log_size': 4433, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36305542, 'num_objects': 4432, 'num_object_clones': 0, 'num_object_copies': 17728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4432, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4432, 'num_write_kb': 35455, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2207}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:13.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:43:13.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4512", 'reported_seq': 4561, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:08.679924+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:43:08.679924+0000', 'last_peered': '2024-09-16T04:43:08.679924+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:43:08.679924+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4512, 'log_dups_size': 0, 'ondisk_log_size': 4512, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36926732, 'num_objects': 4508, 'num_object_clones': 0, 'num_object_copies': 18032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4510, 'num_write_kb': 36076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2274}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:13.343 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:43:13.343 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4373", 'reported_seq': 4448, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:08.967068+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:43:08.967068+0000', 'last_peered': '2024-09-16T04:43:08.967068+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:43:08.967068+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4373, 'log_dups_size': 0, 'ondisk_log_size': 4373, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35796236, 'num_objects': 4370, 'num_object_clones': 0, 'num_object_copies': 17480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4370, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4371, 'num_write_kb': 34965, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2238}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:13.344 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:43:13.344 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4409", 'reported_seq': 4471, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:09.305171+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:43:09.305171+0000', 'last_peered': '2024-09-16T04:43:09.305171+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:43:09.305171+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4409, 'log_dups_size': 0, 'ondisk_log_size': 4409, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36074764, 'num_objects': 4404, 'num_object_clones': 0, 'num_object_copies': 17616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4407, 'num_write_kb': 35244, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2165}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:13.345 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:43:14.055 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:14.055 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34157/285092 objects degraded (11.981%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":206,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":71273,"data_bytes":583829160,"bytes_used":1906872320,"bytes_avail":384640184320,"bytes_total":386547056640,"degraded_objects":34157,"degraded_total":285092,"degraded_ratio":0.1198104471539012,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":89399,"recovering_keys_per_sec":0,"num_objects_recovered":131,"num_bytes_recovered":1073152,"num_keys_recovered":0,"write_bytes_sec":2151048,"read_op_per_sec":0,"write_op_per_sec":262},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:43:14.083 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:43:14.083 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:14.281 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:2024-09-16T04:43:14.280333+0000 min lat: 0.00269411 max lat: 1.73107 avg lat: 0.038434 2024-09-16T04:43:14.281 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T04:43:14.281 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 140 20 72864 72844 4.06384 2.24219 0.0638373 0.038434 2024-09-16T04:43:14.645 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:14.645 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:14.664 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-16T04:43:14.600463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596608680,"num_objects":72833,"num_object_clones":0,"num_object_copies":291332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72860,"num_write_kb":582762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1490,"num_bytes_recovered":11968512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72887,"ondisk_log_size":72887,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1911204,"kb_used_data":1024224,"kb_used_omap":14,"kb_used_meta":886897,"kb_avail":375576156,"statfs":{"total":386547056640,"available":384589983744,"internally_reserved":0,"allocated":1048805376,"data_stored":1048122136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":908183089},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,14,4,12,5,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38600704,"num_objects":4712,"num_object_clones":0,"num_object_copies":18848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4712,"num_write_kb":37696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":196,"num_bytes_recovered":1605632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003900"},"pg_stats":[{"pgid":"1.f","version":"31'4561","reported_seq":4601,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.777905+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:13.777905+0000","last_peered":"2024-09-16T04:43:13.777905+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:13.777905+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37363712,"num_objects":4561,"num_object_clones":0,"num_object_copies":18244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4561,"num_write_kb":36488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2342}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4621","reported_seq":4683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.421092+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:14.421092+0000","last_peered":"2024-09-16T04:43:14.421092+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:14.421092+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37819660,"num_objects":4617,"num_object_clones":0,"num_object_copies":18468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4619,"num_write_kb":36948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4610","reported_seq":4681,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.958448+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:13.958448+0000","last_peered":"2024-09-16T04:43:13.958448+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:13.958448+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4610,"log_dups_size":0,"ondisk_log_size":4610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37739142,"num_objects":4607,"num_object_clones":0,"num_object_copies":18428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4609,"num_write_kb":36862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2380}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4568","reported_seq":4630,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.404445+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:14.404445+0000","last_peered":"2024-09-16T04:43:14.404445+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:14.404445+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37385484,"num_objects":4564,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4566,"num_write_kb":36517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4571","reported_seq":4623,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.830631+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:13.830631+0000","last_peered":"2024-09-16T04:43:13.830631+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:13.830631+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4571,"log_dups_size":0,"ondisk_log_size":4571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37418252,"num_objects":4568,"num_object_clones":0,"num_object_copies":18272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4569,"num_write_kb":36549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2285}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4466","reported_seq":4505,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.847939+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:13.847939+0000","last_peered":"2024-09-16T04:43:13.847939+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:13.847939+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36585472,"num_objects":4466,"num_object_clones":0,"num_object_copies":17864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4466,"num_write_kb":35728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2258}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4630","reported_seq":4691,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.275554+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:14.275554+0000","last_peered":"2024-09-16T04:43:14.275554+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:14.275554+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37877004,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4628,"num_write_kb":37004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2398}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4563","reported_seq":7550,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.433399+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:14.433399+0000","last_peered":"2024-09-16T04:43:14.433399+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:14.433399+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4563,"log_dups_size":0,"ondisk_log_size":4563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37336332,"num_objects":4558,"num_object_clones":0,"num_object_copies":18232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4561,"num_write_kb":36476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":11968512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3781}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4621","reported_seq":4674,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.882244+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:13.882244+0000","last_peered":"2024-09-16T04:43:13.882244+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:13.882244+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37784088,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4618,"num_write_kb":36921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2273}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4551","reported_seq":4625,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.021916+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:14.021916+0000","last_peered":"2024-09-16T04:43:14.021916+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:14.021916+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4551,"log_dups_size":0,"ondisk_log_size":4551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37262604,"num_objects":4549,"num_object_clones":0,"num_object_copies":18196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4549,"num_write_kb":36390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2349}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4504","reported_seq":4558,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.830337+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:13.830337+0000","last_peered":"2024-09-16T04:43:13.830337+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:13.830337+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36851602,"num_objects":4499,"num_object_clones":0,"num_object_copies":17996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4501,"num_write_kb":35996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4503","reported_seq":4537,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.786672+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:13.786672+0000","last_peered":"2024-09-16T04:43:13.786672+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:13.786672+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36878982,"num_objects":4502,"num_object_clones":0,"num_object_copies":18008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4502,"num_write_kb":36015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2374}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4523","reported_seq":4593,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.030448+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:14.030448+0000","last_peered":"2024-09-16T04:43:14.030448+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:14.030448+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4523,"log_dups_size":0,"ondisk_log_size":4523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37042822,"num_objects":4522,"num_object_clones":0,"num_object_copies":18088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4522,"num_write_kb":36175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2297}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4616","reported_seq":4666,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.840187+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:13.840187+0000","last_peered":"2024-09-16T04:43:13.840187+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:13.840187+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37778700,"num_objects":4612,"num_object_clones":0,"num_object_copies":18448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4614,"num_write_kb":36908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2378}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4480","reported_seq":4555,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.044211+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:14.044211+0000","last_peered":"2024-09-16T04:43:14.044211+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:14.044211+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36672780,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4478,"num_write_kb":35821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2345}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4499","reported_seq":4561,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.448565+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:14.448565+0000","last_peered":"2024-09-16T04:43:14.448565+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:14.448565+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36812044,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4497,"num_write_kb":35964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2255}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":596608680,"num_objects":72833,"num_object_clones":0,"num_object_copies":291332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72860,"num_write_kb":582762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1490,"num_bytes_recovered":11968512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1050030080,"data_stored":1050030080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72887,"ondisk_log_size":72887,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525144,"kb_used_data":290760,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93846696,"statfs":{"total":96636764160,"available":96099016704,"internally_reserved":0,"allocated":297738240,"data_stored":297567430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239988431},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,0,1,0,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538716,"kb_used_data":290956,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93833124,"statfs":{"total":96636764160,"available":96085118976,"internally_reserved":0,"allocated":297938944,"data_stored":297768134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051606,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318016,"kb_used_data":151664,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94053824,"statfs":{"total":96636764160,"available":96311115776,"internally_reserved":0,"allocated":155303936,"data_stored":155133126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170323663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529328,"kb_used_data":290844,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93842512,"statfs":{"total":96636764160,"available":96094732288,"internally_reserved":0,"allocated":297824256,"data_stored":297653446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":244182734},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":298688512,"data_stored":298688512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155103232,"data_stored":155103232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298254336,"data_stored":298254336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":297984000,"data_stored":297984000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:15.281 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 141 20 73170 73150 4.05197 2.39062 0.0608208 0.0385483 2024-09-16T04:43:16.281 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 142 20 73493 73473 4.04121 2.52344 0.0421207 0.0386509 2024-09-16T04:43:17.283 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 143 20 73814 73794 4.03043 2.50781 0.0350078 0.0387564 2024-09-16T04:43:17.666 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:18.283 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 144 20 74107 74087 4.01833 2.28906 0.0502286 0.0388671 2024-09-16T04:43:18.287 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:18.287 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:18.321 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-16T04:43:16.601006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596608680,"num_objects":72833,"num_object_clones":0,"num_object_copies":291332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72860,"num_write_kb":582762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1490,"num_bytes_recovered":11968512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72887,"ondisk_log_size":72887,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1903732,"kb_used_data":1029232,"kb_used_omap":14,"kb_used_meta":874417,"kb_avail":375583628,"statfs":{"total":386547056640,"available":384597635072,"internally_reserved":0,"allocated":1053933568,"data_stored":1053250328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":895403569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,10,5,10,5,15,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25444352,"num_objects":3106,"num_object_clones":0,"num_object_copies":12424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3106,"num_write_kb":24848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003947"},"pg_stats":[{"pgid":"1.f","version":"31'4561","reported_seq":4601,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.777905+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:13.777905+0000","last_peered":"2024-09-16T04:43:13.777905+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:13.777905+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37363712,"num_objects":4561,"num_object_clones":0,"num_object_copies":18244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4561,"num_write_kb":36488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2342}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4621","reported_seq":4683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.421092+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:14.421092+0000","last_peered":"2024-09-16T04:43:14.421092+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:14.421092+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37819660,"num_objects":4617,"num_object_clones":0,"num_object_copies":18468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4619,"num_write_kb":36948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4610","reported_seq":4681,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.958448+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:13.958448+0000","last_peered":"2024-09-16T04:43:13.958448+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:13.958448+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4610,"log_dups_size":0,"ondisk_log_size":4610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37739142,"num_objects":4607,"num_object_clones":0,"num_object_copies":18428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4609,"num_write_kb":36862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2380}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4568","reported_seq":4630,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.404445+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:14.404445+0000","last_peered":"2024-09-16T04:43:14.404445+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:14.404445+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37385484,"num_objects":4564,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4566,"num_write_kb":36517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4571","reported_seq":4623,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.830631+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:13.830631+0000","last_peered":"2024-09-16T04:43:13.830631+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:13.830631+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4571,"log_dups_size":0,"ondisk_log_size":4571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37418252,"num_objects":4568,"num_object_clones":0,"num_object_copies":18272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4569,"num_write_kb":36549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2285}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4466","reported_seq":4505,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.847939+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:13.847939+0000","last_peered":"2024-09-16T04:43:13.847939+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:13.847939+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36585472,"num_objects":4466,"num_object_clones":0,"num_object_copies":17864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4466,"num_write_kb":35728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2258}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4630","reported_seq":4691,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.275554+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:14.275554+0000","last_peered":"2024-09-16T04:43:14.275554+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:14.275554+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37877004,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4628,"num_write_kb":37004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2398}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4563","reported_seq":7550,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.433399+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:14.433399+0000","last_peered":"2024-09-16T04:43:14.433399+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:14.433399+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4563,"log_dups_size":0,"ondisk_log_size":4563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37336332,"num_objects":4558,"num_object_clones":0,"num_object_copies":18232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4561,"num_write_kb":36476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":11968512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3781}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4621","reported_seq":4674,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.882244+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:13.882244+0000","last_peered":"2024-09-16T04:43:13.882244+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:13.882244+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37784088,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4618,"num_write_kb":36921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2273}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4551","reported_seq":4625,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.021916+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:14.021916+0000","last_peered":"2024-09-16T04:43:14.021916+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:14.021916+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4551,"log_dups_size":0,"ondisk_log_size":4551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37262604,"num_objects":4549,"num_object_clones":0,"num_object_copies":18196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4549,"num_write_kb":36390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2349}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4504","reported_seq":4558,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.830337+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:13.830337+0000","last_peered":"2024-09-16T04:43:13.830337+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:13.830337+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36851602,"num_objects":4499,"num_object_clones":0,"num_object_copies":17996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4501,"num_write_kb":35996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4503","reported_seq":4537,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.786672+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:13.786672+0000","last_peered":"2024-09-16T04:43:13.786672+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:13.786672+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36878982,"num_objects":4502,"num_object_clones":0,"num_object_copies":18008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4502,"num_write_kb":36015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2374}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4523","reported_seq":4593,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.030448+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:14.030448+0000","last_peered":"2024-09-16T04:43:14.030448+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:14.030448+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4523,"log_dups_size":0,"ondisk_log_size":4523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37042822,"num_objects":4522,"num_object_clones":0,"num_object_copies":18088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4522,"num_write_kb":36175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2297}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4616","reported_seq":4666,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.840187+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:13.840187+0000","last_peered":"2024-09-16T04:43:13.840187+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:13.840187+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37778700,"num_objects":4612,"num_object_clones":0,"num_object_copies":18448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4614,"num_write_kb":36908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2378}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4480","reported_seq":4555,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.044211+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:14.044211+0000","last_peered":"2024-09-16T04:43:14.044211+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:14.044211+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36672780,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4478,"num_write_kb":35821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2345}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4499","reported_seq":4561,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.448565+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:14.448565+0000","last_peered":"2024-09-16T04:43:14.448565+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:14.448565+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36812044,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4497,"num_write_kb":35964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2255}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":596608680,"num_objects":72833,"num_object_clones":0,"num_object_copies":291332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72860,"num_write_kb":582762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1490,"num_bytes_recovered":11968512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1056538624,"data_stored":1056538624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72887,"ondisk_log_size":72887,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525144,"kb_used_data":290760,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93846696,"statfs":{"total":96636764160,"available":96099016704,"internally_reserved":0,"allocated":297738240,"data_stored":297567430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239988431},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,0,1,0,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538716,"kb_used_data":290956,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93833124,"statfs":{"total":96636764160,"available":96085118976,"internally_reserved":0,"allocated":297938944,"data_stored":297768134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051607,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310544,"kb_used_data":156672,"kb_used_omap":4,"kb_used_meta":153851,"kb_avail":94061296,"statfs":{"total":96636764160,"available":96318767104,"internally_reserved":0,"allocated":160432128,"data_stored":160261318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":157544143},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529328,"kb_used_data":290844,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93842512,"statfs":{"total":96636764160,"available":96094732288,"internally_reserved":0,"allocated":297824256,"data_stored":297653446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":244182734},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":298688512,"data_stored":298688512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161611776,"data_stored":161611776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298254336,"data_stored":298254336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":297984000,"data_stored":297984000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:18.322 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:18.917 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:18.917 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:18.946 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-09-16T04:43:18.601500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596608680,"num_objects":72833,"num_object_clones":0,"num_object_copies":291332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72860,"num_write_kb":582762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1490,"num_bytes_recovered":11968512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72887,"ondisk_log_size":72887,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1903732,"kb_used_data":1029232,"kb_used_omap":14,"kb_used_meta":874417,"kb_avail":375583628,"statfs":{"total":386547056640,"available":384597635072,"internally_reserved":0,"allocated":1053933568,"data_stored":1053250328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":895403569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,10,5,10,5,15,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25444352,"num_objects":3106,"num_object_clones":0,"num_object_copies":12424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3106,"num_write_kb":24848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003913"},"pg_stats":[{"pgid":"1.f","version":"31'4561","reported_seq":4601,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.777905+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:13.777905+0000","last_peered":"2024-09-16T04:43:13.777905+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:13.777905+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37363712,"num_objects":4561,"num_object_clones":0,"num_object_copies":18244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4561,"num_write_kb":36488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2342}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4621","reported_seq":4683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.421092+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:14.421092+0000","last_peered":"2024-09-16T04:43:14.421092+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:14.421092+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37819660,"num_objects":4617,"num_object_clones":0,"num_object_copies":18468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4619,"num_write_kb":36948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4610","reported_seq":4681,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.958448+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:13.958448+0000","last_peered":"2024-09-16T04:43:13.958448+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:13.958448+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4610,"log_dups_size":0,"ondisk_log_size":4610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37739142,"num_objects":4607,"num_object_clones":0,"num_object_copies":18428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4609,"num_write_kb":36862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2380}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4568","reported_seq":4630,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.404445+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:14.404445+0000","last_peered":"2024-09-16T04:43:14.404445+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:14.404445+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37385484,"num_objects":4564,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4566,"num_write_kb":36517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4571","reported_seq":4623,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.830631+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:13.830631+0000","last_peered":"2024-09-16T04:43:13.830631+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:13.830631+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4571,"log_dups_size":0,"ondisk_log_size":4571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37418252,"num_objects":4568,"num_object_clones":0,"num_object_copies":18272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4569,"num_write_kb":36549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2285}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4466","reported_seq":4505,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.847939+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:13.847939+0000","last_peered":"2024-09-16T04:43:13.847939+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:13.847939+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36585472,"num_objects":4466,"num_object_clones":0,"num_object_copies":17864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4466,"num_write_kb":35728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2258}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4630","reported_seq":4691,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.275554+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:14.275554+0000","last_peered":"2024-09-16T04:43:14.275554+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:14.275554+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37877004,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4628,"num_write_kb":37004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2398}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4563","reported_seq":7550,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.433399+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:14.433399+0000","last_peered":"2024-09-16T04:43:14.433399+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:14.433399+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4563,"log_dups_size":0,"ondisk_log_size":4563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37336332,"num_objects":4558,"num_object_clones":0,"num_object_copies":18232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4561,"num_write_kb":36476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":11968512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3781}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4621","reported_seq":4674,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.882244+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:13.882244+0000","last_peered":"2024-09-16T04:43:13.882244+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:13.882244+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37784088,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4618,"num_write_kb":36921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2273}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4551","reported_seq":4625,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.021916+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:14.021916+0000","last_peered":"2024-09-16T04:43:14.021916+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:14.021916+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4551,"log_dups_size":0,"ondisk_log_size":4551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37262604,"num_objects":4549,"num_object_clones":0,"num_object_copies":18196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4549,"num_write_kb":36390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2349}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4504","reported_seq":4558,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.830337+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:13.830337+0000","last_peered":"2024-09-16T04:43:13.830337+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:13.830337+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36851602,"num_objects":4499,"num_object_clones":0,"num_object_copies":17996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4501,"num_write_kb":35996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4503","reported_seq":4537,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.786672+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:13.786672+0000","last_peered":"2024-09-16T04:43:13.786672+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:13.786672+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36878982,"num_objects":4502,"num_object_clones":0,"num_object_copies":18008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4502,"num_write_kb":36015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2374}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4523","reported_seq":4593,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.030448+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:14.030448+0000","last_peered":"2024-09-16T04:43:14.030448+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:14.030448+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4523,"log_dups_size":0,"ondisk_log_size":4523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37042822,"num_objects":4522,"num_object_clones":0,"num_object_copies":18088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4522,"num_write_kb":36175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2297}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4616","reported_seq":4666,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.840187+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:13.840187+0000","last_peered":"2024-09-16T04:43:13.840187+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:13.840187+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37778700,"num_objects":4612,"num_object_clones":0,"num_object_copies":18448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4614,"num_write_kb":36908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2378}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4480","reported_seq":4555,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.044211+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:14.044211+0000","last_peered":"2024-09-16T04:43:14.044211+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:14.044211+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36672780,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4478,"num_write_kb":35821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2345}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4499","reported_seq":4561,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.448565+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:14.448565+0000","last_peered":"2024-09-16T04:43:14.448565+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:14.448565+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36812044,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4497,"num_write_kb":35964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2255}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":596608680,"num_objects":72833,"num_object_clones":0,"num_object_copies":291332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72860,"num_write_kb":582762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1490,"num_bytes_recovered":11968512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1056538624,"data_stored":1056538624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72887,"ondisk_log_size":72887,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525144,"kb_used_data":290760,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93846696,"statfs":{"total":96636764160,"available":96099016704,"internally_reserved":0,"allocated":297738240,"data_stored":297567430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239988431},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,0,1,0,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538716,"kb_used_data":290956,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93833124,"statfs":{"total":96636764160,"available":96085118976,"internally_reserved":0,"allocated":297938944,"data_stored":297768134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051607,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310544,"kb_used_data":156672,"kb_used_omap":4,"kb_used_meta":153851,"kb_avail":94061296,"statfs":{"total":96636764160,"available":96318767104,"internally_reserved":0,"allocated":160432128,"data_stored":160261318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":157544143},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529328,"kb_used_data":290844,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93842512,"statfs":{"total":96636764160,"available":96094732288,"internally_reserved":0,"allocated":297824256,"data_stored":297653446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":244182734},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":298688512,"data_stored":298688512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161611776,"data_stored":161611776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298254336,"data_stored":298254336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":297984000,"data_stored":297984000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:18.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:43:18.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4561", 'reported_seq': 4601, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:13.777905+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:43:13.777905+0000', 'last_peered': '2024-09-16T04:43:13.777905+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:43:13.777905+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4561, 'log_dups_size': 0, 'ondisk_log_size': 4561, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37363712, 'num_objects': 4561, 'num_object_clones': 0, 'num_object_copies': 18244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4561, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4561, 'num_write_kb': 36488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2342}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:18.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:43:18.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4621", 'reported_seq': 4683, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:14.421092+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:43:14.421092+0000', 'last_peered': '2024-09-16T04:43:14.421092+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:43:14.421092+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4621, 'log_dups_size': 0, 'ondisk_log_size': 4621, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37819660, 'num_objects': 4617, 'num_object_clones': 0, 'num_object_copies': 18468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4617, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4619, 'num_write_kb': 36948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2406}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:18.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:43:18.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4610", 'reported_seq': 4681, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:13.958448+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:43:13.958448+0000', 'last_peered': '2024-09-16T04:43:13.958448+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:43:13.958448+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4610, 'log_dups_size': 0, 'ondisk_log_size': 4610, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37739142, 'num_objects': 4607, 'num_object_clones': 0, 'num_object_copies': 18428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4607, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4609, 'num_write_kb': 36862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2380}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:18.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:43:18.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4568", 'reported_seq': 4630, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:14.404445+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:43:14.404445+0000', 'last_peered': '2024-09-16T04:43:14.404445+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:43:14.404445+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4568, 'log_dups_size': 0, 'ondisk_log_size': 4568, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37385484, 'num_objects': 4564, 'num_object_clones': 0, 'num_object_copies': 18256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4566, 'num_write_kb': 36517, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2335}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:18.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:43:18.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4571", 'reported_seq': 4623, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:13.830631+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:43:13.830631+0000', 'last_peered': '2024-09-16T04:43:13.830631+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:43:13.830631+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4571, 'log_dups_size': 0, 'ondisk_log_size': 4571, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37418252, 'num_objects': 4568, 'num_object_clones': 0, 'num_object_copies': 18272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4568, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4569, 'num_write_kb': 36549, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2285}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:18.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:43:18.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4466", 'reported_seq': 4505, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:13.847939+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:43:13.847939+0000', 'last_peered': '2024-09-16T04:43:13.847939+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:43:13.847939+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4466, 'log_dups_size': 0, 'ondisk_log_size': 4466, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36585472, 'num_objects': 4466, 'num_object_clones': 0, 'num_object_copies': 17864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4466, 'num_write_kb': 35728, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2258}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:18.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:43:18.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4630", 'reported_seq': 4691, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:14.275554+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:43:14.275554+0000', 'last_peered': '2024-09-16T04:43:14.275554+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:43:14.275554+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4630, 'log_dups_size': 0, 'ondisk_log_size': 4630, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37877004, 'num_objects': 4624, 'num_object_clones': 0, 'num_object_copies': 18496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4628, 'num_write_kb': 37004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2398}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:18.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:43:18.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4563", 'reported_seq': 7550, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:14.433399+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:43:14.433399+0000', 'last_peered': '2024-09-16T04:43:14.433399+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:43:14.433399+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4563, 'log_dups_size': 0, 'ondisk_log_size': 4563, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37336332, 'num_objects': 4558, 'num_object_clones': 0, 'num_object_copies': 18232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 777, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4561, 'num_write_kb': 36476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1465, 'num_bytes_recovered': 11968512, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3781}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:18.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:43:18.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4621", 'reported_seq': 4674, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:13.882244+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:43:13.882244+0000', 'last_peered': '2024-09-16T04:43:13.882244+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:43:13.882244+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4621, 'log_dups_size': 0, 'ondisk_log_size': 4621, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37784088, 'num_objects': 4613, 'num_object_clones': 0, 'num_object_copies': 18452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4618, 'num_write_kb': 36921, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2273}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:18.949 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:43:18.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4551", 'reported_seq': 4625, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:14.021916+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:43:14.021916+0000', 'last_peered': '2024-09-16T04:43:14.021916+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:43:14.021916+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4551, 'log_dups_size': 0, 'ondisk_log_size': 4551, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37262604, 'num_objects': 4549, 'num_object_clones': 0, 'num_object_copies': 18196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4549, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4549, 'num_write_kb': 36390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2349}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:18.949 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:43:18.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4504", 'reported_seq': 4558, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:13.830337+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:43:13.830337+0000', 'last_peered': '2024-09-16T04:43:13.830337+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:43:13.830337+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4504, 'log_dups_size': 0, 'ondisk_log_size': 4504, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36851602, 'num_objects': 4499, 'num_object_clones': 0, 'num_object_copies': 17996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4499, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4501, 'num_write_kb': 35996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2310}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:18.949 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:43:18.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4503", 'reported_seq': 4537, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:13.786672+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:43:13.786672+0000', 'last_peered': '2024-09-16T04:43:13.786672+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:43:13.786672+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36878982, 'num_objects': 4502, 'num_object_clones': 0, 'num_object_copies': 18008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4502, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4502, 'num_write_kb': 36015, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2374}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:18.950 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:43:18.950 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4523", 'reported_seq': 4593, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:14.030448+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:43:14.030448+0000', 'last_peered': '2024-09-16T04:43:14.030448+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:43:14.030448+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4523, 'log_dups_size': 0, 'ondisk_log_size': 4523, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37042822, 'num_objects': 4522, 'num_object_clones': 0, 'num_object_copies': 18088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4522, 'num_write_kb': 36175, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2297}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:18.950 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:43:18.950 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4616", 'reported_seq': 4666, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:13.840187+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:43:13.840187+0000', 'last_peered': '2024-09-16T04:43:13.840187+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:43:13.840187+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4616, 'log_dups_size': 0, 'ondisk_log_size': 4616, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37778700, 'num_objects': 4612, 'num_object_clones': 0, 'num_object_copies': 18448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4612, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4614, 'num_write_kb': 36908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2378}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:18.950 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:43:18.950 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4480", 'reported_seq': 4555, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:14.044211+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:43:14.044211+0000', 'last_peered': '2024-09-16T04:43:14.044211+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:43:14.044211+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4480, 'log_dups_size': 0, 'ondisk_log_size': 4480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36672780, 'num_objects': 4477, 'num_object_clones': 0, 'num_object_copies': 17908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4478, 'num_write_kb': 35821, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2345}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:18.950 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:43:18.950 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4499", 'reported_seq': 4561, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:14.448565+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:43:14.448565+0000', 'last_peered': '2024-09-16T04:43:14.448565+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:43:14.448565+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4499, 'log_dups_size': 0, 'ondisk_log_size': 4499, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36812044, 'num_objects': 4494, 'num_object_clones': 0, 'num_object_copies': 17976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4497, 'num_write_kb': 35964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:18.951 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:43:19.283 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 145 20 74392 74372 4.00598 2.22656 0.0651359 0.0389919 2024-09-16T04:43:19.662 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:19.662 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34092/291332 objects degraded (11.702%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":211,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":72833,"data_bytes":596608680,"bytes_used":1949421568,"bytes_avail":384597635072,"bytes_total":386547056640,"degraded_objects":34092,"degraded_total":291332,"degraded_ratio":0.11702113053148984,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":92812,"recovering_keys_per_sec":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"write_bytes_sec":2119671,"read_op_per_sec":0,"write_op_per_sec":258},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:43:19.684 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:43:19.684 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:20.284 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 146 20 74674 74654 3.99363 2.20312 0.0673762 0.0391068 2024-09-16T04:43:20.331 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:20.331 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:20.363 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-09-16T04:43:18.601500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596608680,"num_objects":72833,"num_object_clones":0,"num_object_copies":291332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72860,"num_write_kb":582762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1490,"num_bytes_recovered":11968512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72887,"ondisk_log_size":72887,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1903732,"kb_used_data":1029232,"kb_used_omap":14,"kb_used_meta":874417,"kb_avail":375583628,"statfs":{"total":386547056640,"available":384597635072,"internally_reserved":0,"allocated":1053933568,"data_stored":1053250328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":895403569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,10,5,10,5,15,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25444352,"num_objects":3106,"num_object_clones":0,"num_object_copies":12424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3106,"num_write_kb":24848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003913"},"pg_stats":[{"pgid":"1.f","version":"31'4561","reported_seq":4601,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.777905+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:13.777905+0000","last_peered":"2024-09-16T04:43:13.777905+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:13.777905+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37363712,"num_objects":4561,"num_object_clones":0,"num_object_copies":18244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4561,"num_write_kb":36488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2342}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4621","reported_seq":4683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.421092+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:14.421092+0000","last_peered":"2024-09-16T04:43:14.421092+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:14.421092+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37819660,"num_objects":4617,"num_object_clones":0,"num_object_copies":18468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4619,"num_write_kb":36948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4610","reported_seq":4681,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.958448+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:13.958448+0000","last_peered":"2024-09-16T04:43:13.958448+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:13.958448+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4610,"log_dups_size":0,"ondisk_log_size":4610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37739142,"num_objects":4607,"num_object_clones":0,"num_object_copies":18428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4609,"num_write_kb":36862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2380}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4568","reported_seq":4630,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.404445+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:14.404445+0000","last_peered":"2024-09-16T04:43:14.404445+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:14.404445+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37385484,"num_objects":4564,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4566,"num_write_kb":36517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4571","reported_seq":4623,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.830631+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:13.830631+0000","last_peered":"2024-09-16T04:43:13.830631+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:13.830631+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4571,"log_dups_size":0,"ondisk_log_size":4571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37418252,"num_objects":4568,"num_object_clones":0,"num_object_copies":18272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4569,"num_write_kb":36549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2285}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4466","reported_seq":4505,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.847939+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:13.847939+0000","last_peered":"2024-09-16T04:43:13.847939+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:13.847939+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36585472,"num_objects":4466,"num_object_clones":0,"num_object_copies":17864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4466,"num_write_kb":35728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2258}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4630","reported_seq":4691,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.275554+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:14.275554+0000","last_peered":"2024-09-16T04:43:14.275554+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:14.275554+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37877004,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4628,"num_write_kb":37004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2398}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4563","reported_seq":7550,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.433399+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:14.433399+0000","last_peered":"2024-09-16T04:43:14.433399+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:14.433399+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4563,"log_dups_size":0,"ondisk_log_size":4563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37336332,"num_objects":4558,"num_object_clones":0,"num_object_copies":18232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4561,"num_write_kb":36476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":11968512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3781}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4621","reported_seq":4674,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.882244+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:13.882244+0000","last_peered":"2024-09-16T04:43:13.882244+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:13.882244+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37784088,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4618,"num_write_kb":36921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2273}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4551","reported_seq":4625,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.021916+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:14.021916+0000","last_peered":"2024-09-16T04:43:14.021916+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:14.021916+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4551,"log_dups_size":0,"ondisk_log_size":4551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37262604,"num_objects":4549,"num_object_clones":0,"num_object_copies":18196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4549,"num_write_kb":36390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2349}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4504","reported_seq":4558,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.830337+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:13.830337+0000","last_peered":"2024-09-16T04:43:13.830337+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:13.830337+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36851602,"num_objects":4499,"num_object_clones":0,"num_object_copies":17996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4501,"num_write_kb":35996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4503","reported_seq":4537,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.786672+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:13.786672+0000","last_peered":"2024-09-16T04:43:13.786672+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:13.786672+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36878982,"num_objects":4502,"num_object_clones":0,"num_object_copies":18008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4502,"num_write_kb":36015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2374}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4523","reported_seq":4593,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.030448+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:14.030448+0000","last_peered":"2024-09-16T04:43:14.030448+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:14.030448+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4523,"log_dups_size":0,"ondisk_log_size":4523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37042822,"num_objects":4522,"num_object_clones":0,"num_object_copies":18088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4522,"num_write_kb":36175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2297}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4616","reported_seq":4666,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:13.840187+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:13.840187+0000","last_peered":"2024-09-16T04:43:13.840187+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:13.840187+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37778700,"num_objects":4612,"num_object_clones":0,"num_object_copies":18448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4614,"num_write_kb":36908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2378}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4480","reported_seq":4555,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.044211+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:14.044211+0000","last_peered":"2024-09-16T04:43:14.044211+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:14.044211+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36672780,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4478,"num_write_kb":35821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2345}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4499","reported_seq":4561,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:14.448565+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:14.448565+0000","last_peered":"2024-09-16T04:43:14.448565+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:14.448565+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36812044,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4497,"num_write_kb":35964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2255}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":596608680,"num_objects":72833,"num_object_clones":0,"num_object_copies":291332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72860,"num_write_kb":582762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1490,"num_bytes_recovered":11968512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1056538624,"data_stored":1056538624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72887,"ondisk_log_size":72887,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525144,"kb_used_data":290760,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93846696,"statfs":{"total":96636764160,"available":96099016704,"internally_reserved":0,"allocated":297738240,"data_stored":297567430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239988431},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,0,1,0,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538716,"kb_used_data":290956,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93833124,"statfs":{"total":96636764160,"available":96085118976,"internally_reserved":0,"allocated":297938944,"data_stored":297768134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051607,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310544,"kb_used_data":156672,"kb_used_omap":4,"kb_used_meta":153851,"kb_avail":94061296,"statfs":{"total":96636764160,"available":96318767104,"internally_reserved":0,"allocated":160432128,"data_stored":160261318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":157544143},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529328,"kb_used_data":290844,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93842512,"statfs":{"total":96636764160,"available":96094732288,"internally_reserved":0,"allocated":297824256,"data_stored":297653446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":244182734},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":298688512,"data_stored":298688512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161611776,"data_stored":161611776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298254336,"data_stored":298254336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":297984000,"data_stored":297984000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:21.284 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 147 20 74996 74976 3.98358 2.51562 0.0471535 0.0392111 2024-09-16T04:43:22.283 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 148 20 75316 75296 3.97355 2.5 0.0830001 0.0393102 2024-09-16T04:43:23.283 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 149 20 75644 75624 3.96408 2.5625 0.0337324 0.0394045 2024-09-16T04:43:23.366 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:23.956 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:23.956 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:23.992 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-09-16T04:43:22.604499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":609142440,"num_objects":74363,"num_object_clones":0,"num_object_copies":297452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74390,"num_write_kb":595002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1562,"num_bytes_recovered":12558336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74417,"ondisk_log_size":74417,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1971080,"kb_used_data":1056328,"kb_used_omap":14,"kb_used_meta":914673,"kb_avail":375516280,"statfs":{"total":386547056640,"available":384528670720,"internally_reserved":0,"allocated":1081679872,"data_stored":1080996632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":936625713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,11,12,14,10,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25313280,"num_objects":3090,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3090,"num_write_kb":24720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":1122304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005108"},"pg_stats":[{"pgid":"1.f","version":"31'4663","reported_seq":4703,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.807952+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:18.807952+0000","last_peered":"2024-09-16T04:43:18.807952+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:18.807952+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4663,"log_dups_size":0,"ondisk_log_size":4663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38199296,"num_objects":4663,"num_object_clones":0,"num_object_copies":18652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4663,"num_write_kb":37304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2444}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4706","reported_seq":4768,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:19.430941+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:19.430941+0000","last_peered":"2024-09-16T04:43:19.430941+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:19.430941+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4706,"log_dups_size":0,"ondisk_log_size":4706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38515980,"num_objects":4702,"num_object_clones":0,"num_object_copies":18808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4704,"num_write_kb":37628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2491}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4696","reported_seq":4767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.920125+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:18.920125+0000","last_peered":"2024-09-16T04:43:18.920125+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:18.920125+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38443654,"num_objects":4693,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":37550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2466}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4656","reported_seq":4718,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:19.468425+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:19.468425+0000","last_peered":"2024-09-16T04:43:19.468425+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:19.468425+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38106380,"num_objects":4652,"num_object_clones":0,"num_object_copies":18608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4654,"num_write_kb":37221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2423}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4665","reported_seq":4717,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.707232+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:18.707232+0000","last_peered":"2024-09-16T04:43:18.707232+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:18.707232+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4665,"log_dups_size":0,"ondisk_log_size":4665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38188300,"num_objects":4662,"num_object_clones":0,"num_object_copies":18648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4663,"num_write_kb":37301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2379}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4559","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.696765+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:18.696765+0000","last_peered":"2024-09-16T04:43:18.696765+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:18.696765+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4559,"log_dups_size":0,"ondisk_log_size":4559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37347328,"num_objects":4559,"num_object_clones":0,"num_object_copies":18236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4559,"num_write_kb":36472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4726","reported_seq":4787,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:19.414489+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:19.414489+0000","last_peered":"2024-09-16T04:43:19.414489+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:19.414489+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4726,"log_dups_size":0,"ondisk_log_size":4726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38663436,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4724,"num_write_kb":37772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2494}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4641","reported_seq":7768,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:19.490903+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:19.490903+0000","last_peered":"2024-09-16T04:43:19.490903+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:19.490903+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4641,"log_dups_size":0,"ondisk_log_size":4641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37975308,"num_objects":4636,"num_object_clones":0,"num_object_copies":18544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4639,"num_write_kb":37100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1537,"num_bytes_recovered":12558336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3931}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4724","reported_seq":4777,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.866040+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:18.866040+0000","last_peered":"2024-09-16T04:43:18.866040+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:18.866040+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4724,"log_dups_size":0,"ondisk_log_size":4724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38627864,"num_objects":4716,"num_object_clones":0,"num_object_copies":18864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4721,"num_write_kb":37745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2376}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4661","reported_seq":4735,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:19.037359+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:19.037359+0000","last_peered":"2024-09-16T04:43:19.037359+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:19.037359+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4661,"log_dups_size":0,"ondisk_log_size":4661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38163724,"num_objects":4659,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4659,"num_write_kb":37270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4612","reported_seq":4666,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.869850+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:18.869850+0000","last_peered":"2024-09-16T04:43:18.869850+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:18.869850+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4612,"log_dups_size":0,"ondisk_log_size":4612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37736338,"num_objects":4607,"num_object_clones":0,"num_object_copies":18428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4609,"num_write_kb":36860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2418}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4591","reported_seq":4627,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.838579+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:18.838579+0000","last_peered":"2024-09-16T04:43:18.838579+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:18.838579+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4591,"log_dups_size":0,"ondisk_log_size":4591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37599878,"num_objects":4590,"num_object_clones":0,"num_object_copies":18360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4590,"num_write_kb":36719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2462}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4614","reported_seq":4683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:19.062833+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:19.062833+0000","last_peered":"2024-09-16T04:43:19.062833+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:19.062833+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4614,"log_dups_size":0,"ondisk_log_size":4614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37788294,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4613,"num_write_kb":36903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2388}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4707","reported_seq":4757,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.737805+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:18.737805+0000","last_peered":"2024-09-16T04:43:18.737805+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:18.737805+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4707,"log_dups_size":0,"ondisk_log_size":4707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38524172,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4705,"num_write_kb":37636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2469}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4593","reported_seq":4668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.918394+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:18.918394+0000","last_peered":"2024-09-16T04:43:18.918394+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:18.918394+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37598476,"num_objects":4590,"num_object_clones":0,"num_object_copies":18360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4591,"num_write_kb":36725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2458}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4603","reported_seq":4665,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:19.493217+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:19.493217+0000","last_peered":"2024-09-16T04:43:19.493217+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:19.493217+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4603,"log_dups_size":0,"ondisk_log_size":4603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37664012,"num_objects":4598,"num_object_clones":0,"num_object_copies":18392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4601,"num_write_kb":36796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2359}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":609142440,"num_objects":74363,"num_object_clones":0,"num_object_copies":297452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74390,"num_write_kb":595002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1562,"num_bytes_recovered":12558336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1081806848,"data_stored":1081806848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74417,"ondisk_log_size":74417,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553464,"kb_used_data":296872,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93818376,"statfs":{"total":96636764160,"available":96070017024,"internally_reserved":0,"allocated":303996928,"data_stored":303826118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":262729423},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,5,8,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548968,"kb_used_data":297048,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93822872,"statfs":{"total":96636764160,"available":96074620928,"internally_reserved":0,"allocated":304177152,"data_stored":304006342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,0,3,3,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051608,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322392,"kb_used_data":164360,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94049448,"statfs":{"total":96636764160,"available":96306634752,"internally_reserved":0,"allocated":168304640,"data_stored":168133830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":161803983},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546256,"kb_used_data":298048,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93825584,"statfs":{"total":96636764160,"available":96077398016,"internally_reserved":0,"allocated":305201152,"data_stored":305030342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":254144206},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":304947200,"data_stored":304947200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168058880,"data_stored":168058880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304508928,"data_stored":304508928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":304291840,"data_stored":304291840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:23.993 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:24.284 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 150 20 75932 75912 3.95266 2.25 0.0466026 0.0395166 2024-09-16T04:43:24.575 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:24.575 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:24.599 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-09-16T04:43:22.604499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":609142440,"num_objects":74363,"num_object_clones":0,"num_object_copies":297452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74390,"num_write_kb":595002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1562,"num_bytes_recovered":12558336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74417,"ondisk_log_size":74417,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1971080,"kb_used_data":1056328,"kb_used_omap":14,"kb_used_meta":914673,"kb_avail":375516280,"statfs":{"total":386547056640,"available":384528670720,"internally_reserved":0,"allocated":1081679872,"data_stored":1080996632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":936625713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,11,12,14,10,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25313280,"num_objects":3090,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3090,"num_write_kb":24720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":1122304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005108"},"pg_stats":[{"pgid":"1.f","version":"31'4663","reported_seq":4703,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.807952+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:18.807952+0000","last_peered":"2024-09-16T04:43:18.807952+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:18.807952+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4663,"log_dups_size":0,"ondisk_log_size":4663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38199296,"num_objects":4663,"num_object_clones":0,"num_object_copies":18652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4663,"num_write_kb":37304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2444}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4706","reported_seq":4768,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:19.430941+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:19.430941+0000","last_peered":"2024-09-16T04:43:19.430941+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:19.430941+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4706,"log_dups_size":0,"ondisk_log_size":4706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38515980,"num_objects":4702,"num_object_clones":0,"num_object_copies":18808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4704,"num_write_kb":37628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2491}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4696","reported_seq":4767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.920125+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:18.920125+0000","last_peered":"2024-09-16T04:43:18.920125+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:18.920125+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38443654,"num_objects":4693,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":37550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2466}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4656","reported_seq":4718,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:19.468425+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:19.468425+0000","last_peered":"2024-09-16T04:43:19.468425+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:19.468425+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38106380,"num_objects":4652,"num_object_clones":0,"num_object_copies":18608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4654,"num_write_kb":37221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2423}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4665","reported_seq":4717,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.707232+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:18.707232+0000","last_peered":"2024-09-16T04:43:18.707232+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:18.707232+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4665,"log_dups_size":0,"ondisk_log_size":4665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38188300,"num_objects":4662,"num_object_clones":0,"num_object_copies":18648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4663,"num_write_kb":37301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2379}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4559","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.696765+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:18.696765+0000","last_peered":"2024-09-16T04:43:18.696765+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:18.696765+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4559,"log_dups_size":0,"ondisk_log_size":4559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37347328,"num_objects":4559,"num_object_clones":0,"num_object_copies":18236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4559,"num_write_kb":36472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4726","reported_seq":4787,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:19.414489+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:19.414489+0000","last_peered":"2024-09-16T04:43:19.414489+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:19.414489+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4726,"log_dups_size":0,"ondisk_log_size":4726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38663436,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4724,"num_write_kb":37772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2494}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4641","reported_seq":7768,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:19.490903+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:19.490903+0000","last_peered":"2024-09-16T04:43:19.490903+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:19.490903+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4641,"log_dups_size":0,"ondisk_log_size":4641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37975308,"num_objects":4636,"num_object_clones":0,"num_object_copies":18544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4639,"num_write_kb":37100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1537,"num_bytes_recovered":12558336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3931}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4724","reported_seq":4777,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.866040+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:18.866040+0000","last_peered":"2024-09-16T04:43:18.866040+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:18.866040+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4724,"log_dups_size":0,"ondisk_log_size":4724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38627864,"num_objects":4716,"num_object_clones":0,"num_object_copies":18864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4721,"num_write_kb":37745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2376}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4661","reported_seq":4735,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:19.037359+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:19.037359+0000","last_peered":"2024-09-16T04:43:19.037359+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:19.037359+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4661,"log_dups_size":0,"ondisk_log_size":4661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38163724,"num_objects":4659,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4659,"num_write_kb":37270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4612","reported_seq":4666,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.869850+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:18.869850+0000","last_peered":"2024-09-16T04:43:18.869850+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:18.869850+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4612,"log_dups_size":0,"ondisk_log_size":4612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37736338,"num_objects":4607,"num_object_clones":0,"num_object_copies":18428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4609,"num_write_kb":36860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2418}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4591","reported_seq":4627,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.838579+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:18.838579+0000","last_peered":"2024-09-16T04:43:18.838579+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:18.838579+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4591,"log_dups_size":0,"ondisk_log_size":4591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37599878,"num_objects":4590,"num_object_clones":0,"num_object_copies":18360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4590,"num_write_kb":36719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2462}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4614","reported_seq":4683,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:19.062833+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:19.062833+0000","last_peered":"2024-09-16T04:43:19.062833+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:19.062833+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4614,"log_dups_size":0,"ondisk_log_size":4614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37788294,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4613,"num_write_kb":36903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2388}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4707","reported_seq":4757,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.737805+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:18.737805+0000","last_peered":"2024-09-16T04:43:18.737805+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:18.737805+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4707,"log_dups_size":0,"ondisk_log_size":4707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38524172,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4705,"num_write_kb":37636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2469}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4593","reported_seq":4668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:18.918394+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:18.918394+0000","last_peered":"2024-09-16T04:43:18.918394+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:18.918394+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37598476,"num_objects":4590,"num_object_clones":0,"num_object_copies":18360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4591,"num_write_kb":36725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2458}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4603","reported_seq":4665,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:19.493217+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:19.493217+0000","last_peered":"2024-09-16T04:43:19.493217+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:19.493217+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4603,"log_dups_size":0,"ondisk_log_size":4603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37664012,"num_objects":4598,"num_object_clones":0,"num_object_copies":18392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4601,"num_write_kb":36796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2359}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":609142440,"num_objects":74363,"num_object_clones":0,"num_object_copies":297452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74390,"num_write_kb":595002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1562,"num_bytes_recovered":12558336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1081806848,"data_stored":1081806848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74417,"ondisk_log_size":74417,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553464,"kb_used_data":296872,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93818376,"statfs":{"total":96636764160,"available":96070017024,"internally_reserved":0,"allocated":303996928,"data_stored":303826118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":262729423},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,5,8,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548968,"kb_used_data":297048,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93822872,"statfs":{"total":96636764160,"available":96074620928,"internally_reserved":0,"allocated":304177152,"data_stored":304006342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,0,3,3,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051608,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322392,"kb_used_data":164360,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94049448,"statfs":{"total":96636764160,"available":96306634752,"internally_reserved":0,"allocated":168304640,"data_stored":168133830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":161803983},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546256,"kb_used_data":298048,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93825584,"statfs":{"total":96636764160,"available":96077398016,"internally_reserved":0,"allocated":305201152,"data_stored":305030342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":254144206},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":304947200,"data_stored":304947200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168058880,"data_stored":168058880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304508928,"data_stored":304508928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":304291840,"data_stored":304291840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:24.600 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:43:24.600 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4663", 'reported_seq': 4703, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:18.807952+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:43:18.807952+0000', 'last_peered': '2024-09-16T04:43:18.807952+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:43:18.807952+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4663, 'log_dups_size': 0, 'ondisk_log_size': 4663, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38199296, 'num_objects': 4663, 'num_object_clones': 0, 'num_object_copies': 18652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4663, 'num_write_kb': 37304, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2444}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:24.600 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:43:24.600 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4706", 'reported_seq': 4768, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:19.430941+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:43:19.430941+0000', 'last_peered': '2024-09-16T04:43:19.430941+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:43:19.430941+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4706, 'log_dups_size': 0, 'ondisk_log_size': 4706, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38515980, 'num_objects': 4702, 'num_object_clones': 0, 'num_object_copies': 18808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4702, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4704, 'num_write_kb': 37628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2491}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:24.601 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:43:24.601 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4696", 'reported_seq': 4767, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:18.920125+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:43:18.920125+0000', 'last_peered': '2024-09-16T04:43:18.920125+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:43:18.920125+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4696, 'log_dups_size': 0, 'ondisk_log_size': 4696, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38443654, 'num_objects': 4693, 'num_object_clones': 0, 'num_object_copies': 18772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4693, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4695, 'num_write_kb': 37550, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2466}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:24.601 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:43:24.601 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4656", 'reported_seq': 4718, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:19.468425+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:43:19.468425+0000', 'last_peered': '2024-09-16T04:43:19.468425+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:43:19.468425+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4656, 'log_dups_size': 0, 'ondisk_log_size': 4656, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38106380, 'num_objects': 4652, 'num_object_clones': 0, 'num_object_copies': 18608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4652, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4654, 'num_write_kb': 37221, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2423}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:24.601 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:43:24.601 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4665", 'reported_seq': 4717, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:18.707232+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:43:18.707232+0000', 'last_peered': '2024-09-16T04:43:18.707232+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:43:18.707232+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4665, 'log_dups_size': 0, 'ondisk_log_size': 4665, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38188300, 'num_objects': 4662, 'num_object_clones': 0, 'num_object_copies': 18648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4662, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4663, 'num_write_kb': 37301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2379}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:24.601 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:43:24.601 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4559", 'reported_seq': 4599, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:18.696765+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:43:18.696765+0000', 'last_peered': '2024-09-16T04:43:18.696765+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:43:18.696765+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4559, 'log_dups_size': 0, 'ondisk_log_size': 4559, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37347328, 'num_objects': 4559, 'num_object_clones': 0, 'num_object_copies': 18236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4559, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4559, 'num_write_kb': 36472, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2351}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:24.601 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:43:24.602 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4726", 'reported_seq': 4787, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:19.414489+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:43:19.414489+0000', 'last_peered': '2024-09-16T04:43:19.414489+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:43:19.414489+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4726, 'log_dups_size': 0, 'ondisk_log_size': 4726, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38663436, 'num_objects': 4720, 'num_object_clones': 0, 'num_object_copies': 18880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4724, 'num_write_kb': 37772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2494}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:24.602 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:43:24.602 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4641", 'reported_seq': 7768, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:19.490903+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:43:19.490903+0000', 'last_peered': '2024-09-16T04:43:19.490903+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:43:19.490903+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4641, 'log_dups_size': 0, 'ondisk_log_size': 4641, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37975308, 'num_objects': 4636, 'num_object_clones': 0, 'num_object_copies': 18544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 705, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4639, 'num_write_kb': 37100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1537, 'num_bytes_recovered': 12558336, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3931}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:24.602 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:43:24.602 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4724", 'reported_seq': 4777, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:18.866040+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:43:18.866040+0000', 'last_peered': '2024-09-16T04:43:18.866040+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:43:18.866040+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4724, 'log_dups_size': 0, 'ondisk_log_size': 4724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38627864, 'num_objects': 4716, 'num_object_clones': 0, 'num_object_copies': 18864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4716, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4721, 'num_write_kb': 37745, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2376}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:24.602 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:43:24.602 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4661", 'reported_seq': 4735, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:19.037359+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:43:19.037359+0000', 'last_peered': '2024-09-16T04:43:19.037359+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:43:19.037359+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4661, 'log_dups_size': 0, 'ondisk_log_size': 4661, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38163724, 'num_objects': 4659, 'num_object_clones': 0, 'num_object_copies': 18636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4659, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4659, 'num_write_kb': 37270, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2459}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:24.602 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:43:24.602 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4612", 'reported_seq': 4666, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:18.869850+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:43:18.869850+0000', 'last_peered': '2024-09-16T04:43:18.869850+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:43:18.869850+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4612, 'log_dups_size': 0, 'ondisk_log_size': 4612, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37736338, 'num_objects': 4607, 'num_object_clones': 0, 'num_object_copies': 18428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4607, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4609, 'num_write_kb': 36860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2418}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:24.602 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:43:24.602 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4591", 'reported_seq': 4627, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:18.838579+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:43:18.838579+0000', 'last_peered': '2024-09-16T04:43:18.838579+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:43:18.838579+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4591, 'log_dups_size': 0, 'ondisk_log_size': 4591, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37599878, 'num_objects': 4590, 'num_object_clones': 0, 'num_object_copies': 18360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4590, 'num_write_kb': 36719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2462}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:24.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:43:24.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4614", 'reported_seq': 4683, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:19.062833+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:43:19.062833+0000', 'last_peered': '2024-09-16T04:43:19.062833+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:43:19.062833+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4614, 'log_dups_size': 0, 'ondisk_log_size': 4614, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37788294, 'num_objects': 4613, 'num_object_clones': 0, 'num_object_copies': 18452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4613, 'num_write_kb': 36903, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2388}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:24.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:43:24.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4707", 'reported_seq': 4757, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:18.737805+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:43:18.737805+0000', 'last_peered': '2024-09-16T04:43:18.737805+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:43:18.737805+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4707, 'log_dups_size': 0, 'ondisk_log_size': 4707, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38524172, 'num_objects': 4703, 'num_object_clones': 0, 'num_object_copies': 18812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4703, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4705, 'num_write_kb': 37636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2469}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:24.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:43:24.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4593", 'reported_seq': 4668, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:18.918394+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:43:18.918394+0000', 'last_peered': '2024-09-16T04:43:18.918394+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:43:18.918394+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37598476, 'num_objects': 4590, 'num_object_clones': 0, 'num_object_copies': 18360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4591, 'num_write_kb': 36725, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2458}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:24.603 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:43:24.603 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4603", 'reported_seq': 4665, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:19.493217+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:43:19.493217+0000', 'last_peered': '2024-09-16T04:43:19.493217+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:43:19.493217+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4603, 'log_dups_size': 0, 'ondisk_log_size': 4603, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37664012, 'num_objects': 4598, 'num_object_clones': 0, 'num_object_copies': 18392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4601, 'num_write_kb': 36796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2359}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:24.604 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:43:25.277 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:25.277 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34020/297452 objects degraded (11.437%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":217,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":74363,"data_bytes":609142440,"bytes_used":2018385920,"bytes_avail":384528670720,"bytes_total":386547056640,"degraded_objects":34020,"degraded_total":297452,"degraded_ratio":0.11437139437623549,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":93485,"recovering_keys_per_sec":0,"num_objects_recovered":137,"num_bytes_recovered":1122304,"num_keys_recovered":0,"write_bytes_sec":2108542,"read_op_per_sec":0,"write_op_per_sec":257},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:43:25.284 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 151 20 76208 76188 3.94076 2.15625 0.0587616 0.0396342 2024-09-16T04:43:25.299 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:43:25.299 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:25.934 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:25.935 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:25.953 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-09-16T04:43:24.604980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622012072,"num_objects":75934,"num_object_clones":0,"num_object_copies":303736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75961,"num_write_kb":607570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1627,"num_bytes_recovered":13090816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75988,"ondisk_log_size":75988,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1985508,"kb_used_data":1074916,"kb_used_omap":14,"kb_used_meta":910513,"kb_avail":375501852,"statfs":{"total":386547056640,"available":384513896448,"internally_reserved":0,"allocated":1100713984,"data_stored":1100030744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":932365873},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,7,11,8,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38182912,"num_objects":4661,"num_object_clones":0,"num_object_copies":18644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4661,"num_write_kb":37288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":1654784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004981"},"pg_stats":[{"pgid":"1.f","version":"31'4765","reported_seq":4805,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.929501+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:23.929501+0000","last_peered":"2024-09-16T04:43:23.929501+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:23.929501+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4765,"log_dups_size":0,"ondisk_log_size":4765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39034880,"num_objects":4765,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4765,"num_write_kb":38120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2546}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4820","reported_seq":4882,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.514602+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:24.514602+0000","last_peered":"2024-09-16T04:43:24.514602+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:24.514602+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39449868,"num_objects":4816,"num_object_clones":0,"num_object_copies":19264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4818,"num_write_kb":38540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2605}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4795","reported_seq":4865,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.214873+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:24.214873+0000","last_peered":"2024-09-16T04:43:24.214873+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:24.214873+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4795,"log_dups_size":0,"ondisk_log_size":4795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39254662,"num_objects":4792,"num_object_clones":0,"num_object_copies":19168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4794,"num_write_kb":38342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2565}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4763","reported_seq":4825,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.281660+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:24.281660+0000","last_peered":"2024-09-16T04:43:24.281660+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:24.281660+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38982924,"num_objects":4759,"num_object_clones":0,"num_object_copies":19036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4761,"num_write_kb":38077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2530}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4761","reported_seq":4811,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.905663+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:23.905663+0000","last_peered":"2024-09-16T04:43:23.905663+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:23.905663+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4761,"log_dups_size":0,"ondisk_log_size":4761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38974732,"num_objects":4758,"num_object_clones":0,"num_object_copies":19032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4759,"num_write_kb":38069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2475}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4651","reported_seq":4691,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.975929+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:23.975929+0000","last_peered":"2024-09-16T04:43:23.975929+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:23.975929+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4651,"log_dups_size":0,"ondisk_log_size":4651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4651,"num_write_kb":37208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2443}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4825","reported_seq":4886,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.380858+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:24.380858+0000","last_peered":"2024-09-16T04:43:24.380858+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:24.380858+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39474444,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4823,"num_write_kb":38564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2593}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4735","reported_seq":7994,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.548339+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:24.548339+0000","last_peered":"2024-09-16T04:43:24.548339+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:24.548339+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38745356,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1602,"num_bytes_recovered":13090816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4090}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4827","reported_seq":4880,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.871234+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:23.871234+0000","last_peered":"2024-09-16T04:43:23.871234+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:23.871234+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4827,"log_dups_size":0,"ondisk_log_size":4827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39471640,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":38569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2479}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4763","reported_seq":4837,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.208958+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:24.208958+0000","last_peered":"2024-09-16T04:43:24.208958+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:24.208958+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38999308,"num_objects":4761,"num_object_clones":0,"num_object_copies":19044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4761,"num_write_kb":38086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2561}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4716","reported_seq":4768,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.936308+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:23.936308+0000","last_peered":"2024-09-16T04:43:23.936308+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:23.936308+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4716,"log_dups_size":0,"ondisk_log_size":4716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38588306,"num_objects":4711,"num_object_clones":0,"num_object_copies":18844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4713,"num_write_kb":37692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4665","reported_seq":4701,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.939615+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:23.939615+0000","last_peered":"2024-09-16T04:43:23.939615+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:23.939615+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4665,"log_dups_size":0,"ondisk_log_size":4665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38206086,"num_objects":4664,"num_object_clones":0,"num_object_copies":18656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4664,"num_write_kb":37311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2536}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4698","reported_seq":4767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.183329+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:24.183329+0000","last_peered":"2024-09-16T04:43:24.183329+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:24.183329+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38476422,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4697,"num_write_kb":37575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2472}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4805","reported_seq":4855,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.888867+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:23.888867+0000","last_peered":"2024-09-16T04:43:23.888867+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:23.888867+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4805,"log_dups_size":0,"ondisk_log_size":4805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39326988,"num_objects":4801,"num_object_clones":0,"num_object_copies":19204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4803,"num_write_kb":38420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2567}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4701","reported_seq":4776,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.127504+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:24.127504+0000","last_peered":"2024-09-16T04:43:24.127504+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:24.127504+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4701,"log_dups_size":0,"ondisk_log_size":4701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38483212,"num_objects":4698,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":37589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2566}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4698","reported_seq":4760,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.440863+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:24.440863+0000","last_peered":"2024-09-16T04:43:24.440863+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:24.440863+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38442252,"num_objects":4693,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4696,"num_write_kb":37556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2454}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":622012072,"num_objects":75934,"num_object_clones":0,"num_object_copies":303736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75961,"num_write_kb":607570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1627,"num_bytes_recovered":13090816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1101025280,"data_stored":1101025280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75988,"ondisk_log_size":75988,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547340,"kb_used_data":303228,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93824500,"statfs":{"total":96636764160,"available":96076288000,"internally_reserved":0,"allocated":310505472,"data_stored":310334662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249949903},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,2,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559336,"kb_used_data":303256,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93812504,"statfs":{"total":96636764160,"available":96064004096,"internally_reserved":0,"allocated":310534144,"data_stored":310363334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051608,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322392,"kb_used_data":164360,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94049448,"statfs":{"total":96636764160,"available":96306634752,"internally_reserved":0,"allocated":168304640,"data_stored":168133830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":161803983},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556440,"kb_used_data":304072,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93815400,"statfs":{"total":96636764160,"available":96066969600,"internally_reserved":0,"allocated":311369728,"data_stored":311198918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":258404046},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311312384,"data_stored":311312384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168058880,"data_stored":168058880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":310972416,"data_stored":310972416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310681600,"data_stored":310681600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:26.284 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 152 20 76512 76492 3.93046 2.375 0.0628948 0.039744 2024-09-16T04:43:27.284 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 153 20 76825 76805 3.92075 2.44531 0.0493521 0.0398428 2024-09-16T04:43:28.287 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 154 20 77136 77116 3.91107 2.42969 0.0477537 0.039936 2024-09-16T04:43:28.955 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:29.284 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 155 20 77435 77415 3.90091 2.33594 0.053605 0.0400391 2024-09-16T04:43:29.539 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:29.540 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:29.562 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-09-16T04:43:28.606032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622012072,"num_objects":75934,"num_object_clones":0,"num_object_copies":303736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75961,"num_write_kb":607570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1627,"num_bytes_recovered":13090816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75988,"ondisk_log_size":75988,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1994768,"kb_used_data":1080016,"kb_used_omap":14,"kb_used_meta":914673,"kb_avail":375492592,"statfs":{"total":386547056640,"available":384504414208,"internally_reserved":0,"allocated":1105936384,"data_stored":1105253144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":936625713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,7,11,5,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25403392,"num_objects":3101,"num_object_clones":0,"num_object_copies":12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3101,"num_write_kb":24808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":1122304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005025"},"pg_stats":[{"pgid":"1.f","version":"31'4765","reported_seq":4805,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.929501+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:23.929501+0000","last_peered":"2024-09-16T04:43:23.929501+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:23.929501+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4765,"log_dups_size":0,"ondisk_log_size":4765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39034880,"num_objects":4765,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4765,"num_write_kb":38120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2546}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4820","reported_seq":4882,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.514602+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:24.514602+0000","last_peered":"2024-09-16T04:43:24.514602+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:24.514602+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39449868,"num_objects":4816,"num_object_clones":0,"num_object_copies":19264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4818,"num_write_kb":38540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2605}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4795","reported_seq":4865,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.214873+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:24.214873+0000","last_peered":"2024-09-16T04:43:24.214873+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:24.214873+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4795,"log_dups_size":0,"ondisk_log_size":4795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39254662,"num_objects":4792,"num_object_clones":0,"num_object_copies":19168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4794,"num_write_kb":38342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2565}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4763","reported_seq":4825,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.281660+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:24.281660+0000","last_peered":"2024-09-16T04:43:24.281660+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:24.281660+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38982924,"num_objects":4759,"num_object_clones":0,"num_object_copies":19036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4761,"num_write_kb":38077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2530}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4761","reported_seq":4811,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.905663+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:23.905663+0000","last_peered":"2024-09-16T04:43:23.905663+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:23.905663+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4761,"log_dups_size":0,"ondisk_log_size":4761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38974732,"num_objects":4758,"num_object_clones":0,"num_object_copies":19032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4759,"num_write_kb":38069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2475}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4651","reported_seq":4691,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.975929+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:23.975929+0000","last_peered":"2024-09-16T04:43:23.975929+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:23.975929+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4651,"log_dups_size":0,"ondisk_log_size":4651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4651,"num_write_kb":37208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2443}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4825","reported_seq":4886,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.380858+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:24.380858+0000","last_peered":"2024-09-16T04:43:24.380858+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:24.380858+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39474444,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4823,"num_write_kb":38564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2593}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4735","reported_seq":7994,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.548339+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:24.548339+0000","last_peered":"2024-09-16T04:43:24.548339+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:24.548339+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38745356,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1602,"num_bytes_recovered":13090816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4090}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4827","reported_seq":4880,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.871234+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:23.871234+0000","last_peered":"2024-09-16T04:43:23.871234+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:23.871234+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4827,"log_dups_size":0,"ondisk_log_size":4827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39471640,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":38569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2479}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4763","reported_seq":4837,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.208958+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:24.208958+0000","last_peered":"2024-09-16T04:43:24.208958+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:24.208958+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38999308,"num_objects":4761,"num_object_clones":0,"num_object_copies":19044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4761,"num_write_kb":38086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2561}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4716","reported_seq":4768,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.936308+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:23.936308+0000","last_peered":"2024-09-16T04:43:23.936308+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:23.936308+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4716,"log_dups_size":0,"ondisk_log_size":4716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38588306,"num_objects":4711,"num_object_clones":0,"num_object_copies":18844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4713,"num_write_kb":37692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4665","reported_seq":4701,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.939615+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:23.939615+0000","last_peered":"2024-09-16T04:43:23.939615+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:23.939615+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4665,"log_dups_size":0,"ondisk_log_size":4665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38206086,"num_objects":4664,"num_object_clones":0,"num_object_copies":18656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4664,"num_write_kb":37311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2536}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4698","reported_seq":4767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.183329+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:24.183329+0000","last_peered":"2024-09-16T04:43:24.183329+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:24.183329+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38476422,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4697,"num_write_kb":37575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2472}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4805","reported_seq":4855,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.888867+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:23.888867+0000","last_peered":"2024-09-16T04:43:23.888867+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:23.888867+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4805,"log_dups_size":0,"ondisk_log_size":4805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39326988,"num_objects":4801,"num_object_clones":0,"num_object_copies":19204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4803,"num_write_kb":38420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2567}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4701","reported_seq":4776,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.127504+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:24.127504+0000","last_peered":"2024-09-16T04:43:24.127504+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:24.127504+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4701,"log_dups_size":0,"ondisk_log_size":4701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38483212,"num_objects":4698,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":37589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2566}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4698","reported_seq":4760,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.440863+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:24.440863+0000","last_peered":"2024-09-16T04:43:24.440863+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:24.440863+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38442252,"num_objects":4693,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4696,"num_write_kb":37556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2454}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":622012072,"num_objects":75934,"num_object_clones":0,"num_object_copies":303736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75961,"num_write_kb":607570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1627,"num_bytes_recovered":13090816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1107488768,"data_stored":1107488768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75988,"ondisk_log_size":75988,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547340,"kb_used_data":303228,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93824500,"statfs":{"total":96636764160,"available":96076288000,"internally_reserved":0,"allocated":310505472,"data_stored":310334662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249949903},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,2,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559336,"kb_used_data":303256,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93812504,"statfs":{"total":96636764160,"available":96064004096,"internally_reserved":0,"allocated":310534144,"data_stored":310363334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051609,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331652,"kb_used_data":169460,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94040188,"statfs":{"total":96636764160,"available":96297152512,"internally_reserved":0,"allocated":173527040,"data_stored":173356230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":166063823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556440,"kb_used_data":304072,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93815400,"statfs":{"total":96636764160,"available":96066969600,"internally_reserved":0,"allocated":311369728,"data_stored":311198918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":258404046},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311312384,"data_stored":311312384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":174522368,"data_stored":174522368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":310972416,"data_stored":310972416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310681600,"data_stored":310681600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:29.564 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:30.199 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:30.200 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:30.230 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-09-16T04:43:28.606032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622012072,"num_objects":75934,"num_object_clones":0,"num_object_copies":303736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75961,"num_write_kb":607570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1627,"num_bytes_recovered":13090816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75988,"ondisk_log_size":75988,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1994768,"kb_used_data":1080016,"kb_used_omap":14,"kb_used_meta":914673,"kb_avail":375492592,"statfs":{"total":386547056640,"available":384504414208,"internally_reserved":0,"allocated":1105936384,"data_stored":1105253144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":936625713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,7,11,5,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25403392,"num_objects":3101,"num_object_clones":0,"num_object_copies":12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3101,"num_write_kb":24808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":1122304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005025"},"pg_stats":[{"pgid":"1.f","version":"31'4765","reported_seq":4805,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.929501+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:23.929501+0000","last_peered":"2024-09-16T04:43:23.929501+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:23.929501+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4765,"log_dups_size":0,"ondisk_log_size":4765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39034880,"num_objects":4765,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4765,"num_write_kb":38120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2546}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4820","reported_seq":4882,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.514602+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:24.514602+0000","last_peered":"2024-09-16T04:43:24.514602+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:24.514602+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39449868,"num_objects":4816,"num_object_clones":0,"num_object_copies":19264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4818,"num_write_kb":38540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2605}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4795","reported_seq":4865,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.214873+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:24.214873+0000","last_peered":"2024-09-16T04:43:24.214873+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:24.214873+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4795,"log_dups_size":0,"ondisk_log_size":4795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39254662,"num_objects":4792,"num_object_clones":0,"num_object_copies":19168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4794,"num_write_kb":38342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2565}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4763","reported_seq":4825,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.281660+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:24.281660+0000","last_peered":"2024-09-16T04:43:24.281660+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:24.281660+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38982924,"num_objects":4759,"num_object_clones":0,"num_object_copies":19036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4761,"num_write_kb":38077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2530}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4761","reported_seq":4811,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.905663+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:23.905663+0000","last_peered":"2024-09-16T04:43:23.905663+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:23.905663+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4761,"log_dups_size":0,"ondisk_log_size":4761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38974732,"num_objects":4758,"num_object_clones":0,"num_object_copies":19032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4759,"num_write_kb":38069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2475}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4651","reported_seq":4691,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.975929+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:23.975929+0000","last_peered":"2024-09-16T04:43:23.975929+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:23.975929+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4651,"log_dups_size":0,"ondisk_log_size":4651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4651,"num_write_kb":37208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2443}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4825","reported_seq":4886,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.380858+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:24.380858+0000","last_peered":"2024-09-16T04:43:24.380858+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:24.380858+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39474444,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4823,"num_write_kb":38564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2593}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4735","reported_seq":7994,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.548339+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:24.548339+0000","last_peered":"2024-09-16T04:43:24.548339+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:24.548339+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38745356,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1602,"num_bytes_recovered":13090816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4090}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4827","reported_seq":4880,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.871234+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:23.871234+0000","last_peered":"2024-09-16T04:43:23.871234+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:23.871234+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4827,"log_dups_size":0,"ondisk_log_size":4827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39471640,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":38569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2479}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4763","reported_seq":4837,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.208958+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:24.208958+0000","last_peered":"2024-09-16T04:43:24.208958+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:24.208958+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38999308,"num_objects":4761,"num_object_clones":0,"num_object_copies":19044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4761,"num_write_kb":38086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2561}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4716","reported_seq":4768,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.936308+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:23.936308+0000","last_peered":"2024-09-16T04:43:23.936308+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:23.936308+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4716,"log_dups_size":0,"ondisk_log_size":4716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38588306,"num_objects":4711,"num_object_clones":0,"num_object_copies":18844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4713,"num_write_kb":37692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4665","reported_seq":4701,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.939615+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:23.939615+0000","last_peered":"2024-09-16T04:43:23.939615+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:23.939615+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4665,"log_dups_size":0,"ondisk_log_size":4665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38206086,"num_objects":4664,"num_object_clones":0,"num_object_copies":18656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4664,"num_write_kb":37311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2536}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4698","reported_seq":4767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.183329+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:24.183329+0000","last_peered":"2024-09-16T04:43:24.183329+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:24.183329+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38476422,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4697,"num_write_kb":37575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2472}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4805","reported_seq":4855,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:23.888867+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:23.888867+0000","last_peered":"2024-09-16T04:43:23.888867+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:23.888867+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4805,"log_dups_size":0,"ondisk_log_size":4805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39326988,"num_objects":4801,"num_object_clones":0,"num_object_copies":19204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4803,"num_write_kb":38420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2567}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4701","reported_seq":4776,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.127504+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:24.127504+0000","last_peered":"2024-09-16T04:43:24.127504+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:24.127504+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4701,"log_dups_size":0,"ondisk_log_size":4701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38483212,"num_objects":4698,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":37589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2566}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4698","reported_seq":4760,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:24.440863+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:24.440863+0000","last_peered":"2024-09-16T04:43:24.440863+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:24.440863+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38442252,"num_objects":4693,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4696,"num_write_kb":37556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2454}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":622012072,"num_objects":75934,"num_object_clones":0,"num_object_copies":303736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75961,"num_write_kb":607570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1627,"num_bytes_recovered":13090816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1107488768,"data_stored":1107488768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75988,"ondisk_log_size":75988,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547340,"kb_used_data":303228,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93824500,"statfs":{"total":96636764160,"available":96076288000,"internally_reserved":0,"allocated":310505472,"data_stored":310334662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249949903},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,2,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559336,"kb_used_data":303256,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93812504,"statfs":{"total":96636764160,"available":96064004096,"internally_reserved":0,"allocated":310534144,"data_stored":310363334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051609,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331652,"kb_used_data":169460,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94040188,"statfs":{"total":96636764160,"available":96297152512,"internally_reserved":0,"allocated":173527040,"data_stored":173356230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":166063823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556440,"kb_used_data":304072,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93815400,"statfs":{"total":96636764160,"available":96066969600,"internally_reserved":0,"allocated":311369728,"data_stored":311198918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":258404046},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311312384,"data_stored":311312384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":174522368,"data_stored":174522368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":310972416,"data_stored":310972416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310681600,"data_stored":310681600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:30.231 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:43:30.231 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4765", 'reported_seq': 4805, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:23.929501+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:43:23.929501+0000', 'last_peered': '2024-09-16T04:43:23.929501+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:43:23.929501+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4765, 'log_dups_size': 0, 'ondisk_log_size': 4765, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39034880, 'num_objects': 4765, 'num_object_clones': 0, 'num_object_copies': 19060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4765, 'num_write_kb': 38120, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2546}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:30.231 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:43:30.231 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4820", 'reported_seq': 4882, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:24.514602+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:43:24.514602+0000', 'last_peered': '2024-09-16T04:43:24.514602+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:43:24.514602+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4820, 'log_dups_size': 0, 'ondisk_log_size': 4820, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39449868, 'num_objects': 4816, 'num_object_clones': 0, 'num_object_copies': 19264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4816, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4818, 'num_write_kb': 38540, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2605}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:30.231 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:43:30.231 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4795", 'reported_seq': 4865, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:24.214873+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:43:24.214873+0000', 'last_peered': '2024-09-16T04:43:24.214873+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:43:24.214873+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4795, 'log_dups_size': 0, 'ondisk_log_size': 4795, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39254662, 'num_objects': 4792, 'num_object_clones': 0, 'num_object_copies': 19168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4792, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4794, 'num_write_kb': 38342, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2565}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:30.231 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:43:30.231 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4763", 'reported_seq': 4825, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:24.281660+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:43:24.281660+0000', 'last_peered': '2024-09-16T04:43:24.281660+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:43:24.281660+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4763, 'log_dups_size': 0, 'ondisk_log_size': 4763, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38982924, 'num_objects': 4759, 'num_object_clones': 0, 'num_object_copies': 19036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4761, 'num_write_kb': 38077, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2530}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:30.231 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:43:30.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4761", 'reported_seq': 4811, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:23.905663+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:43:23.905663+0000', 'last_peered': '2024-09-16T04:43:23.905663+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:43:23.905663+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4761, 'log_dups_size': 0, 'ondisk_log_size': 4761, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38974732, 'num_objects': 4758, 'num_object_clones': 0, 'num_object_copies': 19032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4758, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4759, 'num_write_kb': 38069, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2475}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:30.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:43:30.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4651", 'reported_seq': 4691, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:23.975929+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:43:23.975929+0000', 'last_peered': '2024-09-16T04:43:23.975929+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:43:23.975929+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4651, 'log_dups_size': 0, 'ondisk_log_size': 4651, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38100992, 'num_objects': 4651, 'num_object_clones': 0, 'num_object_copies': 18604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4651, 'num_write_kb': 37208, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2443}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:30.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:43:30.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4825", 'reported_seq': 4886, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:24.380858+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:43:24.380858+0000', 'last_peered': '2024-09-16T04:43:24.380858+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:43:24.380858+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4825, 'log_dups_size': 0, 'ondisk_log_size': 4825, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39474444, 'num_objects': 4819, 'num_object_clones': 0, 'num_object_copies': 19276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4823, 'num_write_kb': 38564, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2593}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:30.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:43:30.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4735", 'reported_seq': 7994, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:24.548339+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:43:24.548339+0000', 'last_peered': '2024-09-16T04:43:24.548339+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:43:24.548339+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4735, 'log_dups_size': 0, 'ondisk_log_size': 4735, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38745356, 'num_objects': 4730, 'num_object_clones': 0, 'num_object_copies': 18920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 640, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4733, 'num_write_kb': 37852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1602, 'num_bytes_recovered': 13090816, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4090}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:30.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:43:30.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4827", 'reported_seq': 4880, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:23.871234+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:43:23.871234+0000', 'last_peered': '2024-09-16T04:43:23.871234+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:43:23.871234+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4827, 'log_dups_size': 0, 'ondisk_log_size': 4827, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39471640, 'num_objects': 4819, 'num_object_clones': 0, 'num_object_copies': 19276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4824, 'num_write_kb': 38569, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2479}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:30.232 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:43:30.232 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4763", 'reported_seq': 4837, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:24.208958+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:43:24.208958+0000', 'last_peered': '2024-09-16T04:43:24.208958+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:43:24.208958+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4763, 'log_dups_size': 0, 'ondisk_log_size': 4763, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38999308, 'num_objects': 4761, 'num_object_clones': 0, 'num_object_copies': 19044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4761, 'num_write_kb': 38086, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2561}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:30.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:43:30.233 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4716", 'reported_seq': 4768, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:23.936308+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:43:23.936308+0000', 'last_peered': '2024-09-16T04:43:23.936308+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:43:23.936308+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4716, 'log_dups_size': 0, 'ondisk_log_size': 4716, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38588306, 'num_objects': 4711, 'num_object_clones': 0, 'num_object_copies': 18844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4711, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4713, 'num_write_kb': 37692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2522}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:30.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:43:30.233 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4665", 'reported_seq': 4701, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:23.939615+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:43:23.939615+0000', 'last_peered': '2024-09-16T04:43:23.939615+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:43:23.939615+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4665, 'log_dups_size': 0, 'ondisk_log_size': 4665, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38206086, 'num_objects': 4664, 'num_object_clones': 0, 'num_object_copies': 18656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4664, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4664, 'num_write_kb': 37311, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:30.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:43:30.233 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4698", 'reported_seq': 4767, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:24.183329+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:43:24.183329+0000', 'last_peered': '2024-09-16T04:43:24.183329+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:43:24.183329+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4698, 'log_dups_size': 0, 'ondisk_log_size': 4698, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38476422, 'num_objects': 4697, 'num_object_clones': 0, 'num_object_copies': 18788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4697, 'num_write_kb': 37575, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:30.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:43:30.233 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4805", 'reported_seq': 4855, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:23.888867+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:43:23.888867+0000', 'last_peered': '2024-09-16T04:43:23.888867+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:43:23.888867+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4805, 'log_dups_size': 0, 'ondisk_log_size': 4805, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39326988, 'num_objects': 4801, 'num_object_clones': 0, 'num_object_copies': 19204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4801, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4803, 'num_write_kb': 38420, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2567}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:30.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:43:30.233 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4701", 'reported_seq': 4776, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:24.127504+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:43:24.127504+0000', 'last_peered': '2024-09-16T04:43:24.127504+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:43:24.127504+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4701, 'log_dups_size': 0, 'ondisk_log_size': 4701, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38483212, 'num_objects': 4698, 'num_object_clones': 0, 'num_object_copies': 18792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4699, 'num_write_kb': 37589, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2566}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:30.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:43:30.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4698", 'reported_seq': 4760, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:24.440863+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:43:24.440863+0000', 'last_peered': '2024-09-16T04:43:24.440863+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:43:24.440863+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4698, 'log_dups_size': 0, 'ondisk_log_size': 4698, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38442252, 'num_objects': 4693, 'num_object_clones': 0, 'num_object_copies': 18772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4693, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4696, 'num_write_kb': 37556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2454}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:30.234 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:43:30.285 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 156 20 77701 77681 3.88921 2.07812 0.0411701 0.0401611 2024-09-16T04:43:30.936 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:30.937 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33895/309812 objects degraded (10.941%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":223,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":77453,"data_bytes":634455720,"bytes_used":2056900608,"bytes_avail":384490156032,"bytes_total":386547056640,"degraded_objects":33895,"degraded_total":309812,"degraded_ratio":0.10940505855163776,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":134429,"recovering_keys_per_sec":0,"num_objects_recovered":197,"num_bytes_recovered":1613824,"num_keys_recovered":0,"write_bytes_sec":3152602,"read_op_per_sec":0,"write_op_per_sec":384},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:43:30.965 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:43:30.966 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:31.287 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 157 20 77971 77951 3.87788 2.10938 0.0400748 0.0402763 2024-09-16T04:43:31.551 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:31.552 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:31.575 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-09-16T04:43:30.606514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634455720,"num_objects":77453,"num_object_clones":0,"num_object_copies":309812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77480,"num_write_kb":619722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1687,"num_bytes_recovered":13582336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77507,"ondisk_log_size":77507,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2008692,"kb_used_data":1098100,"kb_used_omap":14,"kb_used_meta":910513,"kb_avail":375478668,"statfs":{"total":386547056640,"available":384490156032,"internally_reserved":0,"allocated":1124454400,"data_stored":1123771160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":932365873},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,13,3,10,13,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37847040,"num_objects":4620,"num_object_clones":0,"num_object_copies":18480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4620,"num_write_kb":36960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":1613824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005013"},"pg_stats":[{"pgid":"1.f","version":"31'4856","reported_seq":4896,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:28.938931+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:28.938931+0000","last_peered":"2024-09-16T04:43:28.938931+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:28.938931+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39780352,"num_objects":4856,"num_object_clones":0,"num_object_copies":19424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4856,"num_write_kb":38848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2637}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4911","reported_seq":4973,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.370590+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:29.370590+0000","last_peered":"2024-09-16T04:43:29.370590+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:29.370590+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4911,"log_dups_size":0,"ondisk_log_size":4911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40195340,"num_objects":4907,"num_object_clones":0,"num_object_copies":19628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4909,"num_write_kb":39268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2696}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4884","reported_seq":4955,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.158714+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:29.158714+0000","last_peered":"2024-09-16T04:43:29.158714+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:29.158714+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4884,"log_dups_size":0,"ondisk_log_size":4884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39983750,"num_objects":4881,"num_object_clones":0,"num_object_copies":19524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4883,"num_write_kb":39054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2654}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4854","reported_seq":4916,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.478035+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:29.478035+0000","last_peered":"2024-09-16T04:43:29.478035+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:29.478035+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4854,"log_dups_size":0,"ondisk_log_size":4854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39728396,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4852,"num_write_kb":38805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2621}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4860","reported_seq":4912,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.009844+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:29.009844+0000","last_peered":"2024-09-16T04:43:29.009844+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:29.009844+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4860,"log_dups_size":0,"ondisk_log_size":4860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39785740,"num_objects":4857,"num_object_clones":0,"num_object_copies":19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4858,"num_write_kb":38861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2574}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4752","reported_seq":4792,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.059084+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:29.059084+0000","last_peered":"2024-09-16T04:43:29.059084+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:29.059084+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4752,"log_dups_size":0,"ondisk_log_size":4752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38928384,"num_objects":4752,"num_object_clones":0,"num_object_copies":19008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4752,"num_write_kb":38016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2544}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4914","reported_seq":4975,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.551915+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:29.551915+0000","last_peered":"2024-09-16T04:43:29.551915+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:29.551915+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4914,"log_dups_size":0,"ondisk_log_size":4914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40203532,"num_objects":4908,"num_object_clones":0,"num_object_copies":19632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4912,"num_write_kb":39276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4850","reported_seq":8231,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.549457+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:29.549457+0000","last_peered":"2024-09-16T04:43:29.549457+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:29.549457+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39687436,"num_objects":4845,"num_object_clones":0,"num_object_copies":19380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4848,"num_write_kb":38772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1662,"num_bytes_recovered":13582336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4911","reported_seq":4964,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:28.835359+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:28.835359+0000","last_peered":"2024-09-16T04:43:28.835359+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:28.835359+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4911,"log_dups_size":0,"ondisk_log_size":4911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40159768,"num_objects":4903,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4908,"num_write_kb":39241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2563}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4872","reported_seq":4946,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.037193+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:29.037193+0000","last_peered":"2024-09-16T04:43:29.037193+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:29.037193+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4872,"log_dups_size":0,"ondisk_log_size":4872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39892236,"num_objects":4870,"num_object_clones":0,"num_object_copies":19480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4870,"num_write_kb":38958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2670}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4815","reported_seq":4869,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:28.867993+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:28.867993+0000","last_peered":"2024-09-16T04:43:28.867993+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:28.867993+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39399314,"num_objects":4810,"num_object_clones":0,"num_object_copies":19240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4812,"num_write_kb":38484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2621}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4759","reported_seq":4795,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:28.855882+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:28.855882+0000","last_peered":"2024-09-16T04:43:28.855882+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:28.855882+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4759,"log_dups_size":0,"ondisk_log_size":4759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38976134,"num_objects":4758,"num_object_clones":0,"num_object_copies":19032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4758,"num_write_kb":38063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2630}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4787","reported_seq":4857,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.102647+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:29.102647+0000","last_peered":"2024-09-16T04:43:29.102647+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:29.102647+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4787,"log_dups_size":0,"ondisk_log_size":4787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39205510,"num_objects":4786,"num_object_clones":0,"num_object_copies":19144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4786,"num_write_kb":38287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2561}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4899","reported_seq":4949,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.026895+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:29.026895+0000","last_peered":"2024-09-16T04:43:29.026895+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:29.026895+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4899,"log_dups_size":0,"ondisk_log_size":4899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40097036,"num_objects":4895,"num_object_clones":0,"num_object_copies":19580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4897,"num_write_kb":39172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2661}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4793","reported_seq":4867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.202879+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:29.202879+0000","last_peered":"2024-09-16T04:43:29.202879+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:29.202879+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4793,"log_dups_size":0,"ondisk_log_size":4793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39236876,"num_objects":4790,"num_object_clones":0,"num_object_copies":19160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4791,"num_write_kb":38325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2658}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4790","reported_seq":4852,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.558270+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:29.558270+0000","last_peered":"2024-09-16T04:43:29.558270+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:29.558270+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4790,"log_dups_size":0,"ondisk_log_size":4790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39195916,"num_objects":4785,"num_object_clones":0,"num_object_copies":19140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4788,"num_write_kb":38292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2546}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":634455720,"num_objects":77453,"num_object_clones":0,"num_object_copies":309812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77480,"num_write_kb":619722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1687,"num_bytes_recovered":13582336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126158336,"data_stored":1126158336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77507,"ondisk_log_size":77507,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557536,"kb_used_data":309264,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93814304,"statfs":{"total":96636764160,"available":96065847296,"internally_reserved":0,"allocated":316686336,"data_stored":316515526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254209743},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,2,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552792,"kb_used_data":309192,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93819048,"statfs":{"total":96636764160,"available":96070705152,"internally_reserved":0,"allocated":316612608,"data_stored":316441798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,2,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051609,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331652,"kb_used_data":169460,"kb_used_omap":4,"kb_used_meta":162171,"kb_avail":94040188,"statfs":{"total":96636764160,"available":96297152512,"internally_reserved":0,"allocated":173527040,"data_stored":173356230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":166063823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566712,"kb_used_data":310184,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93805128,"statfs":{"total":96636764160,"available":96056451072,"internally_reserved":0,"allocated":317628416,"data_stored":317457606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":262663886},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317513728,"data_stored":317513728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":174522368,"data_stored":174522368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317153280,"data_stored":317153280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":316968960,"data_stored":316968960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:32.285 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 158 20 78279 78259 3.86857 2.40625 0.0573182 0.0403745 2024-09-16T04:43:33.285 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 159 20 78603 78583 3.86015 2.53125 0.0367361 0.0404678 2024-09-16T04:43:34.285 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:2024-09-16T04:43:34.284802+0000 min lat: 0.00269411 max lat: 1.73107 avg lat: 0.0405587 2024-09-16T04:43:34.285 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T04:43:34.285 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 160 20 78917 78897 3.85136 2.45312 0.102441 0.0405587 2024-09-16T04:43:34.577 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:35.159 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:35.159 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:35.184 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-09-16T04:43:34.607605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":642893480,"num_objects":78483,"num_object_clones":0,"num_object_copies":313932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78510,"num_write_kb":627962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1687,"num_bytes_recovered":13582336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78537,"ondisk_log_size":78537,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2040408,"kb_used_data":1117336,"kb_used_omap":14,"kb_used_meta":922993,"kb_avail":375446952,"statfs":{"total":386547056640,"available":384457678848,"internally_reserved":0,"allocated":1144152064,"data_stored":1143468824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":945145393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,7,5,6,9,19,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33751040,"num_objects":4120,"num_object_clones":0,"num_object_copies":16480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4120,"num_write_kb":32960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003105"},"pg_stats":[{"pgid":"1.f","version":"31'4947","reported_seq":4987,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.144020+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:34.144020+0000","last_peered":"2024-09-16T04:43:34.144020+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:34.144020+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4947,"log_dups_size":0,"ondisk_log_size":4947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40525824,"num_objects":4947,"num_object_clones":0,"num_object_copies":19788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4947,"num_write_kb":39576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2728}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4911","reported_seq":4973,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.370590+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:29.370590+0000","last_peered":"2024-09-16T04:43:29.370590+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:29.370590+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4911,"log_dups_size":0,"ondisk_log_size":4911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40195340,"num_objects":4907,"num_object_clones":0,"num_object_copies":19628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4909,"num_write_kb":39268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2696}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4969","reported_seq":5040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.203623+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:34.203623+0000","last_peered":"2024-09-16T04:43:34.203623+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:34.203623+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4969,"log_dups_size":0,"ondisk_log_size":4969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40680070,"num_objects":4966,"num_object_clones":0,"num_object_copies":19864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":39734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2739}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4854","reported_seq":4916,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.478035+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:29.478035+0000","last_peered":"2024-09-16T04:43:29.478035+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:29.478035+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4854,"log_dups_size":0,"ondisk_log_size":4854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39728396,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4852,"num_write_kb":38805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2621}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4954","reported_seq":5006,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.106386+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:34.106386+0000","last_peered":"2024-09-16T04:43:34.106386+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:34.106386+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40555788,"num_objects":4951,"num_object_clones":0,"num_object_copies":19804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4952,"num_write_kb":39613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2668}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4836","reported_seq":4876,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:33.950779+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:33.950779+0000","last_peered":"2024-09-16T04:43:33.950779+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:33.950779+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4836,"log_dups_size":0,"ondisk_log_size":4836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39616512,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4836,"num_write_kb":38688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4914","reported_seq":4975,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.551915+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:29.551915+0000","last_peered":"2024-09-16T04:43:29.551915+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:29.551915+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4914,"log_dups_size":0,"ondisk_log_size":4914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40203532,"num_objects":4908,"num_object_clones":0,"num_object_copies":19632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4912,"num_write_kb":39276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4850","reported_seq":8231,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.549457+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:29.549457+0000","last_peered":"2024-09-16T04:43:29.549457+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:29.549457+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39687436,"num_objects":4845,"num_object_clones":0,"num_object_copies":19380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4848,"num_write_kb":38772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1662,"num_bytes_recovered":13582336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5018","reported_seq":5071,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.141408+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:34.141408+0000","last_peered":"2024-09-16T04:43:34.141408+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:34.141408+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41036312,"num_objects":5010,"num_object_clones":0,"num_object_copies":20040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5015,"num_write_kb":40097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2670}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4983","reported_seq":5057,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.286220+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:34.286220+0000","last_peered":"2024-09-16T04:43:34.286220+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:34.286220+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40801548,"num_objects":4981,"num_object_clones":0,"num_object_copies":19924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4981,"num_write_kb":39846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2781}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4913","reported_seq":4967,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.106667+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:34.106667+0000","last_peered":"2024-09-16T04:43:34.106667+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:34.106667+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40202130,"num_objects":4908,"num_object_clones":0,"num_object_copies":19632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4910,"num_write_kb":39268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2719}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4859","reported_seq":4895,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.168944+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:34.168944+0000","last_peered":"2024-09-16T04:43:34.168944+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:34.168944+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39795334,"num_objects":4858,"num_object_clones":0,"num_object_copies":19432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4858,"num_write_kb":38863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2730}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4876","reported_seq":4946,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.092726+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:34.092726+0000","last_peered":"2024-09-16T04:43:34.092726+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:34.092726+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4876,"log_dups_size":0,"ondisk_log_size":4876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39934598,"num_objects":4875,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4875,"num_write_kb":38999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2650}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4988","reported_seq":5038,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.147389+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:34.147389+0000","last_peered":"2024-09-16T04:43:34.147389+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:34.147389+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4988,"log_dups_size":0,"ondisk_log_size":4988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40826124,"num_objects":4984,"num_object_clones":0,"num_object_copies":19936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4986,"num_write_kb":39884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2750}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4875","reported_seq":4950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.186617+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:34.186617+0000","last_peered":"2024-09-16T04:43:34.186617+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:34.186617+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4875,"log_dups_size":0,"ondisk_log_size":4875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39908620,"num_objects":4872,"num_object_clones":0,"num_object_copies":19488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4873,"num_write_kb":38981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2740}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4790","reported_seq":4852,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.558270+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:29.558270+0000","last_peered":"2024-09-16T04:43:29.558270+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:29.558270+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4790,"log_dups_size":0,"ondisk_log_size":4790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39195916,"num_objects":4785,"num_object_clones":0,"num_object_copies":19140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4788,"num_write_kb":38292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2546}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":642893480,"num_objects":78483,"num_object_clones":0,"num_object_copies":313932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78510,"num_write_kb":627962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1687,"num_bytes_recovered":13582336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1144803328,"data_stored":1144803328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78537,"ondisk_log_size":78537,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567712,"kb_used_data":315280,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93804128,"statfs":{"total":96636764160,"available":96055427072,"internally_reserved":0,"allocated":322846720,"data_stored":322675910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258469583},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562852,"kb_used_data":315092,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93808988,"statfs":{"total":96636764160,"available":96060403712,"internally_reserved":0,"allocated":322654208,"data_stored":322483398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,2,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051610,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":343132,"kb_used_data":176780,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94028708,"statfs":{"total":96636764160,"available":96285396992,"internally_reserved":0,"allocated":181022720,"data_stored":180851910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170323663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566712,"kb_used_data":310184,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93805128,"statfs":{"total":96636764160,"available":96056451072,"internally_reserved":0,"allocated":317628416,"data_stored":317457606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":262663886},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317513728,"data_stored":317513728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180895744,"data_stored":180895744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323235840,"data_stored":323235840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":323158016,"data_stored":323158016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:35.185 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:35.285 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 161 20 79205 79185 3.84142 2.25 0.0494178 0.0406589 2024-09-16T04:43:35.789 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:35.789 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:35.817 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-09-16T04:43:34.607605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":642893480,"num_objects":78483,"num_object_clones":0,"num_object_copies":313932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78510,"num_write_kb":627962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1687,"num_bytes_recovered":13582336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78537,"ondisk_log_size":78537,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2040408,"kb_used_data":1117336,"kb_used_omap":14,"kb_used_meta":922993,"kb_avail":375446952,"statfs":{"total":386547056640,"available":384457678848,"internally_reserved":0,"allocated":1144152064,"data_stored":1143468824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":945145393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,7,5,6,9,19,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33751040,"num_objects":4120,"num_object_clones":0,"num_object_copies":16480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4120,"num_write_kb":32960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003105"},"pg_stats":[{"pgid":"1.f","version":"31'4947","reported_seq":4987,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.144020+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:34.144020+0000","last_peered":"2024-09-16T04:43:34.144020+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:34.144020+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4947,"log_dups_size":0,"ondisk_log_size":4947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40525824,"num_objects":4947,"num_object_clones":0,"num_object_copies":19788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4947,"num_write_kb":39576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2728}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4911","reported_seq":4973,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.370590+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:29.370590+0000","last_peered":"2024-09-16T04:43:29.370590+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:29.370590+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4911,"log_dups_size":0,"ondisk_log_size":4911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40195340,"num_objects":4907,"num_object_clones":0,"num_object_copies":19628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4909,"num_write_kb":39268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2696}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4969","reported_seq":5040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.203623+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:34.203623+0000","last_peered":"2024-09-16T04:43:34.203623+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:34.203623+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4969,"log_dups_size":0,"ondisk_log_size":4969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40680070,"num_objects":4966,"num_object_clones":0,"num_object_copies":19864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":39734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2739}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4854","reported_seq":4916,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.478035+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:29.478035+0000","last_peered":"2024-09-16T04:43:29.478035+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:29.478035+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4854,"log_dups_size":0,"ondisk_log_size":4854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39728396,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4852,"num_write_kb":38805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2621}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4954","reported_seq":5006,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.106386+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:34.106386+0000","last_peered":"2024-09-16T04:43:34.106386+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:34.106386+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40555788,"num_objects":4951,"num_object_clones":0,"num_object_copies":19804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4952,"num_write_kb":39613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2668}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4836","reported_seq":4876,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:33.950779+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:33.950779+0000","last_peered":"2024-09-16T04:43:33.950779+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:33.950779+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4836,"log_dups_size":0,"ondisk_log_size":4836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39616512,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4836,"num_write_kb":38688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4914","reported_seq":4975,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.551915+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:29.551915+0000","last_peered":"2024-09-16T04:43:29.551915+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:29.551915+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4914,"log_dups_size":0,"ondisk_log_size":4914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40203532,"num_objects":4908,"num_object_clones":0,"num_object_copies":19632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4912,"num_write_kb":39276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4850","reported_seq":8231,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.549457+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:29.549457+0000","last_peered":"2024-09-16T04:43:29.549457+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:29.549457+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39687436,"num_objects":4845,"num_object_clones":0,"num_object_copies":19380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4848,"num_write_kb":38772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1662,"num_bytes_recovered":13582336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5018","reported_seq":5071,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.141408+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:34.141408+0000","last_peered":"2024-09-16T04:43:34.141408+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:34.141408+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41036312,"num_objects":5010,"num_object_clones":0,"num_object_copies":20040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5015,"num_write_kb":40097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2670}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4983","reported_seq":5057,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.286220+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:34.286220+0000","last_peered":"2024-09-16T04:43:34.286220+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:34.286220+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40801548,"num_objects":4981,"num_object_clones":0,"num_object_copies":19924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4981,"num_write_kb":39846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2781}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4913","reported_seq":4967,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.106667+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:34.106667+0000","last_peered":"2024-09-16T04:43:34.106667+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:34.106667+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40202130,"num_objects":4908,"num_object_clones":0,"num_object_copies":19632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4910,"num_write_kb":39268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2719}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4859","reported_seq":4895,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.168944+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:34.168944+0000","last_peered":"2024-09-16T04:43:34.168944+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:34.168944+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39795334,"num_objects":4858,"num_object_clones":0,"num_object_copies":19432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4858,"num_write_kb":38863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2730}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4876","reported_seq":4946,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.092726+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:34.092726+0000","last_peered":"2024-09-16T04:43:34.092726+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:34.092726+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4876,"log_dups_size":0,"ondisk_log_size":4876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39934598,"num_objects":4875,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4875,"num_write_kb":38999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2650}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4988","reported_seq":5038,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.147389+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:34.147389+0000","last_peered":"2024-09-16T04:43:34.147389+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:34.147389+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4988,"log_dups_size":0,"ondisk_log_size":4988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40826124,"num_objects":4984,"num_object_clones":0,"num_object_copies":19936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4986,"num_write_kb":39884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2750}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4875","reported_seq":4950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.186617+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:34.186617+0000","last_peered":"2024-09-16T04:43:34.186617+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:34.186617+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4875,"log_dups_size":0,"ondisk_log_size":4875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39908620,"num_objects":4872,"num_object_clones":0,"num_object_copies":19488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4873,"num_write_kb":38981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2740}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4790","reported_seq":4852,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:29.558270+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:29.558270+0000","last_peered":"2024-09-16T04:43:29.558270+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:29.558270+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4790,"log_dups_size":0,"ondisk_log_size":4790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39195916,"num_objects":4785,"num_object_clones":0,"num_object_copies":19140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4788,"num_write_kb":38292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2546}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":642893480,"num_objects":78483,"num_object_clones":0,"num_object_copies":313932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78510,"num_write_kb":627962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1687,"num_bytes_recovered":13582336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1144803328,"data_stored":1144803328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78537,"ondisk_log_size":78537,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567712,"kb_used_data":315280,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93804128,"statfs":{"total":96636764160,"available":96055427072,"internally_reserved":0,"allocated":322846720,"data_stored":322675910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258469583},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562852,"kb_used_data":315092,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93808988,"statfs":{"total":96636764160,"available":96060403712,"internally_reserved":0,"allocated":322654208,"data_stored":322483398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,2,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051610,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":343132,"kb_used_data":176780,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94028708,"statfs":{"total":96636764160,"available":96285396992,"internally_reserved":0,"allocated":181022720,"data_stored":180851910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170323663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566712,"kb_used_data":310184,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93805128,"statfs":{"total":96636764160,"available":96056451072,"internally_reserved":0,"allocated":317628416,"data_stored":317457606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":262663886},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317513728,"data_stored":317513728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180895744,"data_stored":180895744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323235840,"data_stored":323235840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":323158016,"data_stored":323158016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:35.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:43:35.818 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4947", 'reported_seq': 4987, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:34.144020+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:43:34.144020+0000', 'last_peered': '2024-09-16T04:43:34.144020+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:43:34.144020+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4947, 'log_dups_size': 0, 'ondisk_log_size': 4947, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40525824, 'num_objects': 4947, 'num_object_clones': 0, 'num_object_copies': 19788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4947, 'num_write_kb': 39576, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2728}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:35.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:43:35.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4911", 'reported_seq': 4973, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:29.370590+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:43:29.370590+0000', 'last_peered': '2024-09-16T04:43:29.370590+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:43:29.370590+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4911, 'log_dups_size': 0, 'ondisk_log_size': 4911, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40195340, 'num_objects': 4907, 'num_object_clones': 0, 'num_object_copies': 19628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4907, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4909, 'num_write_kb': 39268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2696}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:35.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:43:35.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4969", 'reported_seq': 5040, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:34.203623+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:43:34.203623+0000', 'last_peered': '2024-09-16T04:43:34.203623+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:43:34.203623+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4969, 'log_dups_size': 0, 'ondisk_log_size': 4969, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40680070, 'num_objects': 4966, 'num_object_clones': 0, 'num_object_copies': 19864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4966, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4968, 'num_write_kb': 39734, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2739}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:35.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:43:35.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4854", 'reported_seq': 4916, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:29.478035+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:43:29.478035+0000', 'last_peered': '2024-09-16T04:43:29.478035+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:43:29.478035+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4854, 'log_dups_size': 0, 'ondisk_log_size': 4854, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39728396, 'num_objects': 4850, 'num_object_clones': 0, 'num_object_copies': 19400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4852, 'num_write_kb': 38805, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2621}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:35.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:43:35.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4954", 'reported_seq': 5006, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:34.106386+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:43:34.106386+0000', 'last_peered': '2024-09-16T04:43:34.106386+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:43:34.106386+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4954, 'log_dups_size': 0, 'ondisk_log_size': 4954, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40555788, 'num_objects': 4951, 'num_object_clones': 0, 'num_object_copies': 19804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4951, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4952, 'num_write_kb': 39613, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2668}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:35.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:43:35.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4836", 'reported_seq': 4876, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:33.950779+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:43:33.950779+0000', 'last_peered': '2024-09-16T04:43:33.950779+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:43:33.950779+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4836, 'log_dups_size': 0, 'ondisk_log_size': 4836, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39616512, 'num_objects': 4836, 'num_object_clones': 0, 'num_object_copies': 19344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4836, 'num_write_kb': 38688, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2628}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:35.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:43:35.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4914", 'reported_seq': 4975, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:29.551915+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:43:29.551915+0000', 'last_peered': '2024-09-16T04:43:29.551915+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:43:29.551915+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4914, 'log_dups_size': 0, 'ondisk_log_size': 4914, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40203532, 'num_objects': 4908, 'num_object_clones': 0, 'num_object_copies': 19632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4912, 'num_write_kb': 39276, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2682}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:35.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:43:35.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4850", 'reported_seq': 8231, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:29.549457+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:43:29.549457+0000', 'last_peered': '2024-09-16T04:43:29.549457+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:43:29.549457+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4850, 'log_dups_size': 0, 'ondisk_log_size': 4850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39687436, 'num_objects': 4845, 'num_object_clones': 0, 'num_object_copies': 19380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 580, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4845, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4848, 'num_write_kb': 38772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1662, 'num_bytes_recovered': 13582336, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4265}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:35.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:43:35.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5018", 'reported_seq': 5071, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:34.141408+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:43:34.141408+0000', 'last_peered': '2024-09-16T04:43:34.141408+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:43:34.141408+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5018, 'log_dups_size': 0, 'ondisk_log_size': 5018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41036312, 'num_objects': 5010, 'num_object_clones': 0, 'num_object_copies': 20040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5015, 'num_write_kb': 40097, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2670}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:35.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:43:35.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4983", 'reported_seq': 5057, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:34.286220+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:43:34.286220+0000', 'last_peered': '2024-09-16T04:43:34.286220+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:43:34.286220+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4983, 'log_dups_size': 0, 'ondisk_log_size': 4983, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40801548, 'num_objects': 4981, 'num_object_clones': 0, 'num_object_copies': 19924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4981, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4981, 'num_write_kb': 39846, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2781}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:35.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:43:35.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4913", 'reported_seq': 4967, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:34.106667+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:43:34.106667+0000', 'last_peered': '2024-09-16T04:43:34.106667+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:43:34.106667+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4913, 'log_dups_size': 0, 'ondisk_log_size': 4913, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40202130, 'num_objects': 4908, 'num_object_clones': 0, 'num_object_copies': 19632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4910, 'num_write_kb': 39268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2719}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:35.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:43:35.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4859", 'reported_seq': 4895, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:34.168944+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:43:34.168944+0000', 'last_peered': '2024-09-16T04:43:34.168944+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:43:34.168944+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4859, 'log_dups_size': 0, 'ondisk_log_size': 4859, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39795334, 'num_objects': 4858, 'num_object_clones': 0, 'num_object_copies': 19432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4858, 'num_write_kb': 38863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2730}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:35.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:43:35.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4876", 'reported_seq': 4946, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:34.092726+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:43:34.092726+0000', 'last_peered': '2024-09-16T04:43:34.092726+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:43:34.092726+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4876, 'log_dups_size': 0, 'ondisk_log_size': 4876, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39934598, 'num_objects': 4875, 'num_object_clones': 0, 'num_object_copies': 19500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4875, 'num_write_kb': 38999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2650}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:35.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:43:35.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4988", 'reported_seq': 5038, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:34.147389+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:43:34.147389+0000', 'last_peered': '2024-09-16T04:43:34.147389+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:43:34.147389+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4988, 'log_dups_size': 0, 'ondisk_log_size': 4988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40826124, 'num_objects': 4984, 'num_object_clones': 0, 'num_object_copies': 19936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4986, 'num_write_kb': 39884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2750}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:35.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:43:35.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4875", 'reported_seq': 4950, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:34.186617+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:43:34.186617+0000', 'last_peered': '2024-09-16T04:43:34.186617+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:43:34.186617+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4875, 'log_dups_size': 0, 'ondisk_log_size': 4875, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39908620, 'num_objects': 4872, 'num_object_clones': 0, 'num_object_copies': 19488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4872, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4873, 'num_write_kb': 38981, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2740}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:35.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:43:35.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4790", 'reported_seq': 4852, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:29.558270+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:43:29.558270+0000', 'last_peered': '2024-09-16T04:43:29.558270+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:43:29.558270+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4790, 'log_dups_size': 0, 'ondisk_log_size': 4790, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39195916, 'num_objects': 4785, 'num_object_clones': 0, 'num_object_copies': 19140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4785, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4788, 'num_write_kb': 38292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2546}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:35.822 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:43:36.285 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 162 20 79484 79464 3.83116 2.17969 0.0746361 0.040768 2024-09-16T04:43:36.567 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:36.567 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33895/313932 objects degraded (10.797%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":228,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":78483,"data_bytes":642893480,"bytes_used":2089377792,"bytes_avail":384457678848,"bytes_total":386547056640,"degraded_objects":33895,"degraded_total":313932,"degraded_ratio":0.1079692417466203,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":85311,"recovering_keys_per_sec":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"write_bytes_sec":2811858,"read_op_per_sec":0,"write_op_per_sec":343},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:43:36.593 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:43:36.593 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:37.194 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:37.194 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:37.214 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-09-16T04:43:36.608145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":646751912,"num_objects":78954,"num_object_clones":0,"num_object_copies":315816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78981,"num_write_kb":631730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1754,"num_bytes_recovered":14131200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79008,"ondisk_log_size":79008,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2045552,"kb_used_data":1129392,"kb_used_omap":14,"kb_used_meta":916081,"kb_avail":375441808,"statfs":{"total":386547056640,"available":384452411392,"internally_reserved":0,"allocated":1156497408,"data_stored":1155814168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":938067505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,7,7,7,10,14,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24739840,"num_objects":3020,"num_object_clones":0,"num_object_copies":12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":24160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003165"},"pg_stats":[{"pgid":"1.f","version":"31'4947","reported_seq":4987,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.144020+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:34.144020+0000","last_peered":"2024-09-16T04:43:34.144020+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:34.144020+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4947,"log_dups_size":0,"ondisk_log_size":4947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40525824,"num_objects":4947,"num_object_clones":0,"num_object_copies":19788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4947,"num_write_kb":39576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2728}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4999","reported_seq":5061,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.547898+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:34.547898+0000","last_peered":"2024-09-16T04:43:34.547898+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:34.547898+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4999,"log_dups_size":0,"ondisk_log_size":4999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40916236,"num_objects":4995,"num_object_clones":0,"num_object_copies":19980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4997,"num_write_kb":39972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2784}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4969","reported_seq":5040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.203623+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:34.203623+0000","last_peered":"2024-09-16T04:43:34.203623+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:34.203623+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4969,"log_dups_size":0,"ondisk_log_size":4969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40680070,"num_objects":4966,"num_object_clones":0,"num_object_copies":19864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":39734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2739}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4952","reported_seq":5014,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.483590+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:34.483590+0000","last_peered":"2024-09-16T04:43:34.483590+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:34.483590+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4952,"log_dups_size":0,"ondisk_log_size":4952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40531212,"num_objects":4948,"num_object_clones":0,"num_object_copies":19792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4950,"num_write_kb":39589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2719}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4954","reported_seq":5006,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.106386+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:34.106386+0000","last_peered":"2024-09-16T04:43:34.106386+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:34.106386+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40555788,"num_objects":4951,"num_object_clones":0,"num_object_copies":19804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4952,"num_write_kb":39613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2668}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4836","reported_seq":4876,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:33.950779+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:33.950779+0000","last_peered":"2024-09-16T04:43:33.950779+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:33.950779+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4836,"log_dups_size":0,"ondisk_log_size":4836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39616512,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4836,"num_write_kb":38688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5000","reported_seq":5061,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.481061+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:34.481061+0000","last_peered":"2024-09-16T04:43:34.481061+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:34.481061+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5000,"log_dups_size":0,"ondisk_log_size":5000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40908044,"num_objects":4994,"num_object_clones":0,"num_object_copies":19976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4998,"num_write_kb":39964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2768}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4967","reported_seq":8482,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.625684+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:34.625684+0000","last_peered":"2024-09-16T04:43:34.625684+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:34.625684+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4967,"log_dups_size":0,"ondisk_log_size":4967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40645900,"num_objects":4962,"num_object_clones":0,"num_object_copies":19848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4965,"num_write_kb":39708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1729,"num_bytes_recovered":14131200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5018","reported_seq":5071,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.141408+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:34.141408+0000","last_peered":"2024-09-16T04:43:34.141408+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:34.141408+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41036312,"num_objects":5010,"num_object_clones":0,"num_object_copies":20040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5015,"num_write_kb":40097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2670}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4983","reported_seq":5057,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.286220+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:34.286220+0000","last_peered":"2024-09-16T04:43:34.286220+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:34.286220+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40801548,"num_objects":4981,"num_object_clones":0,"num_object_copies":19924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4981,"num_write_kb":39846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2781}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4913","reported_seq":4967,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.106667+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:34.106667+0000","last_peered":"2024-09-16T04:43:34.106667+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:34.106667+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40202130,"num_objects":4908,"num_object_clones":0,"num_object_copies":19632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4910,"num_write_kb":39268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2719}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4859","reported_seq":4895,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.168944+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:34.168944+0000","last_peered":"2024-09-16T04:43:34.168944+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:34.168944+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39795334,"num_objects":4858,"num_object_clones":0,"num_object_copies":19432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4858,"num_write_kb":38863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2730}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4876","reported_seq":4946,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.092726+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:34.092726+0000","last_peered":"2024-09-16T04:43:34.092726+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:34.092726+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4876,"log_dups_size":0,"ondisk_log_size":4876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39934598,"num_objects":4875,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4875,"num_write_kb":38999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2650}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4988","reported_seq":5038,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.147389+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:34.147389+0000","last_peered":"2024-09-16T04:43:34.147389+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:34.147389+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4988,"log_dups_size":0,"ondisk_log_size":4988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40826124,"num_objects":4984,"num_object_clones":0,"num_object_copies":19936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4986,"num_write_kb":39884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2750}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4875","reported_seq":4950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.186617+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:34.186617+0000","last_peered":"2024-09-16T04:43:34.186617+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:34.186617+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4875,"log_dups_size":0,"ondisk_log_size":4875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39908620,"num_objects":4872,"num_object_clones":0,"num_object_copies":19488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4873,"num_write_kb":38981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2740}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4872","reported_seq":4934,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:34.581371+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:34.581371+0000","last_peered":"2024-09-16T04:43:34.581371+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:34.581371+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4872,"log_dups_size":0,"ondisk_log_size":4872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39867660,"num_objects":4867,"num_object_clones":0,"num_object_copies":19468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4870,"num_write_kb":38948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2628}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":646751912,"num_objects":78954,"num_object_clones":0,"num_object_copies":315816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78981,"num_write_kb":631730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1754,"num_bytes_recovered":14131200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1157435392,"data_stored":1157435392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79008,"ondisk_log_size":79008,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567712,"kb_used_data":315280,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93804128,"statfs":{"total":96636764160,"available":96055427072,"internally_reserved":0,"allocated":322846720,"data_stored":322675910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":258469583},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562852,"kb_used_data":315092,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93808988,"statfs":{"total":96636764160,"available":96060403712,"internally_reserved":0,"allocated":322654208,"data_stored":322483398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,2,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354788,"kb_used_data":182868,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":94017052,"statfs":{"total":96636764160,"available":96273461248,"internally_reserved":0,"allocated":187256832,"data_stored":187086022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":176025295},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560200,"kb_used_data":316152,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93811640,"statfs":{"total":96636764160,"available":96063119360,"internally_reserved":0,"allocated":323739648,"data_stored":323568838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":249884366},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323649536,"data_stored":323649536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":187392000,"data_stored":187392000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323235840,"data_stored":323235840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":323158016,"data_stored":323158016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:37.285 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 163 20 79753 79733 3.82055 2.10156 0.119183 0.0408816 2024-09-16T04:43:38.286 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 164 20 80065 80045 3.81211 2.4375 0.028415 0.0409738 2024-09-16T04:43:39.285 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 165 20 80376 80356 3.80374 2.42969 0.0643267 0.0410643 2024-09-16T04:43:40.216 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:40.286 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 166 20 80685 80665 3.79537 2.41406 0.0483943 0.0411568 2024-09-16T04:43:40.791 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:40.791 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:40.815 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-16T04:43:40.609168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658745000,"num_objects":80418,"num_object_clones":0,"num_object_copies":321672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80445,"num_write_kb":643442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1821,"num_bytes_recovered":14680064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80472,"ondisk_log_size":80472,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2075532,"kb_used_data":1146892,"kb_used_omap":14,"kb_used_meta":928561,"kb_avail":375411828,"statfs":{"total":386547056640,"available":384421711872,"internally_reserved":0,"allocated":1174417408,"data_stored":1173734168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":950847025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,3,5,13,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36732928,"num_objects":4484,"num_object_clones":0,"num_object_copies":17936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4484,"num_write_kb":35872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":1589248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003135"},"pg_stats":[{"pgid":"1.f","version":"31'5035","reported_seq":5075,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.080250+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:39.080250+0000","last_peered":"2024-09-16T04:43:39.080250+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:39.080250+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41246720,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5035,"num_write_kb":40280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2816}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5086","reported_seq":5148,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.524819+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:39.524819+0000","last_peered":"2024-09-16T04:43:39.524819+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:39.524819+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41628940,"num_objects":5082,"num_object_clones":0,"num_object_copies":20328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5084,"num_write_kb":40668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2871}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5057","reported_seq":5128,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.280446+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:39.280446+0000","last_peered":"2024-09-16T04:43:39.280446+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:39.280446+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41400966,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5056,"num_write_kb":40438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2827}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5035","reported_seq":5097,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.509519+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:39.509519+0000","last_peered":"2024-09-16T04:43:39.509519+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:39.509519+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41211148,"num_objects":5031,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5033,"num_write_kb":40253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2802}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5035","reported_seq":5087,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.091952+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:39.091952+0000","last_peered":"2024-09-16T04:43:39.091952+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:39.091952+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41219340,"num_objects":5032,"num_object_clones":0,"num_object_copies":20128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5033,"num_write_kb":40261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4926","reported_seq":4966,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.148473+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:39.148473+0000","last_peered":"2024-09-16T04:43:39.148473+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:39.148473+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40353792,"num_objects":4926,"num_object_clones":0,"num_object_copies":19704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4926,"num_write_kb":39408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5087","reported_seq":5148,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.370320+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:39.370320+0000","last_peered":"2024-09-16T04:43:39.370320+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:39.370320+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41620748,"num_objects":5081,"num_object_clones":0,"num_object_copies":20324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":40660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2855}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5075","reported_seq":8724,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.630076+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:39.630076+0000","last_peered":"2024-09-16T04:43:39.630076+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:39.630076+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5075,"log_dups_size":0,"ondisk_log_size":5075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41530636,"num_objects":5070,"num_object_clones":0,"num_object_copies":20280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5073,"num_write_kb":40572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1796,"num_bytes_recovered":14680064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5109","reported_seq":5161,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.203852+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:39.203852+0000","last_peered":"2024-09-16T04:43:39.203852+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:39.203852+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5109,"log_dups_size":0,"ondisk_log_size":5109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41781784,"num_objects":5101,"num_object_clones":0,"num_object_copies":20404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5106,"num_write_kb":40825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5077","reported_seq":5151,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.243589+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:39.243589+0000","last_peered":"2024-09-16T04:43:39.243589+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:39.243589+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5077,"log_dups_size":0,"ondisk_log_size":5077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41571596,"num_objects":5075,"num_object_clones":0,"num_object_copies":20300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5075,"num_write_kb":40598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2875}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5005","reported_seq":5059,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.125819+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:39.125819+0000","last_peered":"2024-09-16T04:43:39.125819+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:39.125819+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40955794,"num_objects":5000,"num_object_clones":0,"num_object_copies":20000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5002,"num_write_kb":40004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4948","reported_seq":4983,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.167680+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:39.167680+0000","last_peered":"2024-09-16T04:43:39.167680+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:39.167680+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40524422,"num_objects":4947,"num_object_clones":0,"num_object_copies":19788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4947,"num_write_kb":39575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2819}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4978","reported_seq":5048,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.218492+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:39.218492+0000","last_peered":"2024-09-16T04:43:39.218492+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:39.218492+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4978,"log_dups_size":0,"ondisk_log_size":4978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40770182,"num_objects":4977,"num_object_clones":0,"num_object_copies":19908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4977,"num_write_kb":39815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2752}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5085","reported_seq":5134,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.179798+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:39.179798+0000","last_peered":"2024-09-16T04:43:39.179798+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:39.179798+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5085,"log_dups_size":0,"ondisk_log_size":5085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41620748,"num_objects":5081,"num_object_clones":0,"num_object_copies":20324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5083,"num_write_kb":40660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2847}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4958","reported_seq":5033,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.269949+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:39.269949+0000","last_peered":"2024-09-16T04:43:39.269949+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:39.269949+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4958,"log_dups_size":0,"ondisk_log_size":4958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40588556,"num_objects":4955,"num_object_clones":0,"num_object_copies":19820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4956,"num_write_kb":39645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2823}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4976","reported_seq":5038,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.554148+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:39.554148+0000","last_peered":"2024-09-16T04:43:39.554148+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:39.554148+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40719628,"num_objects":4971,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4974,"num_write_kb":39780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":658745000,"num_objects":80418,"num_object_clones":0,"num_object_copies":321672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80445,"num_write_kb":643442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1821,"num_bytes_recovered":14680064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1175523328,"data_stored":1175523328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80472,"ondisk_log_size":80472,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577788,"kb_used_data":321196,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93794052,"statfs":{"total":96636764160,"available":96045109248,"internally_reserved":0,"allocated":328904704,"data_stored":328733894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":262729423},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572848,"kb_used_data":320928,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93798992,"statfs":{"total":96636764160,"available":96050167808,"internally_reserved":0,"allocated":328630272,"data_stored":328459462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354788,"kb_used_data":182868,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":94017052,"statfs":{"total":96636764160,"available":96273461248,"internally_reserved":0,"allocated":187256832,"data_stored":187086022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":176025295},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570108,"kb_used_data":321900,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93801732,"statfs":{"total":96636764160,"available":96052973568,"internally_reserved":0,"allocated":329625600,"data_stored":329454790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":254144206},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":329654272,"data_stored":329654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":187392000,"data_stored":187392000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":329297920,"data_stored":329297920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":329179136,"data_stored":329179136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:40.816 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:41.288 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 167 20 80960 80940 3.78545 2.14844 0.057679 0.0412647 2024-09-16T04:43:41.422 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:41.423 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:41.443 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-16T04:43:40.609168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658745000,"num_objects":80418,"num_object_clones":0,"num_object_copies":321672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80445,"num_write_kb":643442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1821,"num_bytes_recovered":14680064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80472,"ondisk_log_size":80472,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2075532,"kb_used_data":1146892,"kb_used_omap":14,"kb_used_meta":928561,"kb_avail":375411828,"statfs":{"total":386547056640,"available":384421711872,"internally_reserved":0,"allocated":1174417408,"data_stored":1173734168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":950847025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,3,5,13,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36732928,"num_objects":4484,"num_object_clones":0,"num_object_copies":17936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4484,"num_write_kb":35872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":1589248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003135"},"pg_stats":[{"pgid":"1.f","version":"31'5035","reported_seq":5075,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.080250+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:39.080250+0000","last_peered":"2024-09-16T04:43:39.080250+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:39.080250+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41246720,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5035,"num_write_kb":40280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2816}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5086","reported_seq":5148,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.524819+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:39.524819+0000","last_peered":"2024-09-16T04:43:39.524819+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:39.524819+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41628940,"num_objects":5082,"num_object_clones":0,"num_object_copies":20328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5084,"num_write_kb":40668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2871}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5057","reported_seq":5128,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.280446+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:39.280446+0000","last_peered":"2024-09-16T04:43:39.280446+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:39.280446+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41400966,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5056,"num_write_kb":40438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2827}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5035","reported_seq":5097,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.509519+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:39.509519+0000","last_peered":"2024-09-16T04:43:39.509519+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:39.509519+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41211148,"num_objects":5031,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5033,"num_write_kb":40253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2802}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5035","reported_seq":5087,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.091952+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:39.091952+0000","last_peered":"2024-09-16T04:43:39.091952+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:39.091952+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41219340,"num_objects":5032,"num_object_clones":0,"num_object_copies":20128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5033,"num_write_kb":40261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4926","reported_seq":4966,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.148473+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:39.148473+0000","last_peered":"2024-09-16T04:43:39.148473+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:39.148473+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40353792,"num_objects":4926,"num_object_clones":0,"num_object_copies":19704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4926,"num_write_kb":39408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5087","reported_seq":5148,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.370320+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:39.370320+0000","last_peered":"2024-09-16T04:43:39.370320+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:39.370320+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41620748,"num_objects":5081,"num_object_clones":0,"num_object_copies":20324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":40660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2855}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5075","reported_seq":8724,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.630076+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:39.630076+0000","last_peered":"2024-09-16T04:43:39.630076+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:39.630076+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5075,"log_dups_size":0,"ondisk_log_size":5075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41530636,"num_objects":5070,"num_object_clones":0,"num_object_copies":20280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5073,"num_write_kb":40572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1796,"num_bytes_recovered":14680064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5109","reported_seq":5161,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.203852+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:39.203852+0000","last_peered":"2024-09-16T04:43:39.203852+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:39.203852+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5109,"log_dups_size":0,"ondisk_log_size":5109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41781784,"num_objects":5101,"num_object_clones":0,"num_object_copies":20404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5106,"num_write_kb":40825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5077","reported_seq":5151,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.243589+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:39.243589+0000","last_peered":"2024-09-16T04:43:39.243589+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:39.243589+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5077,"log_dups_size":0,"ondisk_log_size":5077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41571596,"num_objects":5075,"num_object_clones":0,"num_object_copies":20300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5075,"num_write_kb":40598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2875}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5005","reported_seq":5059,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.125819+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:39.125819+0000","last_peered":"2024-09-16T04:43:39.125819+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:39.125819+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40955794,"num_objects":5000,"num_object_clones":0,"num_object_copies":20000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5002,"num_write_kb":40004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4948","reported_seq":4983,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.167680+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:39.167680+0000","last_peered":"2024-09-16T04:43:39.167680+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:39.167680+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40524422,"num_objects":4947,"num_object_clones":0,"num_object_copies":19788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4947,"num_write_kb":39575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2819}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4978","reported_seq":5048,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.218492+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:39.218492+0000","last_peered":"2024-09-16T04:43:39.218492+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:39.218492+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4978,"log_dups_size":0,"ondisk_log_size":4978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40770182,"num_objects":4977,"num_object_clones":0,"num_object_copies":19908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4977,"num_write_kb":39815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2752}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5085","reported_seq":5134,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.179798+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:39.179798+0000","last_peered":"2024-09-16T04:43:39.179798+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:39.179798+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5085,"log_dups_size":0,"ondisk_log_size":5085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41620748,"num_objects":5081,"num_object_clones":0,"num_object_copies":20324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5083,"num_write_kb":40660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2847}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4958","reported_seq":5033,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.269949+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:39.269949+0000","last_peered":"2024-09-16T04:43:39.269949+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:39.269949+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4958,"log_dups_size":0,"ondisk_log_size":4958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40588556,"num_objects":4955,"num_object_clones":0,"num_object_copies":19820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4956,"num_write_kb":39645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2823}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4976","reported_seq":5038,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.554148+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:39.554148+0000","last_peered":"2024-09-16T04:43:39.554148+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:39.554148+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40719628,"num_objects":4971,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4974,"num_write_kb":39780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":658745000,"num_objects":80418,"num_object_clones":0,"num_object_copies":321672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80445,"num_write_kb":643442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1821,"num_bytes_recovered":14680064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1175523328,"data_stored":1175523328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80472,"ondisk_log_size":80472,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577788,"kb_used_data":321196,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93794052,"statfs":{"total":96636764160,"available":96045109248,"internally_reserved":0,"allocated":328904704,"data_stored":328733894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":262729423},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572848,"kb_used_data":320928,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93798992,"statfs":{"total":96636764160,"available":96050167808,"internally_reserved":0,"allocated":328630272,"data_stored":328459462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354788,"kb_used_data":182868,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":94017052,"statfs":{"total":96636764160,"available":96273461248,"internally_reserved":0,"allocated":187256832,"data_stored":187086022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":176025295},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570108,"kb_used_data":321900,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93801732,"statfs":{"total":96636764160,"available":96052973568,"internally_reserved":0,"allocated":329625600,"data_stored":329454790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":254144206},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":329654272,"data_stored":329654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":187392000,"data_stored":187392000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":329297920,"data_stored":329297920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":329179136,"data_stored":329179136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:41.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:43:41.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5035", 'reported_seq': 5075, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.080250+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:43:39.080250+0000', 'last_peered': '2024-09-16T04:43:39.080250+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:43:39.080250+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5035, 'log_dups_size': 0, 'ondisk_log_size': 5035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41246720, 'num_objects': 5035, 'num_object_clones': 0, 'num_object_copies': 20140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5035, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5035, 'num_write_kb': 40280, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2816}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:41.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:43:41.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5086", 'reported_seq': 5148, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.524819+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:43:39.524819+0000', 'last_peered': '2024-09-16T04:43:39.524819+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:43:39.524819+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5086, 'log_dups_size': 0, 'ondisk_log_size': 5086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41628940, 'num_objects': 5082, 'num_object_clones': 0, 'num_object_copies': 20328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5082, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5084, 'num_write_kb': 40668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2871}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:41.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:43:41.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5057", 'reported_seq': 5128, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.280446+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:43:39.280446+0000', 'last_peered': '2024-09-16T04:43:39.280446+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:43:39.280446+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5057, 'log_dups_size': 0, 'ondisk_log_size': 5057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41400966, 'num_objects': 5054, 'num_object_clones': 0, 'num_object_copies': 20216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5056, 'num_write_kb': 40438, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2827}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:41.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:43:41.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5035", 'reported_seq': 5097, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.509519+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:43:39.509519+0000', 'last_peered': '2024-09-16T04:43:39.509519+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:43:39.509519+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5035, 'log_dups_size': 0, 'ondisk_log_size': 5035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41211148, 'num_objects': 5031, 'num_object_clones': 0, 'num_object_copies': 20124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5033, 'num_write_kb': 40253, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2802}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:41.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:43:41.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5035", 'reported_seq': 5087, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.091952+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:43:39.091952+0000', 'last_peered': '2024-09-16T04:43:39.091952+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:43:39.091952+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5035, 'log_dups_size': 0, 'ondisk_log_size': 5035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41219340, 'num_objects': 5032, 'num_object_clones': 0, 'num_object_copies': 20128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5033, 'num_write_kb': 40261, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2749}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:41.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:43:41.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4926", 'reported_seq': 4966, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.148473+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:43:39.148473+0000', 'last_peered': '2024-09-16T04:43:39.148473+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:43:39.148473+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4926, 'log_dups_size': 0, 'ondisk_log_size': 4926, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40353792, 'num_objects': 4926, 'num_object_clones': 0, 'num_object_copies': 19704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4926, 'num_write_kb': 39408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2718}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:41.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:43:41.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5087", 'reported_seq': 5148, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.370320+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:43:39.370320+0000', 'last_peered': '2024-09-16T04:43:39.370320+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:43:39.370320+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5087, 'log_dups_size': 0, 'ondisk_log_size': 5087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41620748, 'num_objects': 5081, 'num_object_clones': 0, 'num_object_copies': 20324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5081, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5085, 'num_write_kb': 40660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2855}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:41.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:43:41.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5075", 'reported_seq': 8724, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.630076+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:43:39.630076+0000', 'last_peered': '2024-09-16T04:43:39.630076+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:43:39.630076+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5075, 'log_dups_size': 0, 'ondisk_log_size': 5075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41530636, 'num_objects': 5070, 'num_object_clones': 0, 'num_object_copies': 20280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 446, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5070, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5073, 'num_write_kb': 40572, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1796, 'num_bytes_recovered': 14680064, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4624}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:41.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:43:41.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5109", 'reported_seq': 5161, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.203852+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:43:39.203852+0000', 'last_peered': '2024-09-16T04:43:39.203852+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:43:39.203852+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5109, 'log_dups_size': 0, 'ondisk_log_size': 5109, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41781784, 'num_objects': 5101, 'num_object_clones': 0, 'num_object_copies': 20404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5101, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5106, 'num_write_kb': 40825, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2761}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:41.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:43:41.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5077", 'reported_seq': 5151, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.243589+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:43:39.243589+0000', 'last_peered': '2024-09-16T04:43:39.243589+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:43:39.243589+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5077, 'log_dups_size': 0, 'ondisk_log_size': 5077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41571596, 'num_objects': 5075, 'num_object_clones': 0, 'num_object_copies': 20300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5075, 'num_write_kb': 40598, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2875}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:41.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:43:41.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5005", 'reported_seq': 5059, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.125819+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:43:39.125819+0000', 'last_peered': '2024-09-16T04:43:39.125819+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:43:39.125819+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5005, 'log_dups_size': 0, 'ondisk_log_size': 5005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40955794, 'num_objects': 5000, 'num_object_clones': 0, 'num_object_copies': 20000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5000, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5002, 'num_write_kb': 40004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2811}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:41.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:43:41.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4948", 'reported_seq': 4983, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.167680+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:43:39.167680+0000', 'last_peered': '2024-09-16T04:43:39.167680+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:43:39.167680+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4948, 'log_dups_size': 0, 'ondisk_log_size': 4948, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40524422, 'num_objects': 4947, 'num_object_clones': 0, 'num_object_copies': 19788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4947, 'num_write_kb': 39575, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2819}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:41.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:43:41.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4978", 'reported_seq': 5048, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.218492+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:43:39.218492+0000', 'last_peered': '2024-09-16T04:43:39.218492+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:43:39.218492+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4978, 'log_dups_size': 0, 'ondisk_log_size': 4978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40770182, 'num_objects': 4977, 'num_object_clones': 0, 'num_object_copies': 19908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4977, 'num_write_kb': 39815, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2752}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:41.447 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:43:41.447 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5085", 'reported_seq': 5134, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.179798+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:43:39.179798+0000', 'last_peered': '2024-09-16T04:43:39.179798+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:43:39.179798+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5085, 'log_dups_size': 0, 'ondisk_log_size': 5085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41620748, 'num_objects': 5081, 'num_object_clones': 0, 'num_object_copies': 20324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5081, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5083, 'num_write_kb': 40660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2847}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:41.447 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:43:41.447 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4958", 'reported_seq': 5033, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.269949+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:43:39.269949+0000', 'last_peered': '2024-09-16T04:43:39.269949+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:43:39.269949+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4958, 'log_dups_size': 0, 'ondisk_log_size': 4958, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40588556, 'num_objects': 4955, 'num_object_clones': 0, 'num_object_copies': 19820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4955, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4956, 'num_write_kb': 39645, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2823}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:41.447 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:43:41.447 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4976", 'reported_seq': 5038, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:39.554148+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:43:39.554148+0000', 'last_peered': '2024-09-16T04:43:39.554148+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:43:39.554148+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 4976, 'log_dups_size': 0, 'ondisk_log_size': 4976, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40719628, 'num_objects': 4971, 'num_object_clones': 0, 'num_object_copies': 19884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4974, 'num_write_kb': 39780, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2732}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:41.447 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:43:42.149 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:42.150 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33761/321672 objects degraded (10.495%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":234,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":80418,"data_bytes":658745000,"bytes_used":2125344768,"bytes_avail":384421711872,"bytes_total":386547056640,"degraded_objects":33761,"degraded_total":321672,"degraded_ratio":0.10495473650177821,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":132402,"recovering_keys_per_sec":0,"num_objects_recovered":194,"num_bytes_recovered":1589248,"num_keys_recovered":0,"write_bytes_sec":3060277,"read_op_per_sec":0,"write_op_per_sec":373},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:43:42.182 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:43:42.182 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:42.289 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 168 20 81227 81207 3.77533 2.08594 0.034949 0.0413695 2024-09-16T04:43:42.731 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:42.731 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:42.763 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-09-16T04:43:42.609744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658745000,"num_objects":80418,"num_object_clones":0,"num_object_copies":321672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80445,"num_write_kb":643442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1821,"num_bytes_recovered":14680064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80472,"ondisk_log_size":80472,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2103860,"kb_used_data":1153012,"kb_used_omap":14,"kb_used_meta":950769,"kb_avail":375383500,"statfs":{"total":386547056640,"available":384392704000,"internally_reserved":0,"allocated":1180684288,"data_stored":1180001048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":973588017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,5,7,17,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24289280,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2965,"num_write_kb":23720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1097728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003229"},"pg_stats":[{"pgid":"1.f","version":"31'5035","reported_seq":5075,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.080250+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:39.080250+0000","last_peered":"2024-09-16T04:43:39.080250+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:39.080250+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41246720,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5035,"num_write_kb":40280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2816}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5086","reported_seq":5148,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.524819+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:39.524819+0000","last_peered":"2024-09-16T04:43:39.524819+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:39.524819+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41628940,"num_objects":5082,"num_object_clones":0,"num_object_copies":20328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5084,"num_write_kb":40668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2871}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5057","reported_seq":5128,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.280446+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:39.280446+0000","last_peered":"2024-09-16T04:43:39.280446+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:39.280446+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41400966,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5056,"num_write_kb":40438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2827}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5035","reported_seq":5097,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.509519+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:39.509519+0000","last_peered":"2024-09-16T04:43:39.509519+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:39.509519+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41211148,"num_objects":5031,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5033,"num_write_kb":40253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2802}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5035","reported_seq":5087,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.091952+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:39.091952+0000","last_peered":"2024-09-16T04:43:39.091952+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:39.091952+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41219340,"num_objects":5032,"num_object_clones":0,"num_object_copies":20128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5033,"num_write_kb":40261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4926","reported_seq":4966,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.148473+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:39.148473+0000","last_peered":"2024-09-16T04:43:39.148473+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:39.148473+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40353792,"num_objects":4926,"num_object_clones":0,"num_object_copies":19704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4926,"num_write_kb":39408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5087","reported_seq":5148,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.370320+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:39.370320+0000","last_peered":"2024-09-16T04:43:39.370320+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:39.370320+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41620748,"num_objects":5081,"num_object_clones":0,"num_object_copies":20324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":40660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2855}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5075","reported_seq":8724,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.630076+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:39.630076+0000","last_peered":"2024-09-16T04:43:39.630076+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:39.630076+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5075,"log_dups_size":0,"ondisk_log_size":5075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41530636,"num_objects":5070,"num_object_clones":0,"num_object_copies":20280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5073,"num_write_kb":40572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1796,"num_bytes_recovered":14680064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5109","reported_seq":5161,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.203852+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:39.203852+0000","last_peered":"2024-09-16T04:43:39.203852+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:39.203852+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5109,"log_dups_size":0,"ondisk_log_size":5109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41781784,"num_objects":5101,"num_object_clones":0,"num_object_copies":20404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5106,"num_write_kb":40825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5077","reported_seq":5151,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.243589+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:39.243589+0000","last_peered":"2024-09-16T04:43:39.243589+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:39.243589+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5077,"log_dups_size":0,"ondisk_log_size":5077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41571596,"num_objects":5075,"num_object_clones":0,"num_object_copies":20300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5075,"num_write_kb":40598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2875}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5005","reported_seq":5059,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.125819+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:39.125819+0000","last_peered":"2024-09-16T04:43:39.125819+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:39.125819+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40955794,"num_objects":5000,"num_object_clones":0,"num_object_copies":20000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5002,"num_write_kb":40004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4948","reported_seq":4983,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.167680+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:39.167680+0000","last_peered":"2024-09-16T04:43:39.167680+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:39.167680+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40524422,"num_objects":4947,"num_object_clones":0,"num_object_copies":19788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4947,"num_write_kb":39575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2819}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4978","reported_seq":5048,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.218492+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:39.218492+0000","last_peered":"2024-09-16T04:43:39.218492+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:39.218492+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4978,"log_dups_size":0,"ondisk_log_size":4978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40770182,"num_objects":4977,"num_object_clones":0,"num_object_copies":19908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4977,"num_write_kb":39815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2752}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5085","reported_seq":5134,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.179798+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:39.179798+0000","last_peered":"2024-09-16T04:43:39.179798+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:39.179798+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5085,"log_dups_size":0,"ondisk_log_size":5085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41620748,"num_objects":5081,"num_object_clones":0,"num_object_copies":20324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5083,"num_write_kb":40660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2847}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4958","reported_seq":5033,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.269949+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:39.269949+0000","last_peered":"2024-09-16T04:43:39.269949+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:39.269949+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4958,"log_dups_size":0,"ondisk_log_size":4958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40588556,"num_objects":4955,"num_object_clones":0,"num_object_copies":19820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4956,"num_write_kb":39645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2823}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4976","reported_seq":5038,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.554148+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:39.554148+0000","last_peered":"2024-09-16T04:43:39.554148+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:39.554148+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40719628,"num_objects":4971,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4974,"num_write_kb":39780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":658745000,"num_objects":80418,"num_object_clones":0,"num_object_copies":321672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80445,"num_write_kb":643442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1821,"num_bytes_recovered":14680064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1181888512,"data_stored":1181888512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80472,"ondisk_log_size":80472,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577788,"kb_used_data":321196,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93794052,"statfs":{"total":96636764160,"available":96045109248,"internally_reserved":0,"allocated":328904704,"data_stored":328733894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":262729423},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572848,"kb_used_data":320928,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93798992,"statfs":{"total":96636764160,"available":96050167808,"internally_reserved":0,"allocated":328630272,"data_stored":328459462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051612,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383116,"kb_used_data":188988,"kb_used_omap":4,"kb_used_meta":194107,"kb_avail":93988724,"statfs":{"total":96636764160,"available":96244453376,"internally_reserved":0,"allocated":193523712,"data_stored":193352902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":198766287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,10,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570108,"kb_used_data":321900,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93801732,"statfs":{"total":96636764160,"available":96052973568,"internally_reserved":0,"allocated":329625600,"data_stored":329454790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":254144206},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":329654272,"data_stored":329654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":193757184,"data_stored":193757184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":329297920,"data_stored":329297920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":329179136,"data_stored":329179136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:43.289 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 169 20 81515 81495 3.76631 2.25 0.05661 0.0414701 2024-09-16T04:43:44.288 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 170 20 81808 81788 3.75762 2.28906 0.0159515 0.0415478 2024-09-16T04:43:45.289 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 171 20 82111 82091 3.74949 2.36719 0.0562523 0.041659 2024-09-16T04:43:45.765 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:46.289 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 172 20 82397 82377 3.74068 2.23438 0.0925138 0.0417563 2024-09-16T04:43:46.359 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:46.360 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:46.391 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-09-16T04:43:44.610223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":667149992,"num_objects":81444,"num_object_clones":0,"num_object_copies":325776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81471,"num_write_kb":651650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1821,"num_bytes_recovered":14680064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81498,"ondisk_log_size":81498,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2107216,"kb_used_data":1164684,"kb_used_omap":14,"kb_used_meta":942449,"kb_avail":375380144,"statfs":{"total":386547056640,"available":384389267456,"internally_reserved":0,"allocated":1192636416,"data_stored":1191953176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":965068337},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,10,10,10,25,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32694272,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3991,"num_write_kb":31928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1097728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003146"},"pg_stats":[{"pgid":"1.f","version":"31'5138","reported_seq":5178,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.174365+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:44.174365+0000","last_peered":"2024-09-16T04:43:44.174365+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:44.174365+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5138,"log_dups_size":0,"ondisk_log_size":5138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42090496,"num_objects":5138,"num_object_clones":0,"num_object_copies":20552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5138,"num_write_kb":41104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2919}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5086","reported_seq":5148,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.524819+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:39.524819+0000","last_peered":"2024-09-16T04:43:39.524819+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:39.524819+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41628940,"num_objects":5082,"num_object_clones":0,"num_object_copies":20328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5084,"num_write_kb":40668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2871}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5147","reported_seq":5214,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.356523+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:44.356523+0000","last_peered":"2024-09-16T04:43:44.356523+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:44.356523+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5147,"log_dups_size":0,"ondisk_log_size":5147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42138246,"num_objects":5144,"num_object_clones":0,"num_object_copies":20576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5146,"num_write_kb":41158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2917}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5035","reported_seq":5097,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.509519+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:39.509519+0000","last_peered":"2024-09-16T04:43:39.509519+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:39.509519+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41211148,"num_objects":5031,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5033,"num_write_kb":40253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2802}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5127","reported_seq":5179,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.242845+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:44.242845+0000","last_peered":"2024-09-16T04:43:44.242845+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:44.242845+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5127,"log_dups_size":0,"ondisk_log_size":5127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41973004,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5125,"num_write_kb":40997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5011","reported_seq":5051,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.159447+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:44.159447+0000","last_peered":"2024-09-16T04:43:44.159447+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:44.159447+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5011,"log_dups_size":0,"ondisk_log_size":5011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41050112,"num_objects":5011,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5011,"num_write_kb":40088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2803}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5087","reported_seq":5148,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.370320+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:39.370320+0000","last_peered":"2024-09-16T04:43:39.370320+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:39.370320+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41620748,"num_objects":5081,"num_object_clones":0,"num_object_copies":20324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":40660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2855}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5075","reported_seq":8724,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.630076+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:39.630076+0000","last_peered":"2024-09-16T04:43:39.630076+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:39.630076+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5075,"log_dups_size":0,"ondisk_log_size":5075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41530636,"num_objects":5070,"num_object_clones":0,"num_object_copies":20280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5073,"num_write_kb":40572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1796,"num_bytes_recovered":14680064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5192","reported_seq":5245,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.250792+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:44.250792+0000","last_peered":"2024-09-16T04:43:44.250792+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:44.250792+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5192,"log_dups_size":0,"ondisk_log_size":5192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42461720,"num_objects":5184,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5189,"num_write_kb":41489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2844}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5167","reported_seq":5241,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:43.995700+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:43.995700+0000","last_peered":"2024-09-16T04:43:43.995700+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:43.995700+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5167,"log_dups_size":0,"ondisk_log_size":5167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42308876,"num_objects":5165,"num_object_clones":0,"num_object_copies":20660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5165,"num_write_kb":41318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2965}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5090","reported_seq":5144,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.172048+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:44.172048+0000","last_peered":"2024-09-16T04:43:44.172048+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:44.172048+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41652114,"num_objects":5085,"num_object_clones":0,"num_object_copies":20340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5087,"num_write_kb":40684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5044","reported_seq":5080,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.202478+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:44.202478+0000","last_peered":"2024-09-16T04:43:44.202478+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:44.202478+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41310854,"num_objects":5043,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5043,"num_write_kb":40343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2915}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5095","reported_seq":5165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.350079+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:44.350079+0000","last_peered":"2024-09-16T04:43:44.350079+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:44.350079+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5095,"log_dups_size":0,"ondisk_log_size":5095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41728646,"num_objects":5094,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5094,"num_write_kb":40751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2869}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5183","reported_seq":5233,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.141585+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:44.141585+0000","last_peered":"2024-09-16T04:43:44.141585+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:44.141585+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5183,"log_dups_size":0,"ondisk_log_size":5183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42423564,"num_objects":5179,"num_object_clones":0,"num_object_copies":20716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5181,"num_write_kb":41444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2945}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5045","reported_seq":5120,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.308420+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:44.308420+0000","last_peered":"2024-09-16T04:43:44.308420+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:44.308420+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5045,"log_dups_size":0,"ondisk_log_size":5045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41301260,"num_objects":5042,"num_object_clones":0,"num_object_copies":20168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5043,"num_write_kb":40341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4976","reported_seq":5038,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:39.554148+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:39.554148+0000","last_peered":"2024-09-16T04:43:39.554148+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:39.554148+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40719628,"num_objects":4971,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4974,"num_write_kb":39780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":667149992,"num_objects":81444,"num_object_clones":0,"num_object_copies":325776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81471,"num_write_kb":651650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1821,"num_bytes_recovered":14680064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1193680896,"data_stored":1193680896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81498,"ondisk_log_size":81498,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571208,"kb_used_data":327096,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93800632,"statfs":{"total":96636764160,"available":96051847168,"internally_reserved":0,"allocated":334946304,"data_stored":334775494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249949903},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,0,4,9,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582784,"kb_used_data":326700,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93789056,"statfs":{"total":96636764160,"available":96039993344,"internally_reserved":0,"allocated":334540800,"data_stored":334369990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,5,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051612,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383116,"kb_used_data":188988,"kb_used_omap":4,"kb_used_meta":194107,"kb_avail":93988724,"statfs":{"total":96636764160,"available":96244453376,"internally_reserved":0,"allocated":193523712,"data_stored":193352902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":198766287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,10,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570108,"kb_used_data":321900,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93801732,"statfs":{"total":96636764160,"available":96052973568,"internally_reserved":0,"allocated":329625600,"data_stored":329454790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":254144206},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":329654272,"data_stored":329654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":193757184,"data_stored":193757184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":335179776,"data_stored":335179776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":335089664,"data_stored":335089664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:46.393 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:46.965 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:46.966 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:46.990 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-09-16T04:43:46.610793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":670647976,"num_objects":81871,"num_object_clones":0,"num_object_copies":327484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33686,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81898,"num_write_kb":655066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1896,"num_bytes_recovered":15294464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81925,"ondisk_log_size":81925,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2127184,"kb_used_data":1176332,"kb_used_omap":14,"kb_used_meta":950769,"kb_avail":375360176,"statfs":{"total":386547056640,"available":384368820224,"internally_reserved":0,"allocated":1204563968,"data_stored":1203880728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":973588017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,9,9,10,19,13,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27754496,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3388,"num_write_kb":27104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003187"},"pg_stats":[{"pgid":"1.f","version":"31'5138","reported_seq":5178,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.174365+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:44.174365+0000","last_peered":"2024-09-16T04:43:44.174365+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:44.174365+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5138,"log_dups_size":0,"ondisk_log_size":5138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42090496,"num_objects":5138,"num_object_clones":0,"num_object_copies":20552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5138,"num_write_kb":41104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2919}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5173","reported_seq":5235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.570766+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:44.570766+0000","last_peered":"2024-09-16T04:43:44.570766+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:44.570766+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5173,"log_dups_size":0,"ondisk_log_size":5173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42341644,"num_objects":5169,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5171,"num_write_kb":41364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5147","reported_seq":5214,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.356523+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:44.356523+0000","last_peered":"2024-09-16T04:43:44.356523+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:44.356523+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5147,"log_dups_size":0,"ondisk_log_size":5147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42138246,"num_objects":5144,"num_object_clones":0,"num_object_copies":20576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5146,"num_write_kb":41158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2917}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5131","reported_seq":5193,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.626139+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:44.626139+0000","last_peered":"2024-09-16T04:43:44.626139+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:44.626139+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5131,"log_dups_size":0,"ondisk_log_size":5131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41997580,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5129,"num_write_kb":41021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2898}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5127","reported_seq":5179,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.242845+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:44.242845+0000","last_peered":"2024-09-16T04:43:44.242845+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:44.242845+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5127,"log_dups_size":0,"ondisk_log_size":5127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41973004,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5125,"num_write_kb":40997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5011","reported_seq":5051,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.159447+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:44.159447+0000","last_peered":"2024-09-16T04:43:44.159447+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:44.159447+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5011,"log_dups_size":0,"ondisk_log_size":5011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41050112,"num_objects":5011,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5011,"num_write_kb":40088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2803}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5177","reported_seq":5238,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.623082+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:44.623082+0000","last_peered":"2024-09-16T04:43:44.623082+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:44.623082+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5177,"log_dups_size":0,"ondisk_log_size":5177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42358028,"num_objects":5171,"num_object_clones":0,"num_object_copies":20684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":41380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5162","reported_seq":8961,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.673310+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:44.673310+0000","last_peered":"2024-09-16T04:43:44.673310+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:44.673310+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5162,"log_dups_size":0,"ondisk_log_size":5162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42243340,"num_objects":5157,"num_object_clones":0,"num_object_copies":20628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5160,"num_write_kb":41268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1871,"num_bytes_recovered":15294464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5192","reported_seq":5245,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.250792+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:44.250792+0000","last_peered":"2024-09-16T04:43:44.250792+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:44.250792+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5192,"log_dups_size":0,"ondisk_log_size":5192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42461720,"num_objects":5184,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5189,"num_write_kb":41489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2844}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5167","reported_seq":5241,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:43.995700+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:43.995700+0000","last_peered":"2024-09-16T04:43:43.995700+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:43.995700+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5167,"log_dups_size":0,"ondisk_log_size":5167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42308876,"num_objects":5165,"num_object_clones":0,"num_object_copies":20660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5165,"num_write_kb":41318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2965}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5090","reported_seq":5144,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.172048+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:44.172048+0000","last_peered":"2024-09-16T04:43:44.172048+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:44.172048+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41652114,"num_objects":5085,"num_object_clones":0,"num_object_copies":20340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5087,"num_write_kb":40684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5044","reported_seq":5080,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.202478+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:44.202478+0000","last_peered":"2024-09-16T04:43:44.202478+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:44.202478+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41310854,"num_objects":5043,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5043,"num_write_kb":40343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2915}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5095","reported_seq":5165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.350079+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:44.350079+0000","last_peered":"2024-09-16T04:43:44.350079+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:44.350079+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5095,"log_dups_size":0,"ondisk_log_size":5095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41728646,"num_objects":5094,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5094,"num_write_kb":40751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2869}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5183","reported_seq":5233,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.141585+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:44.141585+0000","last_peered":"2024-09-16T04:43:44.141585+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:44.141585+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5183,"log_dups_size":0,"ondisk_log_size":5183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42423564,"num_objects":5179,"num_object_clones":0,"num_object_copies":20716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5181,"num_write_kb":41444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2945}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5045","reported_seq":5120,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.308420+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:44.308420+0000","last_peered":"2024-09-16T04:43:44.308420+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:44.308420+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5045,"log_dups_size":0,"ondisk_log_size":5045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41301260,"num_objects":5042,"num_object_clones":0,"num_object_copies":20168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5043,"num_write_kb":40341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5043","reported_seq":5105,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.559411+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:44.559411+0000","last_peered":"2024-09-16T04:43:44.559411+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:44.559411+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41268492,"num_objects":5038,"num_object_clones":0,"num_object_copies":20152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5041,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2799}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":670647976,"num_objects":81871,"num_object_clones":0,"num_object_copies":327484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33686,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81898,"num_write_kb":655066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1896,"num_bytes_recovered":15294464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1205805056,"data_stored":1205805056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81925,"ondisk_log_size":81925,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571208,"kb_used_data":327096,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93800632,"statfs":{"total":96636764160,"available":96051847168,"internally_reserved":0,"allocated":334946304,"data_stored":334775494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249949903},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,0,4,9,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582784,"kb_used_data":326700,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93789056,"statfs":{"total":96636764160,"available":96039993344,"internally_reserved":0,"allocated":334540800,"data_stored":334369990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,5,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393296,"kb_used_data":195008,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93978544,"statfs":{"total":96636764160,"available":96234029056,"internally_reserved":0,"allocated":199688192,"data_stored":199517382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":203026127},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579896,"kb_used_data":327528,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93791944,"statfs":{"total":96636764160,"available":96042950656,"internally_reserved":0,"allocated":335388672,"data_stored":335217862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":258404046},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,1,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":335564800,"data_stored":335564800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":199970816,"data_stored":199970816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":335179776,"data_stored":335179776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":335089664,"data_stored":335089664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:46.991 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:43:46.992 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5138", 'reported_seq': 5178, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:44.174365+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:43:44.174365+0000', 'last_peered': '2024-09-16T04:43:44.174365+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:43:44.174365+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5138, 'log_dups_size': 0, 'ondisk_log_size': 5138, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42090496, 'num_objects': 5138, 'num_object_clones': 0, 'num_object_copies': 20552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5138, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5138, 'num_write_kb': 41104, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2919}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:46.992 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:43:46.992 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5173", 'reported_seq': 5235, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:44.570766+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:43:44.570766+0000', 'last_peered': '2024-09-16T04:43:44.570766+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:43:44.570766+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5173, 'log_dups_size': 0, 'ondisk_log_size': 5173, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42341644, 'num_objects': 5169, 'num_object_clones': 0, 'num_object_copies': 20676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5169, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5171, 'num_write_kb': 41364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2958}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:46.992 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:43:46.992 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5147", 'reported_seq': 5214, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:44.356523+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:43:44.356523+0000', 'last_peered': '2024-09-16T04:43:44.356523+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:43:44.356523+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5147, 'log_dups_size': 0, 'ondisk_log_size': 5147, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42138246, 'num_objects': 5144, 'num_object_clones': 0, 'num_object_copies': 20576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5144, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5146, 'num_write_kb': 41158, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2917}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:46.992 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:43:46.992 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5131", 'reported_seq': 5193, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:44.626139+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:43:44.626139+0000', 'last_peered': '2024-09-16T04:43:44.626139+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:43:44.626139+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5131, 'log_dups_size': 0, 'ondisk_log_size': 5131, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41997580, 'num_objects': 5127, 'num_object_clones': 0, 'num_object_copies': 20508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5129, 'num_write_kb': 41021, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2898}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:46.992 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:43:46.992 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5127", 'reported_seq': 5179, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:44.242845+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:43:44.242845+0000', 'last_peered': '2024-09-16T04:43:44.242845+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:43:44.242845+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5127, 'log_dups_size': 0, 'ondisk_log_size': 5127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41973004, 'num_objects': 5124, 'num_object_clones': 0, 'num_object_copies': 20496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5125, 'num_write_kb': 40997, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2841}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:46.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:43:46.993 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5011", 'reported_seq': 5051, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:44.159447+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:43:44.159447+0000', 'last_peered': '2024-09-16T04:43:44.159447+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:43:44.159447+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5011, 'log_dups_size': 0, 'ondisk_log_size': 5011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41050112, 'num_objects': 5011, 'num_object_clones': 0, 'num_object_copies': 20044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5011, 'num_write_kb': 40088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2803}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:46.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:43:46.993 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5177", 'reported_seq': 5238, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:44.623082+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:43:44.623082+0000', 'last_peered': '2024-09-16T04:43:44.623082+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:43:44.623082+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5177, 'log_dups_size': 0, 'ondisk_log_size': 5177, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42358028, 'num_objects': 5171, 'num_object_clones': 0, 'num_object_copies': 20684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5175, 'num_write_kb': 41380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2945}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:46.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:43:46.993 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5162", 'reported_seq': 8961, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:44.673310+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:43:44.673310+0000', 'last_peered': '2024-09-16T04:43:44.673310+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:43:44.673310+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5162, 'log_dups_size': 0, 'ondisk_log_size': 5162, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42243340, 'num_objects': 5157, 'num_object_clones': 0, 'num_object_copies': 20628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 371, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5160, 'num_write_kb': 41268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1871, 'num_bytes_recovered': 15294464, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4786}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:46.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:43:46.993 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5192", 'reported_seq': 5245, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:44.250792+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:43:44.250792+0000', 'last_peered': '2024-09-16T04:43:44.250792+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:43:44.250792+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5192, 'log_dups_size': 0, 'ondisk_log_size': 5192, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42461720, 'num_objects': 5184, 'num_object_clones': 0, 'num_object_copies': 20736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5189, 'num_write_kb': 41489, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2844}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:46.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:43:46.993 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5167", 'reported_seq': 5241, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:43.995700+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:43:43.995700+0000', 'last_peered': '2024-09-16T04:43:43.995700+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:43:43.995700+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5167, 'log_dups_size': 0, 'ondisk_log_size': 5167, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42308876, 'num_objects': 5165, 'num_object_clones': 0, 'num_object_copies': 20660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5165, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5165, 'num_write_kb': 41318, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2965}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:46.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:43:46.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5090", 'reported_seq': 5144, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:44.172048+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:43:44.172048+0000', 'last_peered': '2024-09-16T04:43:44.172048+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:43:44.172048+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5090, 'log_dups_size': 0, 'ondisk_log_size': 5090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41652114, 'num_objects': 5085, 'num_object_clones': 0, 'num_object_copies': 20340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5087, 'num_write_kb': 40684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2896}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:46.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:43:46.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5044", 'reported_seq': 5080, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:44.202478+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:43:44.202478+0000', 'last_peered': '2024-09-16T04:43:44.202478+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:43:44.202478+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5044, 'log_dups_size': 0, 'ondisk_log_size': 5044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41310854, 'num_objects': 5043, 'num_object_clones': 0, 'num_object_copies': 20172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5043, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5043, 'num_write_kb': 40343, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2915}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:46.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:43:46.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5095", 'reported_seq': 5165, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:44.350079+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:43:44.350079+0000', 'last_peered': '2024-09-16T04:43:44.350079+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:43:44.350079+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5095, 'log_dups_size': 0, 'ondisk_log_size': 5095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41728646, 'num_objects': 5094, 'num_object_clones': 0, 'num_object_copies': 20376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5094, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5094, 'num_write_kb': 40751, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2869}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:46.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:43:46.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5183", 'reported_seq': 5233, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:44.141585+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:43:44.141585+0000', 'last_peered': '2024-09-16T04:43:44.141585+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:43:44.141585+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5183, 'log_dups_size': 0, 'ondisk_log_size': 5183, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42423564, 'num_objects': 5179, 'num_object_clones': 0, 'num_object_copies': 20716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5181, 'num_write_kb': 41444, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2945}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:46.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:43:46.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5045", 'reported_seq': 5120, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:44.308420+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:43:44.308420+0000', 'last_peered': '2024-09-16T04:43:44.308420+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:43:44.308420+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5045, 'log_dups_size': 0, 'ondisk_log_size': 5045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41301260, 'num_objects': 5042, 'num_object_clones': 0, 'num_object_copies': 20168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5042, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5043, 'num_write_kb': 40341, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:46.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:43:46.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5043", 'reported_seq': 5105, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:44.559411+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:43:44.559411+0000', 'last_peered': '2024-09-16T04:43:44.559411+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:43:44.559411+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5043, 'log_dups_size': 0, 'ondisk_log_size': 5043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41268492, 'num_objects': 5038, 'num_object_clones': 0, 'num_object_copies': 20152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5041, 'num_write_kb': 40316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2799}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:46.995 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:43:47.289 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 173 20 82650 82630 3.73048 1.97656 0.05155 0.0418759 2024-09-16T04:43:47.790 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:47.791 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33686/327484 objects degraded (10.286%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":240,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":81871,"data_bytes":670647976,"bytes_used":2178236416,"bytes_avail":384368820224,"bytes_total":386547056640,"degraded_objects":33686,"degraded_total":327484,"degraded_ratio":0.10286304063709985,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":142639,"recovering_keys_per_sec":0,"num_objects_recovered":209,"num_bytes_recovered":1712128,"num_keys_recovered":0,"write_bytes_sec":2312260,"read_op_per_sec":0,"write_op_per_sec":282},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:43:47.820 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:43:47.821 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:48.289 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 174 20 82913 82893 3.72085 2.05469 0.103246 0.0419767 2024-09-16T04:43:48.372 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:48.376 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:48.397 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-09-16T04:43:46.610793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":670647976,"num_objects":81871,"num_object_clones":0,"num_object_copies":327484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33686,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81898,"num_write_kb":655066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1896,"num_bytes_recovered":15294464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81925,"ondisk_log_size":81925,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2127184,"kb_used_data":1176332,"kb_used_omap":14,"kb_used_meta":950769,"kb_avail":375360176,"statfs":{"total":386547056640,"available":384368820224,"internally_reserved":0,"allocated":1204563968,"data_stored":1203880728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":973588017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,9,9,10,19,13,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27754496,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3388,"num_write_kb":27104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003187"},"pg_stats":[{"pgid":"1.f","version":"31'5138","reported_seq":5178,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.174365+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:44.174365+0000","last_peered":"2024-09-16T04:43:44.174365+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:44.174365+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5138,"log_dups_size":0,"ondisk_log_size":5138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42090496,"num_objects":5138,"num_object_clones":0,"num_object_copies":20552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5138,"num_write_kb":41104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2919}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5173","reported_seq":5235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.570766+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:44.570766+0000","last_peered":"2024-09-16T04:43:44.570766+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:44.570766+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5173,"log_dups_size":0,"ondisk_log_size":5173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42341644,"num_objects":5169,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5171,"num_write_kb":41364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5147","reported_seq":5214,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.356523+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:44.356523+0000","last_peered":"2024-09-16T04:43:44.356523+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:44.356523+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5147,"log_dups_size":0,"ondisk_log_size":5147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42138246,"num_objects":5144,"num_object_clones":0,"num_object_copies":20576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5146,"num_write_kb":41158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2917}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5131","reported_seq":5193,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.626139+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:44.626139+0000","last_peered":"2024-09-16T04:43:44.626139+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:44.626139+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5131,"log_dups_size":0,"ondisk_log_size":5131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41997580,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5129,"num_write_kb":41021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2898}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5127","reported_seq":5179,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.242845+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:44.242845+0000","last_peered":"2024-09-16T04:43:44.242845+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:44.242845+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5127,"log_dups_size":0,"ondisk_log_size":5127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41973004,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5125,"num_write_kb":40997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5011","reported_seq":5051,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.159447+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:44.159447+0000","last_peered":"2024-09-16T04:43:44.159447+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:44.159447+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5011,"log_dups_size":0,"ondisk_log_size":5011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41050112,"num_objects":5011,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5011,"num_write_kb":40088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2803}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5177","reported_seq":5238,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.623082+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:44.623082+0000","last_peered":"2024-09-16T04:43:44.623082+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:44.623082+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5177,"log_dups_size":0,"ondisk_log_size":5177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42358028,"num_objects":5171,"num_object_clones":0,"num_object_copies":20684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":41380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5162","reported_seq":8961,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.673310+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:44.673310+0000","last_peered":"2024-09-16T04:43:44.673310+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:44.673310+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5162,"log_dups_size":0,"ondisk_log_size":5162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42243340,"num_objects":5157,"num_object_clones":0,"num_object_copies":20628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5160,"num_write_kb":41268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1871,"num_bytes_recovered":15294464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5192","reported_seq":5245,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.250792+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:44.250792+0000","last_peered":"2024-09-16T04:43:44.250792+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:44.250792+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5192,"log_dups_size":0,"ondisk_log_size":5192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42461720,"num_objects":5184,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5189,"num_write_kb":41489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2844}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5167","reported_seq":5241,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:43.995700+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:43.995700+0000","last_peered":"2024-09-16T04:43:43.995700+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:43.995700+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5167,"log_dups_size":0,"ondisk_log_size":5167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42308876,"num_objects":5165,"num_object_clones":0,"num_object_copies":20660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5165,"num_write_kb":41318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2965}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5090","reported_seq":5144,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.172048+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:44.172048+0000","last_peered":"2024-09-16T04:43:44.172048+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:44.172048+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41652114,"num_objects":5085,"num_object_clones":0,"num_object_copies":20340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5087,"num_write_kb":40684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5044","reported_seq":5080,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.202478+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:44.202478+0000","last_peered":"2024-09-16T04:43:44.202478+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:44.202478+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41310854,"num_objects":5043,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5043,"num_write_kb":40343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2915}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5095","reported_seq":5165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.350079+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:44.350079+0000","last_peered":"2024-09-16T04:43:44.350079+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:44.350079+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5095,"log_dups_size":0,"ondisk_log_size":5095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41728646,"num_objects":5094,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5094,"num_write_kb":40751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2869}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5183","reported_seq":5233,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.141585+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:44.141585+0000","last_peered":"2024-09-16T04:43:44.141585+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:44.141585+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5183,"log_dups_size":0,"ondisk_log_size":5183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42423564,"num_objects":5179,"num_object_clones":0,"num_object_copies":20716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5181,"num_write_kb":41444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2945}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5045","reported_seq":5120,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.308420+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:44.308420+0000","last_peered":"2024-09-16T04:43:44.308420+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:44.308420+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5045,"log_dups_size":0,"ondisk_log_size":5045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41301260,"num_objects":5042,"num_object_clones":0,"num_object_copies":20168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5043,"num_write_kb":40341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5043","reported_seq":5105,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:44.559411+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:44.559411+0000","last_peered":"2024-09-16T04:43:44.559411+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:44.559411+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41268492,"num_objects":5038,"num_object_clones":0,"num_object_copies":20152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5041,"num_write_kb":40316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2799}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":670647976,"num_objects":81871,"num_object_clones":0,"num_object_copies":327484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33686,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81898,"num_write_kb":655066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1896,"num_bytes_recovered":15294464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1205805056,"data_stored":1205805056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81925,"ondisk_log_size":81925,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571208,"kb_used_data":327096,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93800632,"statfs":{"total":96636764160,"available":96051847168,"internally_reserved":0,"allocated":334946304,"data_stored":334775494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249949903},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,0,4,9,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582784,"kb_used_data":326700,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93789056,"statfs":{"total":96636764160,"available":96039993344,"internally_reserved":0,"allocated":334540800,"data_stored":334369990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,5,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393296,"kb_used_data":195008,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93978544,"statfs":{"total":96636764160,"available":96234029056,"internally_reserved":0,"allocated":199688192,"data_stored":199517382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":203026127},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579896,"kb_used_data":327528,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93791944,"statfs":{"total":96636764160,"available":96042950656,"internally_reserved":0,"allocated":335388672,"data_stored":335217862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":258404046},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,1,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":335564800,"data_stored":335564800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":199970816,"data_stored":199970816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":335179776,"data_stored":335179776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":335089664,"data_stored":335089664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:49.289 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 175 20 83212 83192 3.71294 2.33594 0.0472157 0.0420689 2024-09-16T04:43:50.289 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 176 20 83514 83494 3.70525 2.35938 0.0871019 0.0421581 2024-09-16T04:43:51.289 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 177 20 83817 83797 3.69769 2.36719 0.0421848 0.0422434 2024-09-16T04:43:51.399 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:51.988 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:51.988 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:52.024 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-16T04:43:50.611778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":682272424,"num_objects":83290,"num_object_clones":0,"num_object_copies":333160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83317,"num_write_kb":666418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":15917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83344,"ondisk_log_size":83344,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2234072,"kb_used_data":1192984,"kb_used_omap":14,"kb_used_meta":1041009,"kb_avail":375253288,"statfs":{"total":386547056640,"available":384259366912,"internally_reserved":0,"allocated":1221615616,"data_stored":1220932376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1065993777},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,8,12,11,13,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35520512,"num_objects":4336,"num_object_clones":0,"num_object_copies":17344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4336,"num_write_kb":34688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":1785856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003106"},"pg_stats":[{"pgid":"1.f","version":"31'5222","reported_seq":5262,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.185349+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:49.185349+0000","last_peered":"2024-09-16T04:43:49.185349+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:49.185349+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5222,"log_dups_size":0,"ondisk_log_size":5222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42778624,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":41776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3003}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5268","reported_seq":5330,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.594883+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:49.594883+0000","last_peered":"2024-09-16T04:43:49.594883+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:49.594883+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5268,"log_dups_size":0,"ondisk_log_size":5268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43119884,"num_objects":5264,"num_object_clones":0,"num_object_copies":21056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5266,"num_write_kb":42124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3053}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5245","reported_seq":5314,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.242042+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:49.242042+0000","last_peered":"2024-09-16T04:43:49.242042+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:49.242042+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42941062,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5244,"num_write_kb":41942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3015}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5226","reported_seq":5287,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.674880+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:49.674880+0000","last_peered":"2024-09-16T04:43:49.674880+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:49.674880+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5226,"log_dups_size":0,"ondisk_log_size":5226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42775820,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5224,"num_write_kb":41781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5230","reported_seq":5282,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.187009+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:49.187009+0000","last_peered":"2024-09-16T04:43:49.187009+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:49.187009+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42816780,"num_objects":5227,"num_object_clones":0,"num_object_copies":20908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5228,"num_write_kb":41821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5105","reported_seq":5145,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.325648+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:49.325648+0000","last_peered":"2024-09-16T04:43:49.325648+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:49.325648+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5105,"log_dups_size":0,"ondisk_log_size":5105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41820160,"num_objects":5105,"num_object_clones":0,"num_object_copies":20420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5105,"num_write_kb":40840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2897}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5261","reported_seq":5321,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.677070+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:49.677070+0000","last_peered":"2024-09-16T04:43:49.677070+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:49.677070+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5261,"log_dups_size":0,"ondisk_log_size":5261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43046156,"num_objects":5255,"num_object_clones":0,"num_object_copies":21020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5259,"num_write_kb":42052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3029}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5249","reported_seq":9200,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.687269+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:49.687269+0000","last_peered":"2024-09-16T04:43:49.687269+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:49.687269+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42956044,"num_objects":5244,"num_object_clones":0,"num_object_copies":20976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5247,"num_write_kb":41964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":15917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4949}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5266","reported_seq":5318,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.259374+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:49.259374+0000","last_peered":"2024-09-16T04:43:49.259374+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:49.259374+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5266,"log_dups_size":0,"ondisk_log_size":5266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43067928,"num_objects":5258,"num_object_clones":0,"num_object_copies":21032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5263,"num_write_kb":42081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5275","reported_seq":5347,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.292960+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:49.292960+0000","last_peered":"2024-09-16T04:43:49.292960+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:49.292960+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5275,"log_dups_size":0,"ondisk_log_size":5275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43193612,"num_objects":5273,"num_object_clones":0,"num_object_copies":21092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5273,"num_write_kb":42182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3073}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5178","reported_seq":5232,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.338979+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:49.338979+0000","last_peered":"2024-09-16T04:43:49.338979+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:49.338979+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5178,"log_dups_size":0,"ondisk_log_size":5178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42373010,"num_objects":5173,"num_object_clones":0,"num_object_copies":20692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":41388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5137","reported_seq":5172,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.327353+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:49.327353+0000","last_peered":"2024-09-16T04:43:49.327353+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:49.327353+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5137,"log_dups_size":0,"ondisk_log_size":5137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42072710,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5136,"num_write_kb":41087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3008}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5174","reported_seq":5244,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.259624+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:49.259624+0000","last_peered":"2024-09-16T04:43:49.259624+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:49.259624+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5174,"log_dups_size":0,"ondisk_log_size":5174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42375814,"num_objects":5173,"num_object_clones":0,"num_object_copies":20692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5173,"num_write_kb":41383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2948}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5271","reported_seq":5321,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.300853+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:49.300853+0000","last_peered":"2024-09-16T04:43:49.300853+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:49.300853+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43144460,"num_objects":5267,"num_object_clones":0,"num_object_copies":21068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5269,"num_write_kb":42148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3033}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5124","reported_seq":5199,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.185076+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:49.185076+0000","last_peered":"2024-09-16T04:43:49.185076+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:49.185076+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41948428,"num_objects":5121,"num_object_clones":0,"num_object_copies":20484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":40973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2989}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5113","reported_seq":5172,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.625404+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:49.625404+0000","last_peered":"2024-09-16T04:43:49.625404+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:49.625404+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41841932,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5111,"num_write_kb":40876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2869}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":682272424,"num_objects":83290,"num_object_clones":0,"num_object_copies":333160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83317,"num_write_kb":666418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":15917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1223159808,"data_stored":1223159808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83344,"ondisk_log_size":83344,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598880,"kb_used_data":332560,"kb_used_omap":4,"kb_used_meta":266299,"kb_avail":93772960,"statfs":{"total":96636764160,"available":96023511040,"internally_reserved":0,"allocated":340541440,"data_stored":340370630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272690895},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,6,4,1,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634104,"kb_used_data":332200,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93737736,"statfs":{"total":96636764160,"available":95987441664,"internally_reserved":0,"allocated":340172800,"data_stored":340001990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393296,"kb_used_data":195008,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93978544,"statfs":{"total":96636764160,"available":96234029056,"internally_reserved":0,"allocated":199688192,"data_stored":199517382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":203026127},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607792,"kb_used_data":333216,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93764048,"statfs":{"total":96636764160,"available":96014385152,"internally_reserved":0,"allocated":341213184,"data_stored":341042374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":281145038},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":341340160,"data_stored":341340160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":199970816,"data_stored":199970816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":340942848,"data_stored":340942848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":340905984,"data_stored":340905984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:52.025 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:52.293 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 178 20 84094 84074 3.68899 2.16406 0.0800291 0.0423427 2024-09-16T04:43:52.706 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:52.707 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:52.725 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-09-16T04:43:52.612342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":682272424,"num_objects":83290,"num_object_clones":0,"num_object_copies":333160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83317,"num_write_kb":666418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":15917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83344,"ondisk_log_size":83344,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2243960,"kb_used_data":1198712,"kb_used_omap":14,"kb_used_meta":1045169,"kb_avail":375243400,"statfs":{"total":386547056640,"available":384249241600,"internally_reserved":0,"allocated":1227481088,"data_stored":1226797848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1070253617},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,16,11,8,11,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23527424,"num_objects":2872,"num_object_clones":0,"num_object_copies":11488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2872,"num_write_kb":22976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003174"},"pg_stats":[{"pgid":"1.f","version":"31'5222","reported_seq":5262,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.185349+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:49.185349+0000","last_peered":"2024-09-16T04:43:49.185349+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:49.185349+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5222,"log_dups_size":0,"ondisk_log_size":5222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42778624,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":41776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3003}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5268","reported_seq":5330,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.594883+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:49.594883+0000","last_peered":"2024-09-16T04:43:49.594883+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:49.594883+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5268,"log_dups_size":0,"ondisk_log_size":5268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43119884,"num_objects":5264,"num_object_clones":0,"num_object_copies":21056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5266,"num_write_kb":42124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3053}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5245","reported_seq":5314,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.242042+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:49.242042+0000","last_peered":"2024-09-16T04:43:49.242042+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:49.242042+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42941062,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5244,"num_write_kb":41942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3015}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5226","reported_seq":5287,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.674880+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:49.674880+0000","last_peered":"2024-09-16T04:43:49.674880+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:49.674880+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5226,"log_dups_size":0,"ondisk_log_size":5226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42775820,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5224,"num_write_kb":41781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5230","reported_seq":5282,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.187009+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:49.187009+0000","last_peered":"2024-09-16T04:43:49.187009+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:49.187009+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42816780,"num_objects":5227,"num_object_clones":0,"num_object_copies":20908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5228,"num_write_kb":41821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5105","reported_seq":5145,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.325648+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:49.325648+0000","last_peered":"2024-09-16T04:43:49.325648+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:49.325648+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5105,"log_dups_size":0,"ondisk_log_size":5105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41820160,"num_objects":5105,"num_object_clones":0,"num_object_copies":20420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5105,"num_write_kb":40840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2897}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5261","reported_seq":5321,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.677070+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:49.677070+0000","last_peered":"2024-09-16T04:43:49.677070+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:49.677070+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5261,"log_dups_size":0,"ondisk_log_size":5261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43046156,"num_objects":5255,"num_object_clones":0,"num_object_copies":21020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5259,"num_write_kb":42052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3029}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5249","reported_seq":9200,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.687269+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:49.687269+0000","last_peered":"2024-09-16T04:43:49.687269+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:49.687269+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42956044,"num_objects":5244,"num_object_clones":0,"num_object_copies":20976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5247,"num_write_kb":41964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":15917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4949}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5266","reported_seq":5318,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.259374+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:49.259374+0000","last_peered":"2024-09-16T04:43:49.259374+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:49.259374+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5266,"log_dups_size":0,"ondisk_log_size":5266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43067928,"num_objects":5258,"num_object_clones":0,"num_object_copies":21032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5263,"num_write_kb":42081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5275","reported_seq":5347,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.292960+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:49.292960+0000","last_peered":"2024-09-16T04:43:49.292960+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:49.292960+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5275,"log_dups_size":0,"ondisk_log_size":5275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43193612,"num_objects":5273,"num_object_clones":0,"num_object_copies":21092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5273,"num_write_kb":42182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3073}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5178","reported_seq":5232,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.338979+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:49.338979+0000","last_peered":"2024-09-16T04:43:49.338979+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:49.338979+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5178,"log_dups_size":0,"ondisk_log_size":5178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42373010,"num_objects":5173,"num_object_clones":0,"num_object_copies":20692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":41388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5137","reported_seq":5172,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.327353+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:49.327353+0000","last_peered":"2024-09-16T04:43:49.327353+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:49.327353+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5137,"log_dups_size":0,"ondisk_log_size":5137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42072710,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5136,"num_write_kb":41087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3008}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5174","reported_seq":5244,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.259624+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:49.259624+0000","last_peered":"2024-09-16T04:43:49.259624+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:49.259624+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5174,"log_dups_size":0,"ondisk_log_size":5174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42375814,"num_objects":5173,"num_object_clones":0,"num_object_copies":20692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5173,"num_write_kb":41383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2948}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5271","reported_seq":5321,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.300853+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:49.300853+0000","last_peered":"2024-09-16T04:43:49.300853+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:49.300853+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43144460,"num_objects":5267,"num_object_clones":0,"num_object_copies":21068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5269,"num_write_kb":42148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3033}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5124","reported_seq":5199,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.185076+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:49.185076+0000","last_peered":"2024-09-16T04:43:49.185076+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:49.185076+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41948428,"num_objects":5121,"num_object_clones":0,"num_object_copies":20484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":40973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2989}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5113","reported_seq":5172,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.625404+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:49.625404+0000","last_peered":"2024-09-16T04:43:49.625404+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:49.625404+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41841932,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5111,"num_write_kb":40876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2869}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":682272424,"num_objects":83290,"num_object_clones":0,"num_object_copies":333160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83317,"num_write_kb":666418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":15917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1229455360,"data_stored":1229455360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83344,"ondisk_log_size":83344,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598880,"kb_used_data":332560,"kb_used_omap":4,"kb_used_meta":266299,"kb_avail":93772960,"statfs":{"total":96636764160,"available":96023511040,"internally_reserved":0,"allocated":340541440,"data_stored":340370630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272690895},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,6,4,1,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634104,"kb_used_data":332200,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93737736,"statfs":{"total":96636764160,"available":95987441664,"internally_reserved":0,"allocated":340172800,"data_stored":340001990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403184,"kb_used_data":200736,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93968656,"statfs":{"total":96636764160,"available":96223903744,"internally_reserved":0,"allocated":205553664,"data_stored":205382854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207285967},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,9,1,1,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607792,"kb_used_data":333216,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93764048,"statfs":{"total":96636764160,"available":96014385152,"internally_reserved":0,"allocated":341213184,"data_stored":341042374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":281145038},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":341340160,"data_stored":341340160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":206266368,"data_stored":206266368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":340942848,"data_stored":340942848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":340905984,"data_stored":340905984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:52.726 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:43:52.726 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5222", 'reported_seq': 5262, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.185349+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:43:49.185349+0000', 'last_peered': '2024-09-16T04:43:49.185349+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:43:49.185349+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5222, 'log_dups_size': 0, 'ondisk_log_size': 5222, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42778624, 'num_objects': 5222, 'num_object_clones': 0, 'num_object_copies': 20888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5222, 'num_write_kb': 41776, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3003}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:52.726 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:43:52.726 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5268", 'reported_seq': 5330, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.594883+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:43:49.594883+0000', 'last_peered': '2024-09-16T04:43:49.594883+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:43:49.594883+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5268, 'log_dups_size': 0, 'ondisk_log_size': 5268, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43119884, 'num_objects': 5264, 'num_object_clones': 0, 'num_object_copies': 21056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5266, 'num_write_kb': 42124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3053}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:52.726 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:43:52.726 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5245", 'reported_seq': 5314, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.242042+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:43:49.242042+0000', 'last_peered': '2024-09-16T04:43:49.242042+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:43:49.242042+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5245, 'log_dups_size': 0, 'ondisk_log_size': 5245, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42941062, 'num_objects': 5242, 'num_object_clones': 0, 'num_object_copies': 20968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5242, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5244, 'num_write_kb': 41942, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3015}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:52.726 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:43:52.726 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5226", 'reported_seq': 5287, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.674880+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:43:49.674880+0000', 'last_peered': '2024-09-16T04:43:49.674880+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:43:49.674880+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5226, 'log_dups_size': 0, 'ondisk_log_size': 5226, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42775820, 'num_objects': 5222, 'num_object_clones': 0, 'num_object_copies': 20888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5224, 'num_write_kb': 41781, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2993}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:52.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:43:52.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5230", 'reported_seq': 5282, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.187009+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:43:49.187009+0000', 'last_peered': '2024-09-16T04:43:49.187009+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:43:49.187009+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5230, 'log_dups_size': 0, 'ondisk_log_size': 5230, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42816780, 'num_objects': 5227, 'num_object_clones': 0, 'num_object_copies': 20908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5228, 'num_write_kb': 41821, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2944}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:52.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:43:52.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5105", 'reported_seq': 5145, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.325648+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:43:49.325648+0000', 'last_peered': '2024-09-16T04:43:49.325648+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:43:49.325648+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5105, 'log_dups_size': 0, 'ondisk_log_size': 5105, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41820160, 'num_objects': 5105, 'num_object_clones': 0, 'num_object_copies': 20420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5105, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5105, 'num_write_kb': 40840, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2897}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:52.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:43:52.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5261", 'reported_seq': 5321, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.677070+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:43:49.677070+0000', 'last_peered': '2024-09-16T04:43:49.677070+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:43:49.677070+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5261, 'log_dups_size': 0, 'ondisk_log_size': 5261, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43046156, 'num_objects': 5255, 'num_object_clones': 0, 'num_object_copies': 21020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5259, 'num_write_kb': 42052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3029}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:52.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:43:52.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5249", 'reported_seq': 9200, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.687269+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:43:49.687269+0000', 'last_peered': '2024-09-16T04:43:49.687269+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:43:49.687269+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5249, 'log_dups_size': 0, 'ondisk_log_size': 5249, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42956044, 'num_objects': 5244, 'num_object_clones': 0, 'num_object_copies': 20976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 295, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5244, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5247, 'num_write_kb': 41964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1947, 'num_bytes_recovered': 15917056, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4949}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:52.727 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:43:52.727 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5266", 'reported_seq': 5318, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.259374+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:43:49.259374+0000', 'last_peered': '2024-09-16T04:43:49.259374+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:43:49.259374+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5266, 'log_dups_size': 0, 'ondisk_log_size': 5266, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43067928, 'num_objects': 5258, 'num_object_clones': 0, 'num_object_copies': 21032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5263, 'num_write_kb': 42081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2918}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:52.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:43:52.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5275", 'reported_seq': 5347, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.292960+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:43:49.292960+0000', 'last_peered': '2024-09-16T04:43:49.292960+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:43:49.292960+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5275, 'log_dups_size': 0, 'ondisk_log_size': 5275, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43193612, 'num_objects': 5273, 'num_object_clones': 0, 'num_object_copies': 21092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5273, 'num_write_kb': 42182, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3073}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:52.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:43:52.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5178", 'reported_seq': 5232, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.338979+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:43:49.338979+0000', 'last_peered': '2024-09-16T04:43:49.338979+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:43:49.338979+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5178, 'log_dups_size': 0, 'ondisk_log_size': 5178, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42373010, 'num_objects': 5173, 'num_object_clones': 0, 'num_object_copies': 20692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5173, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5175, 'num_write_kb': 41388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2984}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:52.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:43:52.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5137", 'reported_seq': 5172, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.327353+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:43:49.327353+0000', 'last_peered': '2024-09-16T04:43:49.327353+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:43:49.327353+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5137, 'log_dups_size': 0, 'ondisk_log_size': 5137, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42072710, 'num_objects': 5136, 'num_object_clones': 0, 'num_object_copies': 20544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5136, 'num_write_kb': 41087, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3008}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:52.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:43:52.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5174", 'reported_seq': 5244, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.259624+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:43:49.259624+0000', 'last_peered': '2024-09-16T04:43:49.259624+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:43:49.259624+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5174, 'log_dups_size': 0, 'ondisk_log_size': 5174, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42375814, 'num_objects': 5173, 'num_object_clones': 0, 'num_object_copies': 20692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5173, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5173, 'num_write_kb': 41383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2948}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:52.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:43:52.728 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5271", 'reported_seq': 5321, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.300853+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:43:49.300853+0000', 'last_peered': '2024-09-16T04:43:49.300853+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:43:49.300853+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5271, 'log_dups_size': 0, 'ondisk_log_size': 5271, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43144460, 'num_objects': 5267, 'num_object_clones': 0, 'num_object_copies': 21068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5267, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5269, 'num_write_kb': 42148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3033}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:52.728 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:43:52.729 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5124", 'reported_seq': 5199, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.185076+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:43:49.185076+0000', 'last_peered': '2024-09-16T04:43:49.185076+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:43:49.185076+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5124, 'log_dups_size': 0, 'ondisk_log_size': 5124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41948428, 'num_objects': 5121, 'num_object_clones': 0, 'num_object_copies': 20484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5121, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5122, 'num_write_kb': 40973, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2989}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:52.729 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:43:52.729 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5113", 'reported_seq': 5172, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:49.625404+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:43:49.625404+0000', 'last_peered': '2024-09-16T04:43:49.625404+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:43:49.625404+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5113, 'log_dups_size': 0, 'ondisk_log_size': 5113, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41841932, 'num_objects': 5108, 'num_object_clones': 0, 'num_object_copies': 20432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5111, 'num_write_kb': 40876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2869}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:52.729 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:43:53.293 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 179 20 84357 84337 3.67986 2.05469 0.0789089 0.0424437 2024-09-16T04:43:53.435 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:53.435 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33610/333160 objects degraded (10.088%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":245,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":83290,"data_bytes":682272424,"bytes_used":2297815040,"bytes_avail":384249241600,"bytes_total":386547056640,"degraded_objects":33610,"degraded_total":333160,"degraded_ratio":0.10088245887861688,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":103055,"recovering_keys_per_sec":0,"num_objects_recovered":151,"num_bytes_recovered":1236992,"num_keys_recovered":0,"write_bytes_sec":1960100,"read_op_per_sec":0,"write_op_per_sec":239},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:43:53.459 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:43:53.460 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:54.023 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:54.023 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:54.046 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-09-16T04:43:52.612342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":682272424,"num_objects":83290,"num_object_clones":0,"num_object_copies":333160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83317,"num_write_kb":666418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":15917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83344,"ondisk_log_size":83344,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2243960,"kb_used_data":1198712,"kb_used_omap":14,"kb_used_meta":1045169,"kb_avail":375243400,"statfs":{"total":386547056640,"available":384249241600,"internally_reserved":0,"allocated":1227481088,"data_stored":1226797848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1070253617},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,16,11,8,11,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23527424,"num_objects":2872,"num_object_clones":0,"num_object_copies":11488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2872,"num_write_kb":22976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003174"},"pg_stats":[{"pgid":"1.f","version":"31'5222","reported_seq":5262,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.185349+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:49.185349+0000","last_peered":"2024-09-16T04:43:49.185349+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:49.185349+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5222,"log_dups_size":0,"ondisk_log_size":5222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42778624,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":41776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3003}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5268","reported_seq":5330,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.594883+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:49.594883+0000","last_peered":"2024-09-16T04:43:49.594883+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:49.594883+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5268,"log_dups_size":0,"ondisk_log_size":5268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43119884,"num_objects":5264,"num_object_clones":0,"num_object_copies":21056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5266,"num_write_kb":42124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3053}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5245","reported_seq":5314,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.242042+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:49.242042+0000","last_peered":"2024-09-16T04:43:49.242042+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:49.242042+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42941062,"num_objects":5242,"num_object_clones":0,"num_object_copies":20968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5244,"num_write_kb":41942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3015}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5226","reported_seq":5287,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.674880+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:49.674880+0000","last_peered":"2024-09-16T04:43:49.674880+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:49.674880+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5226,"log_dups_size":0,"ondisk_log_size":5226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42775820,"num_objects":5222,"num_object_clones":0,"num_object_copies":20888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5224,"num_write_kb":41781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5230","reported_seq":5282,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.187009+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:49.187009+0000","last_peered":"2024-09-16T04:43:49.187009+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:49.187009+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42816780,"num_objects":5227,"num_object_clones":0,"num_object_copies":20908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5228,"num_write_kb":41821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5105","reported_seq":5145,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.325648+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:49.325648+0000","last_peered":"2024-09-16T04:43:49.325648+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:49.325648+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5105,"log_dups_size":0,"ondisk_log_size":5105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41820160,"num_objects":5105,"num_object_clones":0,"num_object_copies":20420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5105,"num_write_kb":40840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2897}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5261","reported_seq":5321,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.677070+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:49.677070+0000","last_peered":"2024-09-16T04:43:49.677070+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:49.677070+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5261,"log_dups_size":0,"ondisk_log_size":5261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43046156,"num_objects":5255,"num_object_clones":0,"num_object_copies":21020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5259,"num_write_kb":42052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3029}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5249","reported_seq":9200,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.687269+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:49.687269+0000","last_peered":"2024-09-16T04:43:49.687269+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:49.687269+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42956044,"num_objects":5244,"num_object_clones":0,"num_object_copies":20976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5247,"num_write_kb":41964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":15917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4949}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5266","reported_seq":5318,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.259374+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:49.259374+0000","last_peered":"2024-09-16T04:43:49.259374+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:49.259374+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5266,"log_dups_size":0,"ondisk_log_size":5266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43067928,"num_objects":5258,"num_object_clones":0,"num_object_copies":21032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5263,"num_write_kb":42081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5275","reported_seq":5347,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.292960+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:49.292960+0000","last_peered":"2024-09-16T04:43:49.292960+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:49.292960+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5275,"log_dups_size":0,"ondisk_log_size":5275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43193612,"num_objects":5273,"num_object_clones":0,"num_object_copies":21092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5273,"num_write_kb":42182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3073}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5178","reported_seq":5232,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.338979+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:49.338979+0000","last_peered":"2024-09-16T04:43:49.338979+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:49.338979+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5178,"log_dups_size":0,"ondisk_log_size":5178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42373010,"num_objects":5173,"num_object_clones":0,"num_object_copies":20692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":41388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5137","reported_seq":5172,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.327353+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:49.327353+0000","last_peered":"2024-09-16T04:43:49.327353+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:49.327353+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5137,"log_dups_size":0,"ondisk_log_size":5137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42072710,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5136,"num_write_kb":41087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3008}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5174","reported_seq":5244,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.259624+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:49.259624+0000","last_peered":"2024-09-16T04:43:49.259624+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:49.259624+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5174,"log_dups_size":0,"ondisk_log_size":5174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42375814,"num_objects":5173,"num_object_clones":0,"num_object_copies":20692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5173,"num_write_kb":41383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2948}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5271","reported_seq":5321,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.300853+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:49.300853+0000","last_peered":"2024-09-16T04:43:49.300853+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:49.300853+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43144460,"num_objects":5267,"num_object_clones":0,"num_object_copies":21068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5269,"num_write_kb":42148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3033}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5124","reported_seq":5199,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.185076+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:49.185076+0000","last_peered":"2024-09-16T04:43:49.185076+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:49.185076+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41948428,"num_objects":5121,"num_object_clones":0,"num_object_copies":20484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":40973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2989}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5113","reported_seq":5172,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:49.625404+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:49.625404+0000","last_peered":"2024-09-16T04:43:49.625404+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:49.625404+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41841932,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5111,"num_write_kb":40876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2869}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":682272424,"num_objects":83290,"num_object_clones":0,"num_object_copies":333160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83317,"num_write_kb":666418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":15917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1229455360,"data_stored":1229455360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83344,"ondisk_log_size":83344,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598880,"kb_used_data":332560,"kb_used_omap":4,"kb_used_meta":266299,"kb_avail":93772960,"statfs":{"total":96636764160,"available":96023511040,"internally_reserved":0,"allocated":340541440,"data_stored":340370630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272690895},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,6,4,1,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634104,"kb_used_data":332200,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93737736,"statfs":{"total":96636764160,"available":95987441664,"internally_reserved":0,"allocated":340172800,"data_stored":340001990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403184,"kb_used_data":200736,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93968656,"statfs":{"total":96636764160,"available":96223903744,"internally_reserved":0,"allocated":205553664,"data_stored":205382854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207285967},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,9,1,1,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607792,"kb_used_data":333216,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93764048,"statfs":{"total":96636764160,"available":96014385152,"internally_reserved":0,"allocated":341213184,"data_stored":341042374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":281145038},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":341340160,"data_stored":341340160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":206266368,"data_stored":206266368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":340942848,"data_stored":340942848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":340905984,"data_stored":340905984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:54.293 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:2024-09-16T04:43:54.292904+0000 min lat: 0.00269411 max lat: 1.73107 avg lat: 0.0425466 2024-09-16T04:43:54.293 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T04:43:54.294 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 180 20 84629 84609 3.67123 2.125 0.0795845 0.0425466 2024-09-16T04:43:55.293 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 181 20 84924 84904 3.66368 2.30469 0.042876 0.0426243 2024-09-16T04:43:56.295 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 182 20 85228 85208 3.65657 2.375 0.0470165 0.0427172 2024-09-16T04:43:57.048 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:57.298 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 183 20 85520 85500 3.64899 2.28125 0.0503371 0.0428091 2024-09-16T04:43:57.651 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:57.652 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:57.673 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-16T04:43:56.613388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":693864104,"num_objects":84705,"num_object_clones":0,"num_object_copies":338820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84732,"num_write_kb":677738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2034,"num_bytes_recovered":16424960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84759,"ondisk_log_size":84759,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2250468,"kb_used_data":1221860,"kb_used_omap":14,"kb_used_meta":1028529,"kb_avail":375236892,"statfs":{"total":386547056640,"available":384242577408,"internally_reserved":0,"allocated":1251184640,"data_stored":1250501400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1053214257},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,4,6,5,7,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26714112,"num_objects":3261,"num_object_clones":0,"num_object_copies":13044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3261,"num_write_kb":26088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":1744896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003165"},"pg_stats":[{"pgid":"1.f","version":"31'5309","reported_seq":5349,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.293057+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:54.293057+0000","last_peered":"2024-09-16T04:43:54.293057+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:54.293057+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5309,"log_dups_size":0,"ondisk_log_size":5309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43491328,"num_objects":5309,"num_object_clones":0,"num_object_copies":21236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5309,"num_write_kb":42472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5368","reported_seq":5430,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.692932+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:54.692932+0000","last_peered":"2024-09-16T04:43:54.692932+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:54.692932+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5368,"log_dups_size":0,"ondisk_log_size":5368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43939084,"num_objects":5364,"num_object_clones":0,"num_object_copies":21456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":42924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3153}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5325","reported_seq":5396,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.239101+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:54.239101+0000","last_peered":"2024-09-16T04:43:54.239101+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:54.239101+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5325,"log_dups_size":0,"ondisk_log_size":5325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43596422,"num_objects":5322,"num_object_clones":0,"num_object_copies":21288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5324,"num_write_kb":42582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3095}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5321","reported_seq":5383,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.527881+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:54.527881+0000","last_peered":"2024-09-16T04:43:54.527881+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:54.527881+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5321,"log_dups_size":0,"ondisk_log_size":5321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43554060,"num_objects":5317,"num_object_clones":0,"num_object_copies":21268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":42541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5317","reported_seq":5369,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.248988+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:54.248988+0000","last_peered":"2024-09-16T04:43:54.248988+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:54.248988+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5317,"log_dups_size":0,"ondisk_log_size":5317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43529484,"num_objects":5314,"num_object_clones":0,"num_object_copies":21256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5315,"num_write_kb":42517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5183","reported_seq":5223,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.212367+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:54.212367+0000","last_peered":"2024-09-16T04:43:54.212367+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:54.212367+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5183,"log_dups_size":0,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42459136,"num_objects":5183,"num_object_clones":0,"num_object_copies":20732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5183,"num_write_kb":41464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2975}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5350","reported_seq":5411,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.425099+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:54.425099+0000","last_peered":"2024-09-16T04:43:54.425099+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:54.425099+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5350,"log_dups_size":0,"ondisk_log_size":5350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43775244,"num_objects":5344,"num_object_clones":0,"num_object_copies":21376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5348,"num_write_kb":42764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3118}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5345","reported_seq":9420,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.768633+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:54.768633+0000","last_peered":"2024-09-16T04:43:54.768633+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:54.768633+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43742476,"num_objects":5340,"num_object_clones":0,"num_object_copies":21360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5343,"num_write_kb":42732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2009,"num_bytes_recovered":16424960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5352","reported_seq":5405,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.342896+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:54.342896+0000","last_peered":"2024-09-16T04:43:54.342896+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:54.342896+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5352,"log_dups_size":0,"ondisk_log_size":5352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43772440,"num_objects":5344,"num_object_clones":0,"num_object_copies":21376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5349,"num_write_kb":42769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5356","reported_seq":5429,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.238288+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:54.238288+0000","last_peered":"2024-09-16T04:43:54.238288+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:54.238288+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5356,"log_dups_size":0,"ondisk_log_size":5356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43857164,"num_objects":5354,"num_object_clones":0,"num_object_copies":21416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5354,"num_write_kb":42830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3154}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5257","reported_seq":5311,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.351175+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:54.351175+0000","last_peered":"2024-09-16T04:43:54.351175+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:54.351175+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43020178,"num_objects":5252,"num_object_clones":0,"num_object_copies":21008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5254,"num_write_kb":42020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3063}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5217","reported_seq":5253,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.336742+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:54.336742+0000","last_peered":"2024-09-16T04:43:54.336742+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:54.336742+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5217,"log_dups_size":0,"ondisk_log_size":5217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42728070,"num_objects":5216,"num_object_clones":0,"num_object_copies":20864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5216,"num_write_kb":41727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3088}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5271","reported_seq":5340,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.388608+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:54.388608+0000","last_peered":"2024-09-16T04:43:54.388608+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:54.388608+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43170438,"num_objects":5270,"num_object_clones":0,"num_object_copies":21080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5270,"num_write_kb":42159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5366","reported_seq":5416,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.295195+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:54.295195+0000","last_peered":"2024-09-16T04:43:54.295195+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:54.295195+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5366,"log_dups_size":0,"ondisk_log_size":5366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43922700,"num_objects":5362,"num_object_clones":0,"num_object_copies":21448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5364,"num_write_kb":42908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3128}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5223","reported_seq":5298,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.385528+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:54.385528+0000","last_peered":"2024-09-16T04:43:54.385528+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:54.385528+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5223,"log_dups_size":0,"ondisk_log_size":5223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42759436,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5221,"num_write_kb":41765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3088}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5199","reported_seq":5260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.698346+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:54.698346+0000","last_peered":"2024-09-16T04:43:54.698346+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:54.698346+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5199,"log_dups_size":0,"ondisk_log_size":5199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42546444,"num_objects":5194,"num_object_clones":0,"num_object_copies":20776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5197,"num_write_kb":41564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2955}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":693864104,"num_objects":84705,"num_object_clones":0,"num_object_copies":338820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84732,"num_write_kb":677738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2034,"num_bytes_recovered":16424960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1253154816,"data_stored":1253154816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84759,"ondisk_log_size":84759,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608812,"kb_used_data":338332,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93763028,"statfs":{"total":96636764160,"available":96013340672,"internally_reserved":0,"allocated":346451968,"data_stored":346281158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276950735},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644036,"kb_used_data":337972,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93727804,"statfs":{"total":96636764160,"available":95977271296,"internally_reserved":0,"allocated":346083328,"data_stored":345912518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396632,"kb_used_data":206664,"kb_used_omap":4,"kb_used_meta":189947,"kb_avail":93975208,"statfs":{"total":96636764160,"available":96230612992,"internally_reserved":0,"allocated":211623936,"data_stored":211453126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":194506447},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4],"upper_bound":2},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600988,"kb_used_data":338892,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93770852,"statfs":{"total":96636764160,"available":96021352448,"internally_reserved":0,"allocated":347025408,"data_stored":346854598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":268365518},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":347238400,"data_stored":347238400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212381696,"data_stored":212381696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346787840,"data_stored":346787840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":346746880,"data_stored":346746880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:57.674 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:58.227 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:58.227 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:58.256 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-16T04:43:56.613388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":693864104,"num_objects":84705,"num_object_clones":0,"num_object_copies":338820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84732,"num_write_kb":677738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2034,"num_bytes_recovered":16424960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84759,"ondisk_log_size":84759,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2250468,"kb_used_data":1221860,"kb_used_omap":14,"kb_used_meta":1028529,"kb_avail":375236892,"statfs":{"total":386547056640,"available":384242577408,"internally_reserved":0,"allocated":1251184640,"data_stored":1250501400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1053214257},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,4,6,5,7,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26714112,"num_objects":3261,"num_object_clones":0,"num_object_copies":13044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3261,"num_write_kb":26088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":1744896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003165"},"pg_stats":[{"pgid":"1.f","version":"31'5309","reported_seq":5349,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.293057+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:54.293057+0000","last_peered":"2024-09-16T04:43:54.293057+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:54.293057+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5309,"log_dups_size":0,"ondisk_log_size":5309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43491328,"num_objects":5309,"num_object_clones":0,"num_object_copies":21236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5309,"num_write_kb":42472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5368","reported_seq":5430,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.692932+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:54.692932+0000","last_peered":"2024-09-16T04:43:54.692932+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:54.692932+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5368,"log_dups_size":0,"ondisk_log_size":5368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43939084,"num_objects":5364,"num_object_clones":0,"num_object_copies":21456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":42924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3153}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5325","reported_seq":5396,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.239101+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:54.239101+0000","last_peered":"2024-09-16T04:43:54.239101+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:54.239101+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5325,"log_dups_size":0,"ondisk_log_size":5325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43596422,"num_objects":5322,"num_object_clones":0,"num_object_copies":21288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5324,"num_write_kb":42582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3095}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5321","reported_seq":5383,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.527881+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:54.527881+0000","last_peered":"2024-09-16T04:43:54.527881+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:54.527881+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5321,"log_dups_size":0,"ondisk_log_size":5321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43554060,"num_objects":5317,"num_object_clones":0,"num_object_copies":21268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":42541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5317","reported_seq":5369,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.248988+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:54.248988+0000","last_peered":"2024-09-16T04:43:54.248988+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:54.248988+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5317,"log_dups_size":0,"ondisk_log_size":5317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43529484,"num_objects":5314,"num_object_clones":0,"num_object_copies":21256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5315,"num_write_kb":42517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5183","reported_seq":5223,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.212367+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:54.212367+0000","last_peered":"2024-09-16T04:43:54.212367+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:54.212367+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5183,"log_dups_size":0,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42459136,"num_objects":5183,"num_object_clones":0,"num_object_copies":20732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5183,"num_write_kb":41464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2975}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5350","reported_seq":5411,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.425099+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:54.425099+0000","last_peered":"2024-09-16T04:43:54.425099+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:54.425099+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5350,"log_dups_size":0,"ondisk_log_size":5350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43775244,"num_objects":5344,"num_object_clones":0,"num_object_copies":21376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5348,"num_write_kb":42764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3118}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5345","reported_seq":9420,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.768633+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:54.768633+0000","last_peered":"2024-09-16T04:43:54.768633+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:54.768633+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43742476,"num_objects":5340,"num_object_clones":0,"num_object_copies":21360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5343,"num_write_kb":42732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2009,"num_bytes_recovered":16424960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5352","reported_seq":5405,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.342896+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:54.342896+0000","last_peered":"2024-09-16T04:43:54.342896+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:54.342896+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5352,"log_dups_size":0,"ondisk_log_size":5352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43772440,"num_objects":5344,"num_object_clones":0,"num_object_copies":21376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5349,"num_write_kb":42769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5356","reported_seq":5429,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.238288+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:54.238288+0000","last_peered":"2024-09-16T04:43:54.238288+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:54.238288+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5356,"log_dups_size":0,"ondisk_log_size":5356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43857164,"num_objects":5354,"num_object_clones":0,"num_object_copies":21416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5354,"num_write_kb":42830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3154}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5257","reported_seq":5311,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.351175+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:54.351175+0000","last_peered":"2024-09-16T04:43:54.351175+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:54.351175+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43020178,"num_objects":5252,"num_object_clones":0,"num_object_copies":21008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5254,"num_write_kb":42020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3063}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5217","reported_seq":5253,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.336742+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:54.336742+0000","last_peered":"2024-09-16T04:43:54.336742+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:54.336742+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5217,"log_dups_size":0,"ondisk_log_size":5217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42728070,"num_objects":5216,"num_object_clones":0,"num_object_copies":20864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5216,"num_write_kb":41727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3088}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5271","reported_seq":5340,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.388608+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:54.388608+0000","last_peered":"2024-09-16T04:43:54.388608+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:54.388608+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43170438,"num_objects":5270,"num_object_clones":0,"num_object_copies":21080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5270,"num_write_kb":42159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5366","reported_seq":5416,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.295195+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:54.295195+0000","last_peered":"2024-09-16T04:43:54.295195+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:54.295195+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5366,"log_dups_size":0,"ondisk_log_size":5366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43922700,"num_objects":5362,"num_object_clones":0,"num_object_copies":21448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5364,"num_write_kb":42908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3128}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5223","reported_seq":5298,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.385528+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:54.385528+0000","last_peered":"2024-09-16T04:43:54.385528+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:54.385528+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5223,"log_dups_size":0,"ondisk_log_size":5223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42759436,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5221,"num_write_kb":41765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3088}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5199","reported_seq":5260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.698346+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:54.698346+0000","last_peered":"2024-09-16T04:43:54.698346+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:54.698346+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5199,"log_dups_size":0,"ondisk_log_size":5199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42546444,"num_objects":5194,"num_object_clones":0,"num_object_copies":20776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5197,"num_write_kb":41564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2955}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":693864104,"num_objects":84705,"num_object_clones":0,"num_object_copies":338820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84732,"num_write_kb":677738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2034,"num_bytes_recovered":16424960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1253154816,"data_stored":1253154816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84759,"ondisk_log_size":84759,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608812,"kb_used_data":338332,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93763028,"statfs":{"total":96636764160,"available":96013340672,"internally_reserved":0,"allocated":346451968,"data_stored":346281158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276950735},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644036,"kb_used_data":337972,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93727804,"statfs":{"total":96636764160,"available":95977271296,"internally_reserved":0,"allocated":346083328,"data_stored":345912518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396632,"kb_used_data":206664,"kb_used_omap":4,"kb_used_meta":189947,"kb_avail":93975208,"statfs":{"total":96636764160,"available":96230612992,"internally_reserved":0,"allocated":211623936,"data_stored":211453126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":194506447},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4],"upper_bound":2},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600988,"kb_used_data":338892,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93770852,"statfs":{"total":96636764160,"available":96021352448,"internally_reserved":0,"allocated":347025408,"data_stored":346854598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":268365518},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":347238400,"data_stored":347238400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212381696,"data_stored":212381696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346787840,"data_stored":346787840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":346746880,"data_stored":346746880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:58.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:43:58.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5309", 'reported_seq': 5349, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.293057+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:43:54.293057+0000', 'last_peered': '2024-09-16T04:43:54.293057+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:43:54.293057+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5309, 'log_dups_size': 0, 'ondisk_log_size': 5309, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43491328, 'num_objects': 5309, 'num_object_clones': 0, 'num_object_copies': 21236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5309, 'num_write_kb': 42472, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3090}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:58.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:43:58.258 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5368", 'reported_seq': 5430, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.692932+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:43:54.692932+0000', 'last_peered': '2024-09-16T04:43:54.692932+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:43:54.692932+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5368, 'log_dups_size': 0, 'ondisk_log_size': 5368, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43939084, 'num_objects': 5364, 'num_object_clones': 0, 'num_object_copies': 21456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5364, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5366, 'num_write_kb': 42924, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3153}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:58.258 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:43:58.258 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5325", 'reported_seq': 5396, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.239101+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:43:54.239101+0000', 'last_peered': '2024-09-16T04:43:54.239101+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:43:54.239101+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5325, 'log_dups_size': 0, 'ondisk_log_size': 5325, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43596422, 'num_objects': 5322, 'num_object_clones': 0, 'num_object_copies': 21288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5322, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5324, 'num_write_kb': 42582, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3095}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:58.258 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:43:58.258 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5321", 'reported_seq': 5383, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.527881+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:43:54.527881+0000', 'last_peered': '2024-09-16T04:43:54.527881+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:43:54.527881+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5321, 'log_dups_size': 0, 'ondisk_log_size': 5321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43554060, 'num_objects': 5317, 'num_object_clones': 0, 'num_object_copies': 21268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5317, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5319, 'num_write_kb': 42541, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3088}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:58.258 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:43:58.258 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5317", 'reported_seq': 5369, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.248988+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:43:54.248988+0000', 'last_peered': '2024-09-16T04:43:54.248988+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:43:54.248988+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5317, 'log_dups_size': 0, 'ondisk_log_size': 5317, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43529484, 'num_objects': 5314, 'num_object_clones': 0, 'num_object_copies': 21256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5314, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5315, 'num_write_kb': 42517, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3031}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:58.258 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:43:58.258 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5183", 'reported_seq': 5223, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.212367+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:43:54.212367+0000', 'last_peered': '2024-09-16T04:43:54.212367+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:43:54.212367+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5183, 'log_dups_size': 0, 'ondisk_log_size': 5183, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42459136, 'num_objects': 5183, 'num_object_clones': 0, 'num_object_copies': 20732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5183, 'num_write_kb': 41464, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2975}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:58.258 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:43:58.259 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5350", 'reported_seq': 5411, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.425099+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:43:54.425099+0000', 'last_peered': '2024-09-16T04:43:54.425099+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:43:54.425099+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5350, 'log_dups_size': 0, 'ondisk_log_size': 5350, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43775244, 'num_objects': 5344, 'num_object_clones': 0, 'num_object_copies': 21376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5344, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5348, 'num_write_kb': 42764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3118}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:58.259 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:43:58.259 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5345", 'reported_seq': 9420, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.768633+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:43:54.768633+0000', 'last_peered': '2024-09-16T04:43:54.768633+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:43:54.768633+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5345, 'log_dups_size': 0, 'ondisk_log_size': 5345, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43742476, 'num_objects': 5340, 'num_object_clones': 0, 'num_object_copies': 21360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 233, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5343, 'num_write_kb': 42732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2009, 'num_bytes_recovered': 16424960, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 5107}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:58.259 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:43:58.259 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5352", 'reported_seq': 5405, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.342896+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:43:54.342896+0000', 'last_peered': '2024-09-16T04:43:54.342896+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:43:54.342896+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5352, 'log_dups_size': 0, 'ondisk_log_size': 5352, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43772440, 'num_objects': 5344, 'num_object_clones': 0, 'num_object_copies': 21376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5344, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5349, 'num_write_kb': 42769, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3004}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:58.259 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:43:58.259 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5356", 'reported_seq': 5429, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.238288+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:43:54.238288+0000', 'last_peered': '2024-09-16T04:43:54.238288+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:43:54.238288+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5356, 'log_dups_size': 0, 'ondisk_log_size': 5356, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43857164, 'num_objects': 5354, 'num_object_clones': 0, 'num_object_copies': 21416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5354, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5354, 'num_write_kb': 42830, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3154}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:58.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:43:58.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5257", 'reported_seq': 5311, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.351175+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:43:54.351175+0000', 'last_peered': '2024-09-16T04:43:54.351175+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:43:54.351175+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5257, 'log_dups_size': 0, 'ondisk_log_size': 5257, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43020178, 'num_objects': 5252, 'num_object_clones': 0, 'num_object_copies': 21008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5254, 'num_write_kb': 42020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3063}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:58.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:43:58.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5217", 'reported_seq': 5253, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.336742+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:43:54.336742+0000', 'last_peered': '2024-09-16T04:43:54.336742+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:43:54.336742+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5217, 'log_dups_size': 0, 'ondisk_log_size': 5217, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42728070, 'num_objects': 5216, 'num_object_clones': 0, 'num_object_copies': 20864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5216, 'num_write_kb': 41727, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3088}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:58.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:43:58.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5271", 'reported_seq': 5340, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.388608+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:43:54.388608+0000', 'last_peered': '2024-09-16T04:43:54.388608+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:43:54.388608+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5271, 'log_dups_size': 0, 'ondisk_log_size': 5271, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43170438, 'num_objects': 5270, 'num_object_clones': 0, 'num_object_copies': 21080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5270, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5270, 'num_write_kb': 42159, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3045}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:58.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:43:58.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5366", 'reported_seq': 5416, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.295195+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:43:54.295195+0000', 'last_peered': '2024-09-16T04:43:54.295195+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:43:54.295195+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5366, 'log_dups_size': 0, 'ondisk_log_size': 5366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43922700, 'num_objects': 5362, 'num_object_clones': 0, 'num_object_copies': 21448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5362, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5364, 'num_write_kb': 42908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3128}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:43:58.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:43:58.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5223", 'reported_seq': 5298, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.385528+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:43:54.385528+0000', 'last_peered': '2024-09-16T04:43:54.385528+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:43:54.385528+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5223, 'log_dups_size': 0, 'ondisk_log_size': 5223, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42759436, 'num_objects': 5220, 'num_object_clones': 0, 'num_object_copies': 20880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5221, 'num_write_kb': 41765, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3088}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:43:58.261 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:43:58.261 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5199", 'reported_seq': 5260, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:54.698346+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:43:54.698346+0000', 'last_peered': '2024-09-16T04:43:54.698346+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:43:54.698346+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5199, 'log_dups_size': 0, 'ondisk_log_size': 5199, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42546444, 'num_objects': 5194, 'num_object_clones': 0, 'num_object_copies': 20776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5197, 'num_write_kb': 41564, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2955}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:43:58.261 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:43:58.298 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 184 20 85781 85761 3.64024 2.03906 0.0437769 0.0429077 2024-09-16T04:43:58.969 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:58.969 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33548/338820 objects degraded (9.901%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":251,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":84705,"data_bytes":693864104,"bytes_used":2304479232,"bytes_avail":384242577408,"bytes_total":386547056640,"degraded_objects":33548,"degraded_total":338820,"degraded_ratio":0.09901422584263031,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":94183,"recovering_keys_per_sec":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"write_bytes_sec":1934173,"read_op_per_sec":0,"write_op_per_sec":236},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:43:58.987 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:43:58.988 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:59.298 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 185 20 86048 86028 3.63184 2.08594 0.083153 0.0430069 2024-09-16T04:43:59.675 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:43:59.675 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:43:59.700 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-09-16T04:43:58.613918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":693864104,"num_objects":84705,"num_object_clones":0,"num_object_copies":338820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84732,"num_write_kb":677738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2034,"num_bytes_recovered":16424960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84759,"ondisk_log_size":84759,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2250468,"kb_used_data":1221860,"kb_used_omap":14,"kb_used_meta":1028529,"kb_avail":375236892,"statfs":{"total":386547056640,"available":384242577408,"internally_reserved":0,"allocated":1251184640,"data_stored":1250501400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1053214257},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,4,6,5,7,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23216128,"num_objects":2834,"num_object_clones":0,"num_object_copies":11336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":22672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003125"},"pg_stats":[{"pgid":"1.f","version":"31'5309","reported_seq":5349,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.293057+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:54.293057+0000","last_peered":"2024-09-16T04:43:54.293057+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:54.293057+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5309,"log_dups_size":0,"ondisk_log_size":5309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43491328,"num_objects":5309,"num_object_clones":0,"num_object_copies":21236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5309,"num_write_kb":42472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3090}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5368","reported_seq":5430,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.692932+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:54.692932+0000","last_peered":"2024-09-16T04:43:54.692932+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:54.692932+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5368,"log_dups_size":0,"ondisk_log_size":5368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43939084,"num_objects":5364,"num_object_clones":0,"num_object_copies":21456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":42924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3153}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5325","reported_seq":5396,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.239101+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:54.239101+0000","last_peered":"2024-09-16T04:43:54.239101+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:54.239101+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5325,"log_dups_size":0,"ondisk_log_size":5325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43596422,"num_objects":5322,"num_object_clones":0,"num_object_copies":21288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5324,"num_write_kb":42582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3095}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5321","reported_seq":5383,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.527881+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:54.527881+0000","last_peered":"2024-09-16T04:43:54.527881+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:54.527881+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5321,"log_dups_size":0,"ondisk_log_size":5321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43554060,"num_objects":5317,"num_object_clones":0,"num_object_copies":21268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":42541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5317","reported_seq":5369,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.248988+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:54.248988+0000","last_peered":"2024-09-16T04:43:54.248988+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:54.248988+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5317,"log_dups_size":0,"ondisk_log_size":5317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43529484,"num_objects":5314,"num_object_clones":0,"num_object_copies":21256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5315,"num_write_kb":42517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5183","reported_seq":5223,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.212367+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:54.212367+0000","last_peered":"2024-09-16T04:43:54.212367+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:54.212367+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5183,"log_dups_size":0,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42459136,"num_objects":5183,"num_object_clones":0,"num_object_copies":20732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5183,"num_write_kb":41464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2975}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5350","reported_seq":5411,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.425099+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:54.425099+0000","last_peered":"2024-09-16T04:43:54.425099+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:54.425099+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5350,"log_dups_size":0,"ondisk_log_size":5350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43775244,"num_objects":5344,"num_object_clones":0,"num_object_copies":21376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5348,"num_write_kb":42764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3118}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5345","reported_seq":9420,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.768633+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:54.768633+0000","last_peered":"2024-09-16T04:43:54.768633+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:54.768633+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43742476,"num_objects":5340,"num_object_clones":0,"num_object_copies":21360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5343,"num_write_kb":42732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2009,"num_bytes_recovered":16424960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5352","reported_seq":5405,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.342896+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:54.342896+0000","last_peered":"2024-09-16T04:43:54.342896+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:54.342896+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5352,"log_dups_size":0,"ondisk_log_size":5352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43772440,"num_objects":5344,"num_object_clones":0,"num_object_copies":21376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5349,"num_write_kb":42769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5356","reported_seq":5429,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.238288+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:54.238288+0000","last_peered":"2024-09-16T04:43:54.238288+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:54.238288+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5356,"log_dups_size":0,"ondisk_log_size":5356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43857164,"num_objects":5354,"num_object_clones":0,"num_object_copies":21416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5354,"num_write_kb":42830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3154}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5257","reported_seq":5311,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.351175+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:54.351175+0000","last_peered":"2024-09-16T04:43:54.351175+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:54.351175+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43020178,"num_objects":5252,"num_object_clones":0,"num_object_copies":21008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5254,"num_write_kb":42020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3063}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5217","reported_seq":5253,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.336742+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:54.336742+0000","last_peered":"2024-09-16T04:43:54.336742+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:54.336742+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5217,"log_dups_size":0,"ondisk_log_size":5217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42728070,"num_objects":5216,"num_object_clones":0,"num_object_copies":20864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5216,"num_write_kb":41727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3088}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5271","reported_seq":5340,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.388608+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:54.388608+0000","last_peered":"2024-09-16T04:43:54.388608+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:54.388608+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43170438,"num_objects":5270,"num_object_clones":0,"num_object_copies":21080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5270,"num_write_kb":42159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5366","reported_seq":5416,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.295195+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:54.295195+0000","last_peered":"2024-09-16T04:43:54.295195+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:54.295195+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5366,"log_dups_size":0,"ondisk_log_size":5366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43922700,"num_objects":5362,"num_object_clones":0,"num_object_copies":21448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5364,"num_write_kb":42908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3128}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5223","reported_seq":5298,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.385528+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:54.385528+0000","last_peered":"2024-09-16T04:43:54.385528+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:54.385528+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5223,"log_dups_size":0,"ondisk_log_size":5223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42759436,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5221,"num_write_kb":41765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3088}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5199","reported_seq":5260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:54.698346+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:54.698346+0000","last_peered":"2024-09-16T04:43:54.698346+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:54.698346+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5199,"log_dups_size":0,"ondisk_log_size":5199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42546444,"num_objects":5194,"num_object_clones":0,"num_object_copies":20776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5197,"num_write_kb":41564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2955}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":693864104,"num_objects":84705,"num_object_clones":0,"num_object_copies":338820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84732,"num_write_kb":677738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2034,"num_bytes_recovered":16424960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1253154816,"data_stored":1253154816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84759,"ondisk_log_size":84759,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608812,"kb_used_data":338332,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93763028,"statfs":{"total":96636764160,"available":96013340672,"internally_reserved":0,"allocated":346451968,"data_stored":346281158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276950735},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644036,"kb_used_data":337972,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93727804,"statfs":{"total":96636764160,"available":95977271296,"internally_reserved":0,"allocated":346083328,"data_stored":345912518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396632,"kb_used_data":206664,"kb_used_omap":4,"kb_used_meta":189947,"kb_avail":93975208,"statfs":{"total":96636764160,"available":96230612992,"internally_reserved":0,"allocated":211623936,"data_stored":211453126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":194506447},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4],"upper_bound":2},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600988,"kb_used_data":338892,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93770852,"statfs":{"total":96636764160,"available":96021352448,"internally_reserved":0,"allocated":347025408,"data_stored":346854598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":268365518},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":347238400,"data_stored":347238400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212381696,"data_stored":212381696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346787840,"data_stored":346787840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":346746880,"data_stored":346746880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:00.298 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 186 20 86335 86315 3.62437 2.24219 0.0515998 0.0430978 2024-09-16T04:44:01.298 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 187 20 86633 86613 3.61744 2.32812 0.0681117 0.0431793 2024-09-16T04:44:02.298 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 188 20 86931 86911 3.61058 2.32812 0.0560797 0.0432634 2024-09-16T04:44:02.702 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:03.256 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:03.257 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:03.285 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-09-16T04:44:02.615050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":705545896,"num_objects":86131,"num_object_clones":0,"num_object_copies":344524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86158,"num_write_kb":689146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":17113088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86185,"ondisk_log_size":86185,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2344100,"kb_used_data":1244708,"kb_used_omap":14,"kb_used_meta":1099313,"kb_avail":375143260,"statfs":{"total":386547056640,"available":384146698240,"internally_reserved":0,"allocated":1274580992,"data_stored":1273897752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1125697073},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,10,4,12,15,9,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23273472,"num_objects":2841,"num_object_clones":0,"num_object_copies":11364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2841,"num_write_kb":22728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003272"},"pg_stats":[{"pgid":"1.f","version":"31'5410","reported_seq":5450,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.357809+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:59.357809+0000","last_peered":"2024-09-16T04:43:59.357809+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:59.357809+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5410,"log_dups_size":0,"ondisk_log_size":5410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44318720,"num_objects":5410,"num_object_clones":0,"num_object_copies":21640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":43280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5460","reported_seq":5522,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.735969+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:59.735969+0000","last_peered":"2024-09-16T04:43:59.735969+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:59.735969+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5460,"log_dups_size":0,"ondisk_log_size":5460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44692748,"num_objects":5456,"num_object_clones":0,"num_object_copies":21824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5458,"num_write_kb":43660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3245}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5430","reported_seq":5501,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.401870+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:59.401870+0000","last_peered":"2024-09-16T04:43:59.401870+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:59.401870+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5430,"log_dups_size":0,"ondisk_log_size":5430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44456582,"num_objects":5427,"num_object_clones":0,"num_object_copies":21708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5429,"num_write_kb":43422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5401","reported_seq":5462,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.664916+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:59.664916+0000","last_peered":"2024-09-16T04:43:59.664916+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:59.664916+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44209420,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":43181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5411","reported_seq":5463,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.110930+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:59.110930+0000","last_peered":"2024-09-16T04:43:59.110930+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:59.110930+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5411,"log_dups_size":0,"ondisk_log_size":5411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44299532,"num_objects":5408,"num_object_clones":0,"num_object_copies":21632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5409,"num_write_kb":43269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5264","reported_seq":5304,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.264309+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:59.264309+0000","last_peered":"2024-09-16T04:43:59.264309+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:59.264309+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5264,"log_dups_size":0,"ondisk_log_size":5264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43122688,"num_objects":5264,"num_object_clones":0,"num_object_copies":21056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5264,"num_write_kb":42112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5421","reported_seq":5482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.711151+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:59.711151+0000","last_peered":"2024-09-16T04:43:59.711151+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:59.711151+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5421,"log_dups_size":0,"ondisk_log_size":5421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44356876,"num_objects":5415,"num_object_clones":0,"num_object_copies":21660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5419,"num_write_kb":43332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3189}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5437","reported_seq":9678,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.780668+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:59.780668+0000","last_peered":"2024-09-16T04:43:59.780668+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:59.780668+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5437,"log_dups_size":0,"ondisk_log_size":5437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44496140,"num_objects":5432,"num_object_clones":0,"num_object_copies":21728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5435,"num_write_kb":43468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2093,"num_bytes_recovered":17113088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5444","reported_seq":5497,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.307508+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:59.307508+0000","last_peered":"2024-09-16T04:43:59.307508+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:59.307508+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44526104,"num_objects":5436,"num_object_clones":0,"num_object_copies":21744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5441,"num_write_kb":43505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5441","reported_seq":5513,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.324010+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:59.324010+0000","last_peered":"2024-09-16T04:43:59.324010+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:59.324010+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44553484,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5439,"num_write_kb":43510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3239}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5353","reported_seq":5407,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.401929+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:59.401929+0000","last_peered":"2024-09-16T04:43:59.401929+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:59.401929+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5353,"log_dups_size":0,"ondisk_log_size":5353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43806610,"num_objects":5348,"num_object_clones":0,"num_object_copies":21392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5350,"num_write_kb":42788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3159}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5288","reported_seq":5324,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.335439+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:59.335439+0000","last_peered":"2024-09-16T04:43:59.335439+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:59.335439+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5288,"log_dups_size":0,"ondisk_log_size":5288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43309702,"num_objects":5287,"num_object_clones":0,"num_object_copies":21148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5287,"num_write_kb":42295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3159}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5367","reported_seq":5437,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.242559+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:59.242559+0000","last_peered":"2024-09-16T04:43:59.242559+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:59.242559+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5367,"log_dups_size":0,"ondisk_log_size":5367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43956870,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":42927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3141}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5466","reported_seq":5516,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.254327+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:59.254327+0000","last_peered":"2024-09-16T04:43:59.254327+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:59.254327+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5466,"log_dups_size":0,"ondisk_log_size":5466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44741900,"num_objects":5462,"num_object_clones":0,"num_object_copies":21848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5464,"num_write_kb":43708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3228}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5311","reported_seq":5386,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.363866+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:59.363866+0000","last_peered":"2024-09-16T04:43:59.363866+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:59.363866+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5311,"log_dups_size":0,"ondisk_log_size":5311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43480332,"num_objects":5308,"num_object_clones":0,"num_object_copies":21232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5309,"num_write_kb":42469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3176}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5281","reported_seq":5343,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.732145+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:59.732145+0000","last_peered":"2024-09-16T04:43:59.732145+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:59.732145+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5281,"log_dups_size":0,"ondisk_log_size":5281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43218188,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5279,"num_write_kb":42220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":705545896,"num_objects":86131,"num_object_clones":0,"num_object_copies":344524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86158,"num_write_kb":689146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":17113088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1276776448,"data_stored":1276776448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86185,"ondisk_log_size":86185,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636680,"kb_used_data":343992,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93735160,"statfs":{"total":96636764160,"available":95984803840,"internally_reserved":0,"allocated":352247808,"data_stored":352076998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299691727},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,3,5,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671936,"kb_used_data":343664,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93699904,"statfs":{"total":96636764160,"available":95948701696,"internally_reserved":0,"allocated":351911936,"data_stored":351741126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,0,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051616,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406636,"kb_used_data":212508,"kb_used_omap":4,"kb_used_meta":194107,"kb_avail":93965204,"statfs":{"total":96636764160,"available":96220368896,"internally_reserved":0,"allocated":217608192,"data_stored":217437382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":198766287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,3,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628848,"kb_used_data":344544,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93742992,"statfs":{"total":96636764160,"available":95992823808,"internally_reserved":0,"allocated":352813056,"data_stored":352642246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":291106510},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":353009664,"data_stored":353009664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":218550272,"data_stored":218550272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":352620544,"data_stored":352620544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352595968,"data_stored":352595968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:03.286 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:03.298 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 189 20 87198 87178 3.60251 2.08594 0.0487898 0.0433546 2024-09-16T04:44:03.864 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:03.864 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:03.891 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-09-16T04:44:02.615050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":705545896,"num_objects":86131,"num_object_clones":0,"num_object_copies":344524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86158,"num_write_kb":689146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":17113088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86185,"ondisk_log_size":86185,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2344100,"kb_used_data":1244708,"kb_used_omap":14,"kb_used_meta":1099313,"kb_avail":375143260,"statfs":{"total":386547056640,"available":384146698240,"internally_reserved":0,"allocated":1274580992,"data_stored":1273897752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1125697073},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,10,4,12,15,9,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23273472,"num_objects":2841,"num_object_clones":0,"num_object_copies":11364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2841,"num_write_kb":22728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003272"},"pg_stats":[{"pgid":"1.f","version":"31'5410","reported_seq":5450,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.357809+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:43:59.357809+0000","last_peered":"2024-09-16T04:43:59.357809+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:43:59.357809+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5410,"log_dups_size":0,"ondisk_log_size":5410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44318720,"num_objects":5410,"num_object_clones":0,"num_object_copies":21640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":43280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5460","reported_seq":5522,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.735969+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:59.735969+0000","last_peered":"2024-09-16T04:43:59.735969+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:59.735969+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5460,"log_dups_size":0,"ondisk_log_size":5460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44692748,"num_objects":5456,"num_object_clones":0,"num_object_copies":21824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5458,"num_write_kb":43660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3245}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5430","reported_seq":5501,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.401870+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:43:59.401870+0000","last_peered":"2024-09-16T04:43:59.401870+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:43:59.401870+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5430,"log_dups_size":0,"ondisk_log_size":5430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44456582,"num_objects":5427,"num_object_clones":0,"num_object_copies":21708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5429,"num_write_kb":43422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5401","reported_seq":5462,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.664916+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:59.664916+0000","last_peered":"2024-09-16T04:43:59.664916+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:59.664916+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44209420,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":43181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5411","reported_seq":5463,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.110930+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:43:59.110930+0000","last_peered":"2024-09-16T04:43:59.110930+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:43:59.110930+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5411,"log_dups_size":0,"ondisk_log_size":5411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44299532,"num_objects":5408,"num_object_clones":0,"num_object_copies":21632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5409,"num_write_kb":43269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5264","reported_seq":5304,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.264309+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:43:59.264309+0000","last_peered":"2024-09-16T04:43:59.264309+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:43:59.264309+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5264,"log_dups_size":0,"ondisk_log_size":5264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43122688,"num_objects":5264,"num_object_clones":0,"num_object_copies":21056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5264,"num_write_kb":42112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5421","reported_seq":5482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.711151+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:59.711151+0000","last_peered":"2024-09-16T04:43:59.711151+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:59.711151+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5421,"log_dups_size":0,"ondisk_log_size":5421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44356876,"num_objects":5415,"num_object_clones":0,"num_object_copies":21660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5419,"num_write_kb":43332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3189}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5437","reported_seq":9678,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.780668+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:59.780668+0000","last_peered":"2024-09-16T04:43:59.780668+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:59.780668+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5437,"log_dups_size":0,"ondisk_log_size":5437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44496140,"num_objects":5432,"num_object_clones":0,"num_object_copies":21728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5435,"num_write_kb":43468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2093,"num_bytes_recovered":17113088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5444","reported_seq":5497,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.307508+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:43:59.307508+0000","last_peered":"2024-09-16T04:43:59.307508+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:43:59.307508+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44526104,"num_objects":5436,"num_object_clones":0,"num_object_copies":21744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5441,"num_write_kb":43505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5441","reported_seq":5513,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.324010+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:43:59.324010+0000","last_peered":"2024-09-16T04:43:59.324010+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:43:59.324010+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44553484,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5439,"num_write_kb":43510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3239}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5353","reported_seq":5407,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.401929+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:43:59.401929+0000","last_peered":"2024-09-16T04:43:59.401929+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:43:59.401929+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5353,"log_dups_size":0,"ondisk_log_size":5353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43806610,"num_objects":5348,"num_object_clones":0,"num_object_copies":21392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5350,"num_write_kb":42788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3159}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5288","reported_seq":5324,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.335439+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:43:59.335439+0000","last_peered":"2024-09-16T04:43:59.335439+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:43:59.335439+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5288,"log_dups_size":0,"ondisk_log_size":5288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43309702,"num_objects":5287,"num_object_clones":0,"num_object_copies":21148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5287,"num_write_kb":42295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3159}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5367","reported_seq":5437,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.242559+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:43:59.242559+0000","last_peered":"2024-09-16T04:43:59.242559+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:43:59.242559+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5367,"log_dups_size":0,"ondisk_log_size":5367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43956870,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":42927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3141}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5466","reported_seq":5516,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.254327+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:43:59.254327+0000","last_peered":"2024-09-16T04:43:59.254327+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:43:59.254327+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5466,"log_dups_size":0,"ondisk_log_size":5466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44741900,"num_objects":5462,"num_object_clones":0,"num_object_copies":21848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5464,"num_write_kb":43708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3228}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5311","reported_seq":5386,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.363866+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:43:59.363866+0000","last_peered":"2024-09-16T04:43:59.363866+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:43:59.363866+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5311,"log_dups_size":0,"ondisk_log_size":5311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43480332,"num_objects":5308,"num_object_clones":0,"num_object_copies":21232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5309,"num_write_kb":42469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3176}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5281","reported_seq":5343,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.732145+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:59.732145+0000","last_peered":"2024-09-16T04:43:59.732145+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:59.732145+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5281,"log_dups_size":0,"ondisk_log_size":5281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43218188,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5279,"num_write_kb":42220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":705545896,"num_objects":86131,"num_object_clones":0,"num_object_copies":344524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86158,"num_write_kb":689146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":17113088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1276776448,"data_stored":1276776448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86185,"ondisk_log_size":86185,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636680,"kb_used_data":343992,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93735160,"statfs":{"total":96636764160,"available":95984803840,"internally_reserved":0,"allocated":352247808,"data_stored":352076998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299691727},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,3,5,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671936,"kb_used_data":343664,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93699904,"statfs":{"total":96636764160,"available":95948701696,"internally_reserved":0,"allocated":351911936,"data_stored":351741126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,0,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051616,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406636,"kb_used_data":212508,"kb_used_omap":4,"kb_used_meta":194107,"kb_avail":93965204,"statfs":{"total":96636764160,"available":96220368896,"internally_reserved":0,"allocated":217608192,"data_stored":217437382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":198766287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,3,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628848,"kb_used_data":344544,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93742992,"statfs":{"total":96636764160,"available":95992823808,"internally_reserved":0,"allocated":352813056,"data_stored":352642246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":291106510},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":353009664,"data_stored":353009664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":218550272,"data_stored":218550272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":352620544,"data_stored":352620544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352595968,"data_stored":352595968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:03.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:44:03.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5410", 'reported_seq': 5450, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.357809+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:43:59.357809+0000', 'last_peered': '2024-09-16T04:43:59.357809+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:43:59.357809+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5410, 'log_dups_size': 0, 'ondisk_log_size': 5410, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44318720, 'num_objects': 5410, 'num_object_clones': 0, 'num_object_copies': 21640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5410, 'num_write_kb': 43280, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3191}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:03.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:44:03.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5460", 'reported_seq': 5522, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.735969+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:43:59.735969+0000', 'last_peered': '2024-09-16T04:43:59.735969+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:43:59.735969+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5460, 'log_dups_size': 0, 'ondisk_log_size': 5460, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44692748, 'num_objects': 5456, 'num_object_clones': 0, 'num_object_copies': 21824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5456, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5458, 'num_write_kb': 43660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3245}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:03.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:44:03.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5430", 'reported_seq': 5501, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.401870+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:43:59.401870+0000', 'last_peered': '2024-09-16T04:43:59.401870+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:43:59.401870+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5430, 'log_dups_size': 0, 'ondisk_log_size': 5430, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44456582, 'num_objects': 5427, 'num_object_clones': 0, 'num_object_copies': 21708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5427, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5429, 'num_write_kb': 43422, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3200}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:03.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:44:03.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5401", 'reported_seq': 5462, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.664916+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:43:59.664916+0000', 'last_peered': '2024-09-16T04:43:59.664916+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:43:59.664916+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5401, 'log_dups_size': 0, 'ondisk_log_size': 5401, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44209420, 'num_objects': 5397, 'num_object_clones': 0, 'num_object_copies': 21588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5399, 'num_write_kb': 43181, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3168}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:03.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:44:03.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5411", 'reported_seq': 5463, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.110930+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:43:59.110930+0000', 'last_peered': '2024-09-16T04:43:59.110930+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:43:59.110930+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5411, 'log_dups_size': 0, 'ondisk_log_size': 5411, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44299532, 'num_objects': 5408, 'num_object_clones': 0, 'num_object_copies': 21632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5409, 'num_write_kb': 43269, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3125}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:03.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:44:03.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5264", 'reported_seq': 5304, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.264309+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:43:59.264309+0000', 'last_peered': '2024-09-16T04:43:59.264309+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:43:59.264309+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5264, 'log_dups_size': 0, 'ondisk_log_size': 5264, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43122688, 'num_objects': 5264, 'num_object_clones': 0, 'num_object_copies': 21056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5264, 'num_write_kb': 42112, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3056}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:03.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:44:03.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5421", 'reported_seq': 5482, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.711151+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:43:59.711151+0000', 'last_peered': '2024-09-16T04:43:59.711151+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:43:59.711151+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5421, 'log_dups_size': 0, 'ondisk_log_size': 5421, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44356876, 'num_objects': 5415, 'num_object_clones': 0, 'num_object_copies': 21660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5415, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5419, 'num_write_kb': 43332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3189}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:03.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:44:03.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5437", 'reported_seq': 9678, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.780668+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:43:59.780668+0000', 'last_peered': '2024-09-16T04:43:59.780668+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:43:59.780668+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5437, 'log_dups_size': 0, 'ondisk_log_size': 5437, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44496140, 'num_objects': 5432, 'num_object_clones': 0, 'num_object_copies': 21728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5432, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5435, 'num_write_kb': 43468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2093, 'num_bytes_recovered': 17113088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 5283}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:03.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:44:03.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5444", 'reported_seq': 5497, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.307508+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:43:59.307508+0000', 'last_peered': '2024-09-16T04:43:59.307508+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:43:59.307508+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5444, 'log_dups_size': 0, 'ondisk_log_size': 5444, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44526104, 'num_objects': 5436, 'num_object_clones': 0, 'num_object_copies': 21744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5436, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5441, 'num_write_kb': 43505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3096}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:03.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:44:03.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5441", 'reported_seq': 5513, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.324010+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:43:59.324010+0000', 'last_peered': '2024-09-16T04:43:59.324010+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:43:59.324010+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5441, 'log_dups_size': 0, 'ondisk_log_size': 5441, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44553484, 'num_objects': 5439, 'num_object_clones': 0, 'num_object_copies': 21756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5439, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5439, 'num_write_kb': 43510, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3239}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:03.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:44:03.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5353", 'reported_seq': 5407, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.401929+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:43:59.401929+0000', 'last_peered': '2024-09-16T04:43:59.401929+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:43:59.401929+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5353, 'log_dups_size': 0, 'ondisk_log_size': 5353, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43806610, 'num_objects': 5348, 'num_object_clones': 0, 'num_object_copies': 21392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5348, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5350, 'num_write_kb': 42788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3159}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:03.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:44:03.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5288", 'reported_seq': 5324, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.335439+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:43:59.335439+0000', 'last_peered': '2024-09-16T04:43:59.335439+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:43:59.335439+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5288, 'log_dups_size': 0, 'ondisk_log_size': 5288, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43309702, 'num_objects': 5287, 'num_object_clones': 0, 'num_object_copies': 21148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5287, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5287, 'num_write_kb': 42295, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:03.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:44:03.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5367", 'reported_seq': 5437, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.242559+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:43:59.242559+0000', 'last_peered': '2024-09-16T04:43:59.242559+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:43:59.242559+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5367, 'log_dups_size': 0, 'ondisk_log_size': 5367, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43956870, 'num_objects': 5366, 'num_object_clones': 0, 'num_object_copies': 21464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5366, 'num_write_kb': 42927, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3141}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:03.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:44:03.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5466", 'reported_seq': 5516, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.254327+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:43:59.254327+0000', 'last_peered': '2024-09-16T04:43:59.254327+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:43:59.254327+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5466, 'log_dups_size': 0, 'ondisk_log_size': 5466, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44741900, 'num_objects': 5462, 'num_object_clones': 0, 'num_object_copies': 21848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5464, 'num_write_kb': 43708, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3228}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:03.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:44:03.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5311", 'reported_seq': 5386, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.363866+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:43:59.363866+0000', 'last_peered': '2024-09-16T04:43:59.363866+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:43:59.363866+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5311, 'log_dups_size': 0, 'ondisk_log_size': 5311, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43480332, 'num_objects': 5308, 'num_object_clones': 0, 'num_object_copies': 21232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5309, 'num_write_kb': 42469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3176}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:03.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:44:03.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5281", 'reported_seq': 5343, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:43:59.732145+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:43:59.732145+0000', 'last_peered': '2024-09-16T04:43:59.732145+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:43:59.732145+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5281, 'log_dups_size': 0, 'ondisk_log_size': 5281, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43218188, 'num_objects': 5276, 'num_object_clones': 0, 'num_object_copies': 21104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5279, 'num_write_kb': 42220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:03.896 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:44:04.299 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 190 20 87455 87435 3.59412 2.00781 0.102821 0.0434582 2024-09-16T04:44:04.604 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:04.604 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33464/344524 objects degraded (9.713%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":256,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":86131,"data_bytes":705545896,"bytes_used":2400358400,"bytes_avail":384146698240,"bytes_total":386547056640,"degraded_objects":33464,"degraded_total":344524,"degraded_ratio":0.097131114232970703,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":99642,"recovering_keys_per_sec":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"write_bytes_sec":1938927,"read_op_per_sec":0,"write_op_per_sec":236},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:44:04.641 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:44:04.642 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:05.235 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:05.236 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:05.259 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-09-16T04:44:04.615517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":713631400,"num_objects":87118,"num_object_clones":0,"num_object_copies":348472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87145,"num_write_kb":697042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":17113088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87172,"ondisk_log_size":87172,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2347164,"kb_used_data":1256092,"kb_used_omap":14,"kb_used_meta":1090993,"kb_avail":375140196,"statfs":{"total":386547056640,"available":384143560704,"internally_reserved":0,"allocated":1286238208,"data_stored":1285554968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1117177393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,6,4,10,9,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31358976,"num_objects":3828,"num_object_clones":0,"num_object_copies":15312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3828,"num_write_kb":30624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003174"},"pg_stats":[{"pgid":"1.f","version":"31'5493","reported_seq":5530,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.363812+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:04.363812+0000","last_peered":"2024-09-16T04:44:04.363812+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:04.363812+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5493,"log_dups_size":0,"ondisk_log_size":5493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44998656,"num_objects":5493,"num_object_clones":0,"num_object_copies":21972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5493,"num_write_kb":43944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5460","reported_seq":5522,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.735969+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:43:59.735969+0000","last_peered":"2024-09-16T04:43:59.735969+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:43:59.735969+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5460,"log_dups_size":0,"ondisk_log_size":5460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44692748,"num_objects":5456,"num_object_clones":0,"num_object_copies":21824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5458,"num_write_kb":43660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3245}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5523","reported_seq":5594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.409588+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:04.409588+0000","last_peered":"2024-09-16T04:44:04.409588+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:04.409588+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5523,"log_dups_size":0,"ondisk_log_size":5523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45218438,"num_objects":5520,"num_object_clones":0,"num_object_copies":22080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5522,"num_write_kb":44166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3293}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5401","reported_seq":5462,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.664916+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:43:59.664916+0000","last_peered":"2024-09-16T04:43:59.664916+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:43:59.664916+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44209420,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":43181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5534","reported_seq":5586,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.280784+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:04.280784+0000","last_peered":"2024-09-16T04:44:04.280784+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:04.280784+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5534,"log_dups_size":0,"ondisk_log_size":5534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45307148,"num_objects":5531,"num_object_clones":0,"num_object_copies":22124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5532,"num_write_kb":44253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3248}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5356","reported_seq":5392,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.387460+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:04.387460+0000","last_peered":"2024-09-16T04:44:04.387460+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:04.387460+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5356,"log_dups_size":0,"ondisk_log_size":5356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43876352,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5356,"num_write_kb":42848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5421","reported_seq":5482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.711151+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:43:59.711151+0000","last_peered":"2024-09-16T04:43:59.711151+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:43:59.711151+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5421,"log_dups_size":0,"ondisk_log_size":5421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44356876,"num_objects":5415,"num_object_clones":0,"num_object_copies":21660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5419,"num_write_kb":43332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3189}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5437","reported_seq":9678,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.780668+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:43:59.780668+0000","last_peered":"2024-09-16T04:43:59.780668+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:43:59.780668+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5437,"log_dups_size":0,"ondisk_log_size":5437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44496140,"num_objects":5432,"num_object_clones":0,"num_object_copies":21728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5435,"num_write_kb":43468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2093,"num_bytes_recovered":17113088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5532","reported_seq":5584,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.381501+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:44:04.381501+0000","last_peered":"2024-09-16T04:44:04.381501+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:04.381501+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45247000,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5529,"num_write_kb":44209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5521","reported_seq":5594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.338815+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:04.338815+0000","last_peered":"2024-09-16T04:44:04.338815+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:04.338815+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5521,"log_dups_size":0,"ondisk_log_size":5521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45208844,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5519,"num_write_kb":44150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3319}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5433","reported_seq":5487,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.413286+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:04.413286+0000","last_peered":"2024-09-16T04:44:04.413286+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:04.413286+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5433,"log_dups_size":0,"ondisk_log_size":5433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44461970,"num_objects":5428,"num_object_clones":0,"num_object_copies":21712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5430,"num_write_kb":43428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5370","reported_seq":5406,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.323473+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:04.323473+0000","last_peered":"2024-09-16T04:44:04.323473+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:04.323473+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43981446,"num_objects":5369,"num_object_clones":0,"num_object_copies":21476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5369,"num_write_kb":42951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3241}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5457","reported_seq":5527,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.344428+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:04.344428+0000","last_peered":"2024-09-16T04:44:04.344428+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:04.344428+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5457,"log_dups_size":0,"ondisk_log_size":5457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44694150,"num_objects":5456,"num_object_clones":0,"num_object_copies":21824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5456,"num_write_kb":43647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3231}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5556","reported_seq":5606,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.421050+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:04.421050+0000","last_peered":"2024-09-16T04:44:04.421050+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:04.421050+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5556,"log_dups_size":0,"ondisk_log_size":5556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45479180,"num_objects":5552,"num_object_clones":0,"num_object_copies":22208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5554,"num_write_kb":44428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3318}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5397","reported_seq":5472,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.351395+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:04.351395+0000","last_peered":"2024-09-16T04:44:04.351395+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:04.351395+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5397,"log_dups_size":0,"ondisk_log_size":5397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44184844,"num_objects":5394,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5395,"num_write_kb":43157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3262}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5281","reported_seq":5343,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:59.732145+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:43:59.732145+0000","last_peered":"2024-09-16T04:43:59.732145+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:43:59.732145+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5281,"log_dups_size":0,"ondisk_log_size":5281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43218188,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5279,"num_write_kb":42220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":713631400,"num_objects":87118,"num_object_clones":0,"num_object_copies":348472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87145,"num_write_kb":697042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":17113088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1288314880,"data_stored":1288314880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87172,"ondisk_log_size":87172,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629928,"kb_used_data":349720,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93741912,"statfs":{"total":96636764160,"available":95991717888,"internally_reserved":0,"allocated":358113280,"data_stored":357942470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":286912207},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681752,"kb_used_data":349320,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":93690088,"statfs":{"total":96636764160,"available":95938650112,"internally_reserved":0,"allocated":357703680,"data_stored":357532870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":340392389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051616,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406636,"kb_used_data":212508,"kb_used_omap":4,"kb_used_meta":194107,"kb_avail":93965204,"statfs":{"total":96636764160,"available":96220368896,"internally_reserved":0,"allocated":217608192,"data_stored":217437382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":198766287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,3,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628848,"kb_used_data":344544,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93742992,"statfs":{"total":96636764160,"available":95992823808,"internally_reserved":0,"allocated":352813056,"data_stored":352642246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":291106510},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":353009664,"data_stored":353009664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":218550272,"data_stored":218550272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":358383616,"data_stored":358383616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":358371328,"data_stored":358371328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:05.299 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 191 20 87717 87697 3.58602 2.04688 0.0677114 0.0435592 2024-09-16T04:44:06.299 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 192 20 88013 87993 3.57939 2.3125 0.0505649 0.0436371 2024-09-16T04:44:07.300 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 193 20 88315 88295 3.57306 2.35938 0.0476166 0.0437176 2024-09-16T04:44:08.262 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:08.302 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 194 20 88610 88590 3.56651 2.30469 0.0666844 0.0437977 2024-09-16T04:44:08.839 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:08.840 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:08.864 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-09-16T04:44:08.616646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":717170344,"num_objects":87550,"num_object_clones":0,"num_object_copies":350200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87577,"num_write_kb":700498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2192,"num_bytes_recovered":17719296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87604,"ondisk_log_size":87604,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2367004,"kb_used_data":1267612,"kb_used_omap":14,"kb_used_meta":1099313,"kb_avail":375120356,"statfs":{"total":386547056640,"available":384123244544,"internally_reserved":0,"allocated":1298034688,"data_stored":1297351448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1125697073},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,5,6,10,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23306240,"num_objects":2845,"num_object_clones":0,"num_object_copies":11380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2845,"num_write_kb":22760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1294336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003257"},"pg_stats":[{"pgid":"1.f","version":"31'5493","reported_seq":5530,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.363812+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:04.363812+0000","last_peered":"2024-09-16T04:44:04.363812+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:04.363812+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5493,"log_dups_size":0,"ondisk_log_size":5493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44998656,"num_objects":5493,"num_object_clones":0,"num_object_copies":21972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5493,"num_write_kb":43944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5550","reported_seq":5612,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.520824+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:04.520824+0000","last_peered":"2024-09-16T04:44:04.520824+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:04.520824+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5550,"log_dups_size":0,"ondisk_log_size":5550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45430028,"num_objects":5546,"num_object_clones":0,"num_object_copies":22184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5548,"num_write_kb":44380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5523","reported_seq":5594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.409588+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:04.409588+0000","last_peered":"2024-09-16T04:44:04.409588+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:04.409588+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5523,"log_dups_size":0,"ondisk_log_size":5523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45218438,"num_objects":5520,"num_object_clones":0,"num_object_copies":22080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5522,"num_write_kb":44166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3293}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5495","reported_seq":5556,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.705252+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:04.705252+0000","last_peered":"2024-09-16T04:44:04.705252+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:04.705252+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5495,"log_dups_size":0,"ondisk_log_size":5495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44979468,"num_objects":5491,"num_object_clones":0,"num_object_copies":21964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5493,"num_write_kb":43933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5534","reported_seq":5586,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.280784+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:04.280784+0000","last_peered":"2024-09-16T04:44:04.280784+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:04.280784+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5534,"log_dups_size":0,"ondisk_log_size":5534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45307148,"num_objects":5531,"num_object_clones":0,"num_object_copies":22124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5532,"num_write_kb":44253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3248}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5356","reported_seq":5392,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.387460+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:04.387460+0000","last_peered":"2024-09-16T04:44:04.387460+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:04.387460+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5356,"log_dups_size":0,"ondisk_log_size":5356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43876352,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5356,"num_write_kb":42848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5498","reported_seq":5558,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.782979+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:04.782979+0000","last_peered":"2024-09-16T04:44:04.782979+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:04.782979+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5498,"log_dups_size":0,"ondisk_log_size":5498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44987660,"num_objects":5492,"num_object_clones":0,"num_object_copies":21968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5496,"num_write_kb":43948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5538","reported_seq":9927,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.754986+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:44:04.754986+0000","last_peered":"2024-09-16T04:44:04.754986+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:44:04.754986+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45323532,"num_objects":5533,"num_object_clones":0,"num_object_copies":22132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5536,"num_write_kb":44276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2167,"num_bytes_recovered":17719296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5458}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5532","reported_seq":5584,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.381501+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:44:04.381501+0000","last_peered":"2024-09-16T04:44:04.381501+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:04.381501+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45247000,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5529,"num_write_kb":44209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5521","reported_seq":5594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.338815+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:04.338815+0000","last_peered":"2024-09-16T04:44:04.338815+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:04.338815+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5521,"log_dups_size":0,"ondisk_log_size":5521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45208844,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5519,"num_write_kb":44150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3319}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5433","reported_seq":5487,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.413286+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:04.413286+0000","last_peered":"2024-09-16T04:44:04.413286+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:04.413286+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5433,"log_dups_size":0,"ondisk_log_size":5433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44461970,"num_objects":5428,"num_object_clones":0,"num_object_copies":21712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5430,"num_write_kb":43428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5370","reported_seq":5406,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.323473+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:04.323473+0000","last_peered":"2024-09-16T04:44:04.323473+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:04.323473+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43981446,"num_objects":5369,"num_object_clones":0,"num_object_copies":21476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5369,"num_write_kb":42951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3241}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5457","reported_seq":5527,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.344428+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:04.344428+0000","last_peered":"2024-09-16T04:44:04.344428+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:04.344428+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5457,"log_dups_size":0,"ondisk_log_size":5457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44694150,"num_objects":5456,"num_object_clones":0,"num_object_copies":21824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5456,"num_write_kb":43647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3231}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5556","reported_seq":5606,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.421050+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:04.421050+0000","last_peered":"2024-09-16T04:44:04.421050+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:04.421050+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5556,"log_dups_size":0,"ondisk_log_size":5556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45479180,"num_objects":5552,"num_object_clones":0,"num_object_copies":22208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5554,"num_write_kb":44428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3318}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5397","reported_seq":5472,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.351395+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:04.351395+0000","last_peered":"2024-09-16T04:44:04.351395+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:04.351395+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5397,"log_dups_size":0,"ondisk_log_size":5397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44184844,"num_objects":5394,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5395,"num_write_kb":43157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3262}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5351","reported_seq":5413,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.666024+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:04.666024+0000","last_peered":"2024-09-16T04:44:04.666024+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:04.666024+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5351,"log_dups_size":0,"ondisk_log_size":5351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43791628,"num_objects":5346,"num_object_clones":0,"num_object_copies":21384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5349,"num_write_kb":42780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":717170344,"num_objects":87550,"num_object_clones":0,"num_object_copies":350200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87577,"num_write_kb":700498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2192,"num_bytes_recovered":17719296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1300115456,"data_stored":1300115456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87604,"ondisk_log_size":87604,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629928,"kb_used_data":349720,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93741912,"statfs":{"total":96636764160,"available":95991717888,"internally_reserved":0,"allocated":358113280,"data_stored":357942470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":286912207},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681752,"kb_used_data":349320,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":93690088,"statfs":{"total":96636764160,"available":95938650112,"internally_reserved":0,"allocated":357703680,"data_stored":357532870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":340392389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051617,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416688,"kb_used_data":218400,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93955152,"statfs":{"total":96636764160,"available":96210075648,"internally_reserved":0,"allocated":223641600,"data_stored":223470790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":203026127},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":638636,"kb_used_data":350172,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93733204,"statfs":{"total":96636764160,"available":95982800896,"internally_reserved":0,"allocated":358576128,"data_stored":358405318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":295366350},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":358768640,"data_stored":358768640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":224591872,"data_stored":224591872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":358383616,"data_stored":358383616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":358371328,"data_stored":358371328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:08.865 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:09.302 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 195 20 88874 88854 3.55876 2.0625 0.0921991 0.0438917 2024-09-16T04:44:09.500 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:09.500 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:09.524 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-09-16T04:44:08.616646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":717170344,"num_objects":87550,"num_object_clones":0,"num_object_copies":350200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87577,"num_write_kb":700498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2192,"num_bytes_recovered":17719296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87604,"ondisk_log_size":87604,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2367004,"kb_used_data":1267612,"kb_used_omap":14,"kb_used_meta":1099313,"kb_avail":375120356,"statfs":{"total":386547056640,"available":384123244544,"internally_reserved":0,"allocated":1298034688,"data_stored":1297351448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1125697073},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,5,6,10,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23306240,"num_objects":2845,"num_object_clones":0,"num_object_copies":11380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2845,"num_write_kb":22760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":1294336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003257"},"pg_stats":[{"pgid":"1.f","version":"31'5493","reported_seq":5530,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.363812+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:04.363812+0000","last_peered":"2024-09-16T04:44:04.363812+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:04.363812+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5493,"log_dups_size":0,"ondisk_log_size":5493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44998656,"num_objects":5493,"num_object_clones":0,"num_object_copies":21972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5493,"num_write_kb":43944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5550","reported_seq":5612,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.520824+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:04.520824+0000","last_peered":"2024-09-16T04:44:04.520824+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:04.520824+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5550,"log_dups_size":0,"ondisk_log_size":5550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45430028,"num_objects":5546,"num_object_clones":0,"num_object_copies":22184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5548,"num_write_kb":44380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5523","reported_seq":5594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.409588+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:04.409588+0000","last_peered":"2024-09-16T04:44:04.409588+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:04.409588+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5523,"log_dups_size":0,"ondisk_log_size":5523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45218438,"num_objects":5520,"num_object_clones":0,"num_object_copies":22080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5522,"num_write_kb":44166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3293}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5495","reported_seq":5556,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.705252+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:04.705252+0000","last_peered":"2024-09-16T04:44:04.705252+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:04.705252+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5495,"log_dups_size":0,"ondisk_log_size":5495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44979468,"num_objects":5491,"num_object_clones":0,"num_object_copies":21964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5493,"num_write_kb":43933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5534","reported_seq":5586,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.280784+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:04.280784+0000","last_peered":"2024-09-16T04:44:04.280784+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:04.280784+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5534,"log_dups_size":0,"ondisk_log_size":5534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45307148,"num_objects":5531,"num_object_clones":0,"num_object_copies":22124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5532,"num_write_kb":44253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3248}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5356","reported_seq":5392,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.387460+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:04.387460+0000","last_peered":"2024-09-16T04:44:04.387460+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:04.387460+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5356,"log_dups_size":0,"ondisk_log_size":5356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43876352,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5356,"num_write_kb":42848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5498","reported_seq":5558,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.782979+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:04.782979+0000","last_peered":"2024-09-16T04:44:04.782979+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:04.782979+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5498,"log_dups_size":0,"ondisk_log_size":5498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44987660,"num_objects":5492,"num_object_clones":0,"num_object_copies":21968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5496,"num_write_kb":43948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5538","reported_seq":9927,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.754986+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:44:04.754986+0000","last_peered":"2024-09-16T04:44:04.754986+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:44:04.754986+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45323532,"num_objects":5533,"num_object_clones":0,"num_object_copies":22132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5536,"num_write_kb":44276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2167,"num_bytes_recovered":17719296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5458}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5532","reported_seq":5584,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.381501+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:44:04.381501+0000","last_peered":"2024-09-16T04:44:04.381501+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:04.381501+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45247000,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5529,"num_write_kb":44209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5521","reported_seq":5594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.338815+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:04.338815+0000","last_peered":"2024-09-16T04:44:04.338815+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:04.338815+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5521,"log_dups_size":0,"ondisk_log_size":5521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45208844,"num_objects":5519,"num_object_clones":0,"num_object_copies":22076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5519,"num_write_kb":44150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3319}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5433","reported_seq":5487,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.413286+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:04.413286+0000","last_peered":"2024-09-16T04:44:04.413286+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:04.413286+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5433,"log_dups_size":0,"ondisk_log_size":5433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44461970,"num_objects":5428,"num_object_clones":0,"num_object_copies":21712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5430,"num_write_kb":43428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5370","reported_seq":5406,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.323473+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:04.323473+0000","last_peered":"2024-09-16T04:44:04.323473+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:04.323473+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43981446,"num_objects":5369,"num_object_clones":0,"num_object_copies":21476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5369,"num_write_kb":42951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3241}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5457","reported_seq":5527,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.344428+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:04.344428+0000","last_peered":"2024-09-16T04:44:04.344428+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:04.344428+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5457,"log_dups_size":0,"ondisk_log_size":5457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44694150,"num_objects":5456,"num_object_clones":0,"num_object_copies":21824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5456,"num_write_kb":43647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3231}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5556","reported_seq":5606,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.421050+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:04.421050+0000","last_peered":"2024-09-16T04:44:04.421050+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:04.421050+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5556,"log_dups_size":0,"ondisk_log_size":5556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45479180,"num_objects":5552,"num_object_clones":0,"num_object_copies":22208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5554,"num_write_kb":44428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3318}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5397","reported_seq":5472,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.351395+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:04.351395+0000","last_peered":"2024-09-16T04:44:04.351395+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:04.351395+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5397,"log_dups_size":0,"ondisk_log_size":5397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44184844,"num_objects":5394,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5395,"num_write_kb":43157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3262}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5351","reported_seq":5413,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:04.666024+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:04.666024+0000","last_peered":"2024-09-16T04:44:04.666024+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:04.666024+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5351,"log_dups_size":0,"ondisk_log_size":5351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43791628,"num_objects":5346,"num_object_clones":0,"num_object_copies":21384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5349,"num_write_kb":42780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":717170344,"num_objects":87550,"num_object_clones":0,"num_object_copies":350200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87577,"num_write_kb":700498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2192,"num_bytes_recovered":17719296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1300115456,"data_stored":1300115456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87604,"ondisk_log_size":87604,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629928,"kb_used_data":349720,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93741912,"statfs":{"total":96636764160,"available":95991717888,"internally_reserved":0,"allocated":358113280,"data_stored":357942470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":286912207},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681752,"kb_used_data":349320,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":93690088,"statfs":{"total":96636764160,"available":95938650112,"internally_reserved":0,"allocated":357703680,"data_stored":357532870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":340392389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051617,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416688,"kb_used_data":218400,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93955152,"statfs":{"total":96636764160,"available":96210075648,"internally_reserved":0,"allocated":223641600,"data_stored":223470790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":203026127},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":638636,"kb_used_data":350172,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93733204,"statfs":{"total":96636764160,"available":95982800896,"internally_reserved":0,"allocated":358576128,"data_stored":358405318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":295366350},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":358768640,"data_stored":358768640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":224591872,"data_stored":224591872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":358383616,"data_stored":358383616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":358371328,"data_stored":358371328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:09.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:44:09.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5493", 'reported_seq': 5530, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.363812+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:44:04.363812+0000', 'last_peered': '2024-09-16T04:44:04.363812+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:44:04.363812+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5493, 'log_dups_size': 0, 'ondisk_log_size': 5493, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44998656, 'num_objects': 5493, 'num_object_clones': 0, 'num_object_copies': 21972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5493, 'num_write_kb': 43944, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3274}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:09.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:44:09.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5550", 'reported_seq': 5612, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.520824+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:44:04.520824+0000', 'last_peered': '2024-09-16T04:44:04.520824+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:44:04.520824+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5550, 'log_dups_size': 0, 'ondisk_log_size': 5550, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45430028, 'num_objects': 5546, 'num_object_clones': 0, 'num_object_copies': 22184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5546, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5548, 'num_write_kb': 44380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3335}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:09.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:44:09.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5523", 'reported_seq': 5594, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.409588+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:44:04.409588+0000', 'last_peered': '2024-09-16T04:44:04.409588+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:44:04.409588+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5523, 'log_dups_size': 0, 'ondisk_log_size': 5523, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45218438, 'num_objects': 5520, 'num_object_clones': 0, 'num_object_copies': 22080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5520, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5522, 'num_write_kb': 44166, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3293}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:09.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:44:09.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5495", 'reported_seq': 5556, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.705252+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:44:04.705252+0000', 'last_peered': '2024-09-16T04:44:04.705252+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:44:04.705252+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5495, 'log_dups_size': 0, 'ondisk_log_size': 5495, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44979468, 'num_objects': 5491, 'num_object_clones': 0, 'num_object_copies': 21964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5493, 'num_write_kb': 43933, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:09.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:44:09.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5534", 'reported_seq': 5586, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.280784+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:44:04.280784+0000', 'last_peered': '2024-09-16T04:44:04.280784+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:44:04.280784+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5534, 'log_dups_size': 0, 'ondisk_log_size': 5534, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45307148, 'num_objects': 5531, 'num_object_clones': 0, 'num_object_copies': 22124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5532, 'num_write_kb': 44253, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3248}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:09.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:44:09.527 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5356", 'reported_seq': 5392, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.387460+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:44:04.387460+0000', 'last_peered': '2024-09-16T04:44:04.387460+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:44:04.387460+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5356, 'log_dups_size': 0, 'ondisk_log_size': 5356, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43876352, 'num_objects': 5356, 'num_object_clones': 0, 'num_object_copies': 21424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5356, 'num_write_kb': 42848, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3148}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:09.527 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:44:09.527 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5498", 'reported_seq': 5558, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.782979+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:44:04.782979+0000', 'last_peered': '2024-09-16T04:44:04.782979+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:44:04.782979+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5498, 'log_dups_size': 0, 'ondisk_log_size': 5498, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44987660, 'num_objects': 5492, 'num_object_clones': 0, 'num_object_copies': 21968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5496, 'num_write_kb': 43948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3266}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:09.527 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:44:09.527 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5538", 'reported_seq': 9927, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.754986+0000', 'last_change': '2024-09-16T04:41:27.727916+0000', 'last_active': '2024-09-16T04:44:04.754986+0000', 'last_peered': '2024-09-16T04:44:04.754986+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:44:04.754986+0000', 'last_undegraded': '2024-09-16T04:41:27.527383+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5538, 'log_dups_size': 0, 'ondisk_log_size': 5538, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45323532, 'num_objects': 5533, 'num_object_clones': 0, 'num_object_copies': 22132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 75, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5533, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5536, 'num_write_kb': 44276, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2167, 'num_bytes_recovered': 17719296, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 5458}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:09.527 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:44:09.527 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5532", 'reported_seq': 5584, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.381501+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:44:04.381501+0000', 'last_peered': '2024-09-16T04:44:04.381501+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:44:04.381501+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5532, 'log_dups_size': 0, 'ondisk_log_size': 5532, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45247000, 'num_objects': 5524, 'num_object_clones': 0, 'num_object_copies': 22096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5529, 'num_write_kb': 44209, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3184}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:09.527 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:44:09.527 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5521", 'reported_seq': 5594, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.338815+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:44:04.338815+0000', 'last_peered': '2024-09-16T04:44:04.338815+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:44:04.338815+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5521, 'log_dups_size': 0, 'ondisk_log_size': 5521, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45208844, 'num_objects': 5519, 'num_object_clones': 0, 'num_object_copies': 22076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5519, 'num_write_kb': 44150, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3319}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:09.527 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:44:09.527 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5433", 'reported_seq': 5487, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.413286+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:44:04.413286+0000', 'last_peered': '2024-09-16T04:44:04.413286+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:44:04.413286+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5433, 'log_dups_size': 0, 'ondisk_log_size': 5433, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44461970, 'num_objects': 5428, 'num_object_clones': 0, 'num_object_copies': 21712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5428, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5430, 'num_write_kb': 43428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3239}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:09.528 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:44:09.528 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5370", 'reported_seq': 5406, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.323473+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:44:04.323473+0000', 'last_peered': '2024-09-16T04:44:04.323473+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:44:04.323473+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5370, 'log_dups_size': 0, 'ondisk_log_size': 5370, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43981446, 'num_objects': 5369, 'num_object_clones': 0, 'num_object_copies': 21476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5369, 'num_write_kb': 42951, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3241}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:09.528 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:44:09.528 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5457", 'reported_seq': 5527, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.344428+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:44:04.344428+0000', 'last_peered': '2024-09-16T04:44:04.344428+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:44:04.344428+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5457, 'log_dups_size': 0, 'ondisk_log_size': 5457, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44694150, 'num_objects': 5456, 'num_object_clones': 0, 'num_object_copies': 21824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5456, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5456, 'num_write_kb': 43647, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3231}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:09.528 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:44:09.528 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5556", 'reported_seq': 5606, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.421050+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:44:04.421050+0000', 'last_peered': '2024-09-16T04:44:04.421050+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:44:04.421050+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5556, 'log_dups_size': 0, 'ondisk_log_size': 5556, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45479180, 'num_objects': 5552, 'num_object_clones': 0, 'num_object_copies': 22208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5552, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5554, 'num_write_kb': 44428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:09.528 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:44:09.528 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5397", 'reported_seq': 5472, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.351395+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:44:04.351395+0000', 'last_peered': '2024-09-16T04:44:04.351395+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:44:04.351395+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5397, 'log_dups_size': 0, 'ondisk_log_size': 5397, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44184844, 'num_objects': 5394, 'num_object_clones': 0, 'num_object_copies': 21576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5394, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5395, 'num_write_kb': 43157, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:09.528 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:44:09.528 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5351", 'reported_seq': 5413, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:04.666024+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:44:04.666024+0000', 'last_peered': '2024-09-16T04:44:04.666024+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:44:04.666024+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5351, 'log_dups_size': 0, 'ondisk_log_size': 5351, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43791628, 'num_objects': 5346, 'num_object_clones': 0, 'num_object_copies': 21384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5349, 'num_write_kb': 42780, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3107}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:09.529 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:44:10.288 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:10.288 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33390/350200 objects degraded (9.535%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":262,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":87550,"data_bytes":717170344,"bytes_used":2423812096,"bytes_avail":384123244544,"bytes_total":386547056640,"degraded_objects":33390,"degraded_total":350200,"degraded_ratio":0.095345516847515702,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":107832,"recovering_keys_per_sec":0,"num_objects_recovered":158,"num_bytes_recovered":1294336,"num_keys_recovered":0,"write_bytes_sec":1941659,"read_op_per_sec":0,"write_op_per_sec":237},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:44:10.314 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 196 20 89143 89123 3.55111 2.10156 0.0790686 0.0439881 2024-09-16T04:44:10.322 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:44:10.322 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:10.924 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:10.924 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:10.955 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-16T04:44:10.617130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":729015976,"num_objects":88996,"num_object_clones":0,"num_object_copies":355984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89023,"num_write_kb":712066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2260,"num_bytes_recovered":18276352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89050,"ondisk_log_size":89050,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2488180,"kb_used_data":1284652,"kb_used_omap":14,"kb_used_meta":1203441,"kb_avail":374999180,"statfs":{"total":386547056640,"available":383999160320,"internally_reserved":0,"allocated":1315483648,"data_stored":1314800408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1232324145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,8,10,10,15,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35151872,"num_objects":4291,"num_object_clones":0,"num_object_copies":17164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4291,"num_write_kb":34328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":1851392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003211"},"pg_stats":[{"pgid":"1.f","version":"31'5584","reported_seq":5624,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.525729+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:09.525729+0000","last_peered":"2024-09-16T04:44:09.525729+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:09.525729+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45744128,"num_objects":5584,"num_object_clones":0,"num_object_copies":22336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5584,"num_write_kb":44672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5639","reported_seq":5701,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.881664+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:09.881664+0000","last_peered":"2024-09-16T04:44:09.881664+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:09.881664+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46159116,"num_objects":5635,"num_object_clones":0,"num_object_copies":22540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5637,"num_write_kb":45092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5623","reported_seq":5694,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.420174+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:09.420174+0000","last_peered":"2024-09-16T04:44:09.420174+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:09.420174+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5623,"log_dups_size":0,"ondisk_log_size":5623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46037638,"num_objects":5620,"num_object_clones":0,"num_object_copies":22480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5622,"num_write_kb":44966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3393}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5584","reported_seq":5646,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.841291+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:09.841291+0000","last_peered":"2024-09-16T04:44:09.841291+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:09.841291+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708556,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5582,"num_write_kb":44645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3351}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5642","reported_seq":5694,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.362093+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:09.362093+0000","last_peered":"2024-09-16T04:44:09.362093+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:09.362093+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5642,"log_dups_size":0,"ondisk_log_size":5642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46191884,"num_objects":5639,"num_object_clones":0,"num_object_copies":22556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5640,"num_write_kb":45117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5441","reported_seq":5479,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.527818+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:09.527818+0000","last_peered":"2024-09-16T04:44:09.527818+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:09.527818+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44572672,"num_objects":5441,"num_object_clones":0,"num_object_copies":21764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5441,"num_write_kb":43528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5597","reported_seq":5658,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.844599+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:09.844599+0000","last_peered":"2024-09-16T04:44:09.844599+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:09.844599+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5597,"log_dups_size":0,"ondisk_log_size":5597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45798668,"num_objects":5591,"num_object_clones":0,"num_object_copies":22364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5595,"num_write_kb":44740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3365}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5640","reported_seq":10167,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.904728+0000","last_change":"2024-09-16T04:41:27.727916+0000","last_active":"2024-09-16T04:44:09.904728+0000","last_peered":"2024-09-16T04:44:09.904728+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:44:09.904728+0000","last_undegraded":"2024-09-16T04:41:27.527383+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5640,"log_dups_size":0,"ondisk_log_size":5640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46159116,"num_objects":5635,"num_object_clones":0,"num_object_copies":22540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5638,"num_write_kb":45092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2235,"num_bytes_recovered":18276352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":5628}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5616","reported_seq":5666,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.448563+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:44:09.448563+0000","last_peered":"2024-09-16T04:44:09.448563+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:09.448563+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5616,"log_dups_size":0,"ondisk_log_size":5616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45935128,"num_objects":5608,"num_object_clones":0,"num_object_copies":22432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5613,"num_write_kb":44881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5610","reported_seq":5684,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.341518+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:09.341518+0000","last_peered":"2024-09-16T04:44:09.341518+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:09.341518+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5610,"log_dups_size":0,"ondisk_log_size":5610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45937932,"num_objects":5608,"num_object_clones":0,"num_object_copies":22432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5608,"num_write_kb":44862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3408}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5523","reported_seq":5577,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.493207+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:09.493207+0000","last_peered":"2024-09-16T04:44:09.493207+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:09.493207+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5523,"log_dups_size":0,"ondisk_log_size":5523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45199250,"num_objects":5518,"num_object_clones":0,"num_object_copies":22072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5520,"num_write_kb":44148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5457","reported_seq":5493,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.421765+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:09.421765+0000","last_peered":"2024-09-16T04:44:09.421765+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:09.421765+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5457,"log_dups_size":0,"ondisk_log_size":5457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44694150,"num_objects":5456,"num_object_clones":0,"num_object_copies":21824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5456,"num_write_kb":43647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3328}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5526","reported_seq":5596,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.356567+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:09.356567+0000","last_peered":"2024-09-16T04:44:09.356567+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:09.356567+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5526,"log_dups_size":0,"ondisk_log_size":5526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45259398,"num_objects":5525,"num_object_clones":0,"num_object_copies":22100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5525,"num_write_kb":44199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3300}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5645","reported_seq":5695,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.499417+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:09.499417+0000","last_peered":"2024-09-16T04:44:09.499417+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:09.499417+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5645,"log_dups_size":0,"ondisk_log_size":5645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46208268,"num_objects":5641,"num_object_clones":0,"num_object_copies":22564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5643,"num_write_kb":45140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3407}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5487","reported_seq":5562,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.442343+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:09.442343+0000","last_peered":"2024-09-16T04:44:09.442343+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:09.442343+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5487,"log_dups_size":0,"ondisk_log_size":5487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44922124,"num_objects":5484,"num_object_clones":0,"num_object_copies":21936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5485,"num_write_kb":43877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3352}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5436","reported_seq":5495,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.877097+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:09.877097+0000","last_peered":"2024-09-16T04:44:09.877097+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:09.877097+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5436,"log_dups_size":0,"ondisk_log_size":5436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44487948,"num_objects":5431,"num_object_clones":0,"num_object_copies":21724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5434,"num_write_kb":43460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":729015976,"num_objects":88996,"num_object_clones":0,"num_object_copies":355984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89023,"num_write_kb":712066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2260,"num_bytes_recovered":18276352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1317851136,"data_stored":1317851136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89050,"ondisk_log_size":89050,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693888,"kb_used_data":355372,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93677952,"statfs":{"total":96636764160,"available":95926222848,"internally_reserved":0,"allocated":363900928,"data_stored":363730118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346615503},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674892,"kb_used_data":354936,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93696948,"statfs":{"total":96636764160,"available":95945674752,"internally_reserved":0,"allocated":363454464,"data_stored":363283654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051617,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416688,"kb_used_data":218400,"kb_used_omap":4,"kb_used_meta":198267,"kb_avail":93955152,"statfs":{"total":96636764160,"available":96210075648,"internally_reserved":0,"allocated":223641600,"data_stored":223470790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":203026127},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702712,"kb_used_data":355944,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93669128,"statfs":{"total":96636764160,"available":95917187072,"internally_reserved":0,"allocated":364486656,"data_stored":364315846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":355069646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,0,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":364675072,"data_stored":364675072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":224591872,"data_stored":224591872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":364265472,"data_stored":364265472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":364318720,"data_stored":364318720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:11.314 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 197 20 89396 89376 3.54312 1.97656 0.0427465 0.0440845 2024-09-16T04:44:12.314 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 198 20 89651 89631 3.53529 1.99219 0.073024 0.0441826 2024-09-16T04:44:13.314 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 199 20 89933 89913 3.5286 2.20312 0.0586675 0.0442651 2024-09-16T04:44:13.957 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:14.314 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:2024-09-16T04:44:14.314113+0000 min lat: 0.00269411 max lat: 1.73107 avg lat: 0.044352 2024-09-16T04:44:14.314 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T04:44:14.314 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 200 20 90213 90193 3.52189 2.1875 0.0577391 0.044352 2024-09-16T04:44:14.531 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:14.531 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:14.558 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-09-16T04:44:12.617685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":729179816,"num_objects":89016,"num_object_clones":0,"num_object_copies":356064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89043,"num_write_kb":712226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2267,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89070,"ondisk_log_size":89070,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2498268,"kb_used_data":1290580,"kb_used_omap":14,"kb_used_meta":1207601,"kb_avail":374989092,"statfs":{"total":386547056640,"available":383988830208,"internally_reserved":0,"allocated":1321553920,"data_stored":1320870680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1236583985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,8,8,9,12,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15548416,"num_objects":1898,"num_object_clones":0,"num_object_copies":7592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1898,"num_write_kb":15184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":1220608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002168"},"pg_stats":[{"pgid":"1.f","version":"31'5584","reported_seq":5624,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.525729+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:09.525729+0000","last_peered":"2024-09-16T04:44:09.525729+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:09.525729+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45744128,"num_objects":5584,"num_object_clones":0,"num_object_copies":22336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5584,"num_write_kb":44672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5639","reported_seq":5701,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.881664+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:09.881664+0000","last_peered":"2024-09-16T04:44:09.881664+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:09.881664+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46159116,"num_objects":5635,"num_object_clones":0,"num_object_copies":22540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5637,"num_write_kb":45092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5623","reported_seq":5694,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.420174+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:09.420174+0000","last_peered":"2024-09-16T04:44:09.420174+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:09.420174+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5623,"log_dups_size":0,"ondisk_log_size":5623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46037638,"num_objects":5620,"num_object_clones":0,"num_object_copies":22480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5622,"num_write_kb":44966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3393}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5584","reported_seq":5646,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.841291+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:09.841291+0000","last_peered":"2024-09-16T04:44:09.841291+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:09.841291+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708556,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5582,"num_write_kb":44645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3351}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5642","reported_seq":5694,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.362093+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:09.362093+0000","last_peered":"2024-09-16T04:44:09.362093+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:09.362093+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5642,"log_dups_size":0,"ondisk_log_size":5642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46191884,"num_objects":5639,"num_object_clones":0,"num_object_copies":22556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5640,"num_write_kb":45117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5441","reported_seq":5479,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.527818+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:09.527818+0000","last_peered":"2024-09-16T04:44:09.527818+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:09.527818+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44572672,"num_objects":5441,"num_object_clones":0,"num_object_copies":21764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5441,"num_write_kb":43528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5597","reported_seq":5658,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.844599+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:09.844599+0000","last_peered":"2024-09-16T04:44:09.844599+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:09.844599+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5597,"log_dups_size":0,"ondisk_log_size":5597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45798668,"num_objects":5591,"num_object_clones":0,"num_object_copies":22364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5595,"num_write_kb":44740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3365}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5660","reported_seq":10206,"reported_epoch":32,"state":"peering","last_fresh":"2024-09-16T04:44:10.871081+0000","last_change":"2024-09-16T04:44:11.786574+0000","last_active":"2024-09-16T04:44:11.786574+0000","last_peered":"2024-09-16T04:44:10.871081+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:44:10.871081+0000","last_undegraded":"2024-09-16T04:44:10.871081+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5660,"log_dups_size":0,"ondisk_log_size":5660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46322956,"num_objects":5655,"num_object_clones":0,"num_object_copies":22620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5658,"num_write_kb":45252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"31'5616","reported_seq":5666,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.448563+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:44:09.448563+0000","last_peered":"2024-09-16T04:44:09.448563+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:09.448563+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5616,"log_dups_size":0,"ondisk_log_size":5616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45935128,"num_objects":5608,"num_object_clones":0,"num_object_copies":22432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5613,"num_write_kb":44881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5610","reported_seq":5684,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.341518+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:09.341518+0000","last_peered":"2024-09-16T04:44:09.341518+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:09.341518+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5610,"log_dups_size":0,"ondisk_log_size":5610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45937932,"num_objects":5608,"num_object_clones":0,"num_object_copies":22432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5608,"num_write_kb":44862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3408}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5523","reported_seq":5577,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.493207+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:09.493207+0000","last_peered":"2024-09-16T04:44:09.493207+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:09.493207+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5523,"log_dups_size":0,"ondisk_log_size":5523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45199250,"num_objects":5518,"num_object_clones":0,"num_object_copies":22072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5520,"num_write_kb":44148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5457","reported_seq":5493,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.421765+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:09.421765+0000","last_peered":"2024-09-16T04:44:09.421765+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:09.421765+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5457,"log_dups_size":0,"ondisk_log_size":5457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44694150,"num_objects":5456,"num_object_clones":0,"num_object_copies":21824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5456,"num_write_kb":43647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3328}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5526","reported_seq":5596,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.356567+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:09.356567+0000","last_peered":"2024-09-16T04:44:09.356567+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:09.356567+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5526,"log_dups_size":0,"ondisk_log_size":5526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45259398,"num_objects":5525,"num_object_clones":0,"num_object_copies":22100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5525,"num_write_kb":44199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3300}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5645","reported_seq":5695,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.499417+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:09.499417+0000","last_peered":"2024-09-16T04:44:09.499417+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:09.499417+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5645,"log_dups_size":0,"ondisk_log_size":5645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46208268,"num_objects":5641,"num_object_clones":0,"num_object_copies":22564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5643,"num_write_kb":45140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3407}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5487","reported_seq":5562,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.442343+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:09.442343+0000","last_peered":"2024-09-16T04:44:09.442343+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:09.442343+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5487,"log_dups_size":0,"ondisk_log_size":5487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44922124,"num_objects":5484,"num_object_clones":0,"num_object_copies":21936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5485,"num_write_kb":43877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3352}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5436","reported_seq":5495,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.877097+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:09.877097+0000","last_peered":"2024-09-16T04:44:09.877097+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:09.877097+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5436,"log_dups_size":0,"ondisk_log_size":5436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44487948,"num_objects":5431,"num_object_clones":0,"num_object_copies":21724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5434,"num_write_kb":43460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":729179816,"num_objects":89016,"num_object_clones":0,"num_object_copies":356064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89043,"num_write_kb":712226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2267,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1323687936,"data_stored":1323687936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89070,"ondisk_log_size":89070,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693888,"kb_used_data":355372,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93677952,"statfs":{"total":96636764160,"available":95926222848,"internally_reserved":0,"allocated":363900928,"data_stored":363730118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346615503},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674892,"kb_used_data":354936,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93696948,"statfs":{"total":96636764160,"available":95945674752,"internally_reserved":0,"allocated":363454464,"data_stored":363283654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051618,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426776,"kb_used_data":224328,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93945064,"statfs":{"total":96636764160,"available":96199745536,"internally_reserved":0,"allocated":229711872,"data_stored":229541062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207285967},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702712,"kb_used_data":355944,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93669128,"statfs":{"total":96636764160,"available":95917187072,"internally_reserved":0,"allocated":364486656,"data_stored":364315846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":355069646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,0,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":364675072,"data_stored":364675072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":230428672,"data_stored":230428672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":364265472,"data_stored":364265472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":364318720,"data_stored":364318720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:14.559 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:15.126 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:15.127 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:15.159 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-16T04:44:14.618324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":731752104,"num_objects":89330,"num_object_clones":0,"num_object_copies":357320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89357,"num_write_kb":714738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2267,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89384,"ondisk_log_size":89384,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2507808,"kb_used_data":1295960,"kb_used_omap":14,"kb_used_meta":1211761,"kb_avail":374979552,"statfs":{"total":386547056640,"available":383979061248,"internally_reserved":0,"allocated":1327063040,"data_stored":1326370641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1240843825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,5,6,8,10,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14581760,"num_objects":1780,"num_object_clones":0,"num_object_copies":7120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1780,"num_write_kb":14240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":614400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002258"},"pg_stats":[{"pgid":"1.f","version":"31'5584","reported_seq":5624,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.525729+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:09.525729+0000","last_peered":"2024-09-16T04:44:09.525729+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:09.525729+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45744128,"num_objects":5584,"num_object_clones":0,"num_object_copies":22336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5584,"num_write_kb":44672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5639","reported_seq":5701,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.881664+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:09.881664+0000","last_peered":"2024-09-16T04:44:09.881664+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:09.881664+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46159116,"num_objects":5635,"num_object_clones":0,"num_object_copies":22540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5637,"num_write_kb":45092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5703","reported_seq":5778,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.514184+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:14.514184+0000","last_peered":"2024-09-16T04:44:14.514184+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:14.514184+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5703,"log_dups_size":0,"ondisk_log_size":5703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46692998,"num_objects":5700,"num_object_clones":0,"num_object_copies":22800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5702,"num_write_kb":45606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3473}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5584","reported_seq":5646,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.841291+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:09.841291+0000","last_peered":"2024-09-16T04:44:09.841291+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:09.841291+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708556,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5582,"num_write_kb":44645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3351}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5642","reported_seq":5694,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.362093+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:09.362093+0000","last_peered":"2024-09-16T04:44:09.362093+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:09.362093+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5642,"log_dups_size":0,"ondisk_log_size":5642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46191884,"num_objects":5639,"num_object_clones":0,"num_object_copies":22556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5640,"num_write_kb":45117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5441","reported_seq":5479,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.527818+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:09.527818+0000","last_peered":"2024-09-16T04:44:09.527818+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:09.527818+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44572672,"num_objects":5441,"num_object_clones":0,"num_object_copies":21764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5441,"num_write_kb":43528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5597","reported_seq":5658,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.844599+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:09.844599+0000","last_peered":"2024-09-16T04:44:09.844599+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:09.844599+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5597,"log_dups_size":0,"ondisk_log_size":5597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45798668,"num_objects":5591,"num_object_clones":0,"num_object_copies":22364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5595,"num_write_kb":44740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3365}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5660","reported_seq":10206,"reported_epoch":32,"state":"peering","last_fresh":"2024-09-16T04:44:10.871081+0000","last_change":"2024-09-16T04:44:11.786574+0000","last_active":"2024-09-16T04:44:11.786574+0000","last_peered":"2024-09-16T04:44:10.871081+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:44:10.871081+0000","last_undegraded":"2024-09-16T04:44:10.871081+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5660,"log_dups_size":0,"ondisk_log_size":5660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46322956,"num_objects":5655,"num_object_clones":0,"num_object_copies":22620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5658,"num_write_kb":45252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"31'5616","reported_seq":5666,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.448563+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:44:09.448563+0000","last_peered":"2024-09-16T04:44:09.448563+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:09.448563+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5616,"log_dups_size":0,"ondisk_log_size":5616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45935128,"num_objects":5608,"num_object_clones":0,"num_object_copies":22432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5613,"num_write_kb":44881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5694","reported_seq":5772,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.441389+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:14.441389+0000","last_peered":"2024-09-16T04:44:14.441389+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:14.441389+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5694,"log_dups_size":0,"ondisk_log_size":5694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46626060,"num_objects":5692,"num_object_clones":0,"num_object_copies":22768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5692,"num_write_kb":45534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3492}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5523","reported_seq":5577,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.493207+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:09.493207+0000","last_peered":"2024-09-16T04:44:09.493207+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:09.493207+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5523,"log_dups_size":0,"ondisk_log_size":5523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45199250,"num_objects":5518,"num_object_clones":0,"num_object_copies":22072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5520,"num_write_kb":44148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5457","reported_seq":5493,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.421765+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:09.421765+0000","last_peered":"2024-09-16T04:44:09.421765+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:09.421765+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5457,"log_dups_size":0,"ondisk_log_size":5457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44694150,"num_objects":5456,"num_object_clones":0,"num_object_copies":21824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5456,"num_write_kb":43647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3328}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5601","reported_seq":5675,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.375902+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:14.375902+0000","last_peered":"2024-09-16T04:44:14.375902+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:14.375902+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45873798,"num_objects":5600,"num_object_clones":0,"num_object_copies":22400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5600,"num_write_kb":44799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3375}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5645","reported_seq":5695,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.499417+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:09.499417+0000","last_peered":"2024-09-16T04:44:09.499417+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:09.499417+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5645,"log_dups_size":0,"ondisk_log_size":5645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46208268,"num_objects":5641,"num_object_clones":0,"num_object_copies":22564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5643,"num_write_kb":45140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3407}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5562","reported_seq":5641,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.549865+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:14.549865+0000","last_peered":"2024-09-16T04:44:14.549865+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:14.549865+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45536524,"num_objects":5559,"num_object_clones":0,"num_object_copies":22236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5560,"num_write_kb":44477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3427}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5436","reported_seq":5495,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.877097+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:09.877097+0000","last_peered":"2024-09-16T04:44:09.877097+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:09.877097+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5436,"log_dups_size":0,"ondisk_log_size":5436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44487948,"num_objects":5431,"num_object_clones":0,"num_object_copies":21724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5434,"num_write_kb":43460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":731752104,"num_objects":89330,"num_object_clones":0,"num_object_copies":357320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89357,"num_write_kb":714738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2267,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1329192960,"data_stored":1329192960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89384,"ondisk_log_size":89384,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693888,"kb_used_data":355372,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93677952,"statfs":{"total":96636764160,"available":95926222848,"internally_reserved":0,"allocated":363900928,"data_stored":363730118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346615503},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684432,"kb_used_data":360316,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93687408,"statfs":{"total":96636764160,"available":95935905792,"internally_reserved":0,"allocated":368963584,"data_stored":368783615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051618,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426776,"kb_used_data":224328,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93945064,"statfs":{"total":96636764160,"available":96199745536,"internally_reserved":0,"allocated":229711872,"data_stored":229541062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207285967},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702712,"kb_used_data":355944,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93669128,"statfs":{"total":96636764160,"available":95917187072,"internally_reserved":0,"allocated":364486656,"data_stored":364315846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":355069646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,0,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":364675072,"data_stored":364675072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":230428672,"data_stored":230428672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":369770496,"data_stored":369770496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":364318720,"data_stored":364318720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:15.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:44:15.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5584", 'reported_seq': 5624, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:09.525729+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:44:09.525729+0000', 'last_peered': '2024-09-16T04:44:09.525729+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:44:09.525729+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5584, 'log_dups_size': 0, 'ondisk_log_size': 5584, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45744128, 'num_objects': 5584, 'num_object_clones': 0, 'num_object_copies': 22336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5584, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5584, 'num_write_kb': 44672, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3365}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:15.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:44:15.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5639", 'reported_seq': 5701, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:09.881664+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:44:09.881664+0000', 'last_peered': '2024-09-16T04:44:09.881664+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:44:09.881664+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5639, 'log_dups_size': 0, 'ondisk_log_size': 5639, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46159116, 'num_objects': 5635, 'num_object_clones': 0, 'num_object_copies': 22540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5635, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5637, 'num_write_kb': 45092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3424}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:15.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:44:15.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5703", 'reported_seq': 5778, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:14.514184+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:44:14.514184+0000', 'last_peered': '2024-09-16T04:44:14.514184+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:44:14.514184+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5703, 'log_dups_size': 0, 'ondisk_log_size': 5703, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46692998, 'num_objects': 5700, 'num_object_clones': 0, 'num_object_copies': 22800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5700, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5702, 'num_write_kb': 45606, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3473}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:44:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5584", 'reported_seq': 5646, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:09.841291+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:44:09.841291+0000', 'last_peered': '2024-09-16T04:44:09.841291+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:44:09.841291+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5584, 'log_dups_size': 0, 'ondisk_log_size': 5584, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45708556, 'num_objects': 5580, 'num_object_clones': 0, 'num_object_copies': 22320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5580, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5582, 'num_write_kb': 44645, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3351}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:44:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5642", 'reported_seq': 5694, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:09.362093+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:44:09.362093+0000', 'last_peered': '2024-09-16T04:44:09.362093+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:44:09.362093+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5642, 'log_dups_size': 0, 'ondisk_log_size': 5642, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46191884, 'num_objects': 5639, 'num_object_clones': 0, 'num_object_copies': 22556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5639, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5640, 'num_write_kb': 45117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3356}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:44:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5441", 'reported_seq': 5479, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:09.527818+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:44:09.527818+0000', 'last_peered': '2024-09-16T04:44:09.527818+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:44:09.527818+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5441, 'log_dups_size': 0, 'ondisk_log_size': 5441, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44572672, 'num_objects': 5441, 'num_object_clones': 0, 'num_object_copies': 21764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5441, 'num_write_kb': 43528, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3233}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:44:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5597", 'reported_seq': 5658, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:09.844599+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:44:09.844599+0000', 'last_peered': '2024-09-16T04:44:09.844599+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:44:09.844599+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5597, 'log_dups_size': 0, 'ondisk_log_size': 5597, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45798668, 'num_objects': 5591, 'num_object_clones': 0, 'num_object_copies': 22364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5595, 'num_write_kb': 44740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3365}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-16T04:44:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5660", 'reported_seq': 10206, 'reported_epoch': 32, 'state': 'peering', 'last_fresh': '2024-09-16T04:44:10.871081+0000', 'last_change': '2024-09-16T04:44:11.786574+0000', 'last_active': '2024-09-16T04:44:11.786574+0000', 'last_peered': '2024-09-16T04:44:10.871081+0000', 'last_clean': '2024-09-16T04:40:23.173443+0000', 'last_became_active': '2024-09-16T04:41:27.717124+0000', 'last_became_peered': '2024-09-16T04:41:27.717124+0000', 'last_unstale': '2024-09-16T04:44:10.871081+0000', 'last_undegraded': '2024-09-16T04:44:10.871081+0000', 'last_fullsized': '2024-09-16T04:41:27.484801+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5660, 'log_dups_size': 0, 'ondisk_log_size': 5660, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46322956, 'num_objects': 5655, 'num_object_clones': 0, 'num_object_copies': 22620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5655, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5658, 'num_write_kb': 45252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2242, 'num_bytes_recovered': 18333696, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [1, 0, 2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T04:44:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:44:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5616", 'reported_seq': 5666, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:09.448563+0000', 'last_change': '2024-09-16T04:41:27.770135+0000', 'last_active': '2024-09-16T04:44:09.448563+0000', 'last_peered': '2024-09-16T04:44:09.448563+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:44:09.448563+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5616, 'log_dups_size': 0, 'ondisk_log_size': 5616, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45935128, 'num_objects': 5608, 'num_object_clones': 0, 'num_object_copies': 22432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2343, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5613, 'num_write_kb': 44881, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3268}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:44:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5694", 'reported_seq': 5772, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:14.441389+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:44:14.441389+0000', 'last_peered': '2024-09-16T04:44:14.441389+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:44:14.441389+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5694, 'log_dups_size': 0, 'ondisk_log_size': 5694, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46626060, 'num_objects': 5692, 'num_object_clones': 0, 'num_object_copies': 22768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5692, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5692, 'num_write_kb': 45534, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3492}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:44:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5523", 'reported_seq': 5577, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:09.493207+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:44:09.493207+0000', 'last_peered': '2024-09-16T04:44:09.493207+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:44:09.493207+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5523, 'log_dups_size': 0, 'ondisk_log_size': 5523, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45199250, 'num_objects': 5518, 'num_object_clones': 0, 'num_object_copies': 22072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5520, 'num_write_kb': 44148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3329}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:44:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5457", 'reported_seq': 5493, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:09.421765+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:44:09.421765+0000', 'last_peered': '2024-09-16T04:44:09.421765+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:44:09.421765+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5457, 'log_dups_size': 0, 'ondisk_log_size': 5457, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44694150, 'num_objects': 5456, 'num_object_clones': 0, 'num_object_copies': 21824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5456, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5456, 'num_write_kb': 43647, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3328}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:44:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5601", 'reported_seq': 5675, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:14.375902+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:44:14.375902+0000', 'last_peered': '2024-09-16T04:44:14.375902+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:44:14.375902+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5601, 'log_dups_size': 0, 'ondisk_log_size': 5601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45873798, 'num_objects': 5600, 'num_object_clones': 0, 'num_object_copies': 22400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5600, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5600, 'num_write_kb': 44799, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3375}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:15.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:44:15.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5645", 'reported_seq': 5695, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:09.499417+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:44:09.499417+0000', 'last_peered': '2024-09-16T04:44:09.499417+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:44:09.499417+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5645, 'log_dups_size': 0, 'ondisk_log_size': 5645, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46208268, 'num_objects': 5641, 'num_object_clones': 0, 'num_object_copies': 22564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5641, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5643, 'num_write_kb': 45140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3407}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:15.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:44:15.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'5562", 'reported_seq': 5641, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:14.549865+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:44:14.549865+0000', 'last_peered': '2024-09-16T04:44:14.549865+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:44:14.549865+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5562, 'log_dups_size': 0, 'ondisk_log_size': 5562, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45536524, 'num_objects': 5559, 'num_object_clones': 0, 'num_object_copies': 22236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5559, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5560, 'num_write_kb': 44477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3427}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:15.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:44:15.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5436", 'reported_seq': 5495, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:09.877097+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:44:09.877097+0000', 'last_peered': '2024-09-16T04:44:09.877097+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:44:09.877097+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5436, 'log_dups_size': 0, 'ondisk_log_size': 5436, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44487948, 'num_objects': 5431, 'num_object_clones': 0, 'num_object_copies': 21724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5434, 'num_write_kb': 43460, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3192}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:15.164 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:44:15.314 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 201 20 90463 90443 3.51409 1.95312 0.0858746 0.0444471 2024-09-16T04:44:15.908 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:15.909 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33315/357320 objects degraded (9.324%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":268,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"peering","count":1}],"num_pgs":16,"num_pools":1,"num_objects":89330,"data_bytes":731752104,"bytes_used":2567995392,"bytes_avail":383979061248,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":33315,"degraded_total":357320,"degraded_ratio":0.093235755065487508,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":76778,"recovering_keys_per_sec":0,"num_objects_recovered":75,"num_bytes_recovered":614400,"num_keys_recovered":0,"write_bytes_sec":1822205,"read_op_per_sec":0,"write_op_per_sec":222},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:44:15.935 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:44:15.935 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:16.315 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 202 20 90727 90707 3.5069 2.0625 0.0592532 0.044543 2024-09-16T04:44:16.498 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:16.499 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:16.520 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-16T04:44:14.618324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":731752104,"num_objects":89330,"num_object_clones":0,"num_object_copies":357320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89357,"num_write_kb":714738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2267,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89384,"ondisk_log_size":89384,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2507808,"kb_used_data":1295960,"kb_used_omap":14,"kb_used_meta":1211761,"kb_avail":374979552,"statfs":{"total":386547056640,"available":383979061248,"internally_reserved":0,"allocated":1327063040,"data_stored":1326370641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1240843825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,5,6,8,10,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14581760,"num_objects":1780,"num_object_clones":0,"num_object_copies":7120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1780,"num_write_kb":14240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":614400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002258"},"pg_stats":[{"pgid":"1.f","version":"31'5584","reported_seq":5624,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.525729+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:09.525729+0000","last_peered":"2024-09-16T04:44:09.525729+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:09.525729+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45744128,"num_objects":5584,"num_object_clones":0,"num_object_copies":22336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5584,"num_write_kb":44672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5639","reported_seq":5701,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.881664+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:09.881664+0000","last_peered":"2024-09-16T04:44:09.881664+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:09.881664+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46159116,"num_objects":5635,"num_object_clones":0,"num_object_copies":22540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5637,"num_write_kb":45092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5703","reported_seq":5778,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.514184+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:14.514184+0000","last_peered":"2024-09-16T04:44:14.514184+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:14.514184+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5703,"log_dups_size":0,"ondisk_log_size":5703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46692998,"num_objects":5700,"num_object_clones":0,"num_object_copies":22800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5702,"num_write_kb":45606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3473}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5584","reported_seq":5646,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.841291+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:09.841291+0000","last_peered":"2024-09-16T04:44:09.841291+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:09.841291+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708556,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5582,"num_write_kb":44645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3351}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5642","reported_seq":5694,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.362093+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:09.362093+0000","last_peered":"2024-09-16T04:44:09.362093+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:09.362093+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5642,"log_dups_size":0,"ondisk_log_size":5642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46191884,"num_objects":5639,"num_object_clones":0,"num_object_copies":22556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5640,"num_write_kb":45117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5441","reported_seq":5479,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.527818+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:09.527818+0000","last_peered":"2024-09-16T04:44:09.527818+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:09.527818+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44572672,"num_objects":5441,"num_object_clones":0,"num_object_copies":21764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5441,"num_write_kb":43528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5597","reported_seq":5658,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.844599+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:09.844599+0000","last_peered":"2024-09-16T04:44:09.844599+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:09.844599+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5597,"log_dups_size":0,"ondisk_log_size":5597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45798668,"num_objects":5591,"num_object_clones":0,"num_object_copies":22364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5595,"num_write_kb":44740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3365}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5660","reported_seq":10206,"reported_epoch":32,"state":"peering","last_fresh":"2024-09-16T04:44:10.871081+0000","last_change":"2024-09-16T04:44:11.786574+0000","last_active":"2024-09-16T04:44:11.786574+0000","last_peered":"2024-09-16T04:44:10.871081+0000","last_clean":"2024-09-16T04:40:23.173443+0000","last_became_active":"2024-09-16T04:41:27.717124+0000","last_became_peered":"2024-09-16T04:41:27.717124+0000","last_unstale":"2024-09-16T04:44:10.871081+0000","last_undegraded":"2024-09-16T04:44:10.871081+0000","last_fullsized":"2024-09-16T04:41:27.484801+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5660,"log_dups_size":0,"ondisk_log_size":5660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46322956,"num_objects":5655,"num_object_clones":0,"num_object_copies":22620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5658,"num_write_kb":45252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"31'5616","reported_seq":5666,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.448563+0000","last_change":"2024-09-16T04:41:27.770135+0000","last_active":"2024-09-16T04:44:09.448563+0000","last_peered":"2024-09-16T04:44:09.448563+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:09.448563+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5616,"log_dups_size":0,"ondisk_log_size":5616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45935128,"num_objects":5608,"num_object_clones":0,"num_object_copies":22432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5613,"num_write_kb":44881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5694","reported_seq":5772,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.441389+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:14.441389+0000","last_peered":"2024-09-16T04:44:14.441389+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:14.441389+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5694,"log_dups_size":0,"ondisk_log_size":5694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46626060,"num_objects":5692,"num_object_clones":0,"num_object_copies":22768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5692,"num_write_kb":45534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3492}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5523","reported_seq":5577,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.493207+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:09.493207+0000","last_peered":"2024-09-16T04:44:09.493207+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:09.493207+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5523,"log_dups_size":0,"ondisk_log_size":5523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45199250,"num_objects":5518,"num_object_clones":0,"num_object_copies":22072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5520,"num_write_kb":44148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3329}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5457","reported_seq":5493,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.421765+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:09.421765+0000","last_peered":"2024-09-16T04:44:09.421765+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:09.421765+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5457,"log_dups_size":0,"ondisk_log_size":5457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44694150,"num_objects":5456,"num_object_clones":0,"num_object_copies":21824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5456,"num_write_kb":43647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3328}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5601","reported_seq":5675,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.375902+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:14.375902+0000","last_peered":"2024-09-16T04:44:14.375902+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:14.375902+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45873798,"num_objects":5600,"num_object_clones":0,"num_object_copies":22400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5600,"num_write_kb":44799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3375}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5645","reported_seq":5695,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.499417+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:09.499417+0000","last_peered":"2024-09-16T04:44:09.499417+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:09.499417+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5645,"log_dups_size":0,"ondisk_log_size":5645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46208268,"num_objects":5641,"num_object_clones":0,"num_object_copies":22564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5643,"num_write_kb":45140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3407}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5562","reported_seq":5641,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.549865+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:14.549865+0000","last_peered":"2024-09-16T04:44:14.549865+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:14.549865+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45536524,"num_objects":5559,"num_object_clones":0,"num_object_copies":22236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5560,"num_write_kb":44477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3427}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5436","reported_seq":5495,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:09.877097+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:09.877097+0000","last_peered":"2024-09-16T04:44:09.877097+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:09.877097+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5436,"log_dups_size":0,"ondisk_log_size":5436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44487948,"num_objects":5431,"num_object_clones":0,"num_object_copies":21724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5434,"num_write_kb":43460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":731752104,"num_objects":89330,"num_object_clones":0,"num_object_copies":357320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89357,"num_write_kb":714738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2267,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1329192960,"data_stored":1329192960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89384,"ondisk_log_size":89384,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693888,"kb_used_data":355372,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93677952,"statfs":{"total":96636764160,"available":95926222848,"internally_reserved":0,"allocated":363900928,"data_stored":363730118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346615503},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684432,"kb_used_data":360316,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93687408,"statfs":{"total":96636764160,"available":95935905792,"internally_reserved":0,"allocated":368963584,"data_stored":368783615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051618,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426776,"kb_used_data":224328,"kb_used_omap":4,"kb_used_meta":202427,"kb_avail":93945064,"statfs":{"total":96636764160,"available":96199745536,"internally_reserved":0,"allocated":229711872,"data_stored":229541062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207285967},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702712,"kb_used_data":355944,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93669128,"statfs":{"total":96636764160,"available":95917187072,"internally_reserved":0,"allocated":364486656,"data_stored":364315846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":355069646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,0,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":364675072,"data_stored":364675072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":230428672,"data_stored":230428672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":369770496,"data_stored":369770496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":364318720,"data_stored":364318720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:17.315 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 203 20 91006 90986 3.50037 2.17969 0.0546017 0.0446249 2024-09-16T04:44:18.318 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 204 20 91300 91280 3.49447 2.29688 0.0847363 0.0446997 2024-09-16T04:44:19.315 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 205 20 91596 91576 3.4887 2.3125 0.0942966 0.0447762 2024-09-16T04:44:19.522 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:20.114 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:20.114 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:20.147 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-09-16T04:44:18.619556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":740378280,"num_objects":90383,"num_object_clones":0,"num_object_copies":361532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90410,"num_write_kb":723162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2305,"num_bytes_recovered":18644992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90437,"ondisk_log_size":90437,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2522456,"kb_used_data":1313356,"kb_used_omap":14,"kb_used_meta":1209009,"kb_avail":374964904,"statfs":{"total":386547056640,"available":383964061696,"internally_reserved":0,"allocated":1344876544,"data_stored":1344156668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1238025777},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,10,14,17,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11362304,"num_objects":1387,"num_object_clones":0,"num_object_copies":5548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1387,"num_write_kb":11096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":368640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002425"},"pg_stats":[{"pgid":"1.f","version":"33'5676","reported_seq":5719,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.667273+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:14.667273+0000","last_peered":"2024-09-16T04:44:14.667273+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:14.667273+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5676,"log_dups_size":0,"ondisk_log_size":5676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46497792,"num_objects":5676,"num_object_clones":0,"num_object_copies":22704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5676,"num_write_kb":45408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3457}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5711","reported_seq":5777,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.920015+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:14.920015+0000","last_peered":"2024-09-16T04:44:14.920015+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:14.920015+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5711,"log_dups_size":0,"ondisk_log_size":5711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46748940,"num_objects":5707,"num_object_clones":0,"num_object_copies":22828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5709,"num_write_kb":45668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5703","reported_seq":5778,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.514184+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:14.514184+0000","last_peered":"2024-09-16T04:44:14.514184+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:14.514184+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5703,"log_dups_size":0,"ondisk_log_size":5703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46692998,"num_objects":5700,"num_object_clones":0,"num_object_copies":22800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5702,"num_write_kb":45606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3473}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5683","reported_seq":5749,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.847212+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:14.847212+0000","last_peered":"2024-09-16T04:44:14.847212+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:14.847212+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5683,"log_dups_size":0,"ondisk_log_size":5683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46519564,"num_objects":5679,"num_object_clones":0,"num_object_copies":22716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5681,"num_write_kb":45437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3450}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5730","reported_seq":5784,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.572226+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:14.572226+0000","last_peered":"2024-09-16T04:44:14.572226+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:14.572226+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5730,"log_dups_size":0,"ondisk_log_size":5730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46912780,"num_objects":5727,"num_object_clones":0,"num_object_copies":22908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5728,"num_write_kb":45821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3444}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5524","reported_seq":5568,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.650369+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:14.650369+0000","last_peered":"2024-09-16T04:44:14.650369+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:14.650369+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5524,"log_dups_size":0,"ondisk_log_size":5524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45252608,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5524,"num_write_kb":44192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5685","reported_seq":5749,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.887676+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:14.887676+0000","last_peered":"2024-09-16T04:44:14.887676+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:14.887676+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5685,"log_dups_size":0,"ondisk_log_size":5685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46519564,"num_objects":5679,"num_object_clones":0,"num_object_copies":22716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5683,"num_write_kb":45444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3453}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5769","reported_seq":10325,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:16.752845+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:16.752845+0000","last_peered":"2024-09-16T04:44:16.752845+0000","last_clean":"2024-09-16T04:44:16.752845+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:16.752845+0000","last_undegraded":"2024-09-16T04:44:16.752845+0000","last_fullsized":"2024-09-16T04:44:16.752845+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5769,"log_dups_size":0,"ondisk_log_size":5769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:17.859161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47215884,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5767,"num_write_kb":46124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'5712","reported_seq":5846,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.636795+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:14.636795+0000","last_peered":"2024-09-16T04:44:14.636795+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:14.636795+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5712,"log_dups_size":0,"ondisk_log_size":5712,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46721560,"num_objects":5704,"num_object_clones":0,"num_object_copies":22816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5709,"num_write_kb":45649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":311296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3402}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5694","reported_seq":5772,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.441389+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:14.441389+0000","last_peered":"2024-09-16T04:44:14.441389+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:14.441389+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5694,"log_dups_size":0,"ondisk_log_size":5694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46626060,"num_objects":5692,"num_object_clones":0,"num_object_copies":22768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5692,"num_write_kb":45534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3492}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5607","reported_seq":5663,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.541750+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:14.541750+0000","last_peered":"2024-09-16T04:44:14.541750+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:14.541750+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45887378,"num_objects":5602,"num_object_clones":0,"num_object_copies":22408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5604,"num_write_kb":44820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3413}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5541","reported_seq":5581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.599237+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:14.599237+0000","last_peered":"2024-09-16T04:44:14.599237+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:14.599237+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5541,"log_dups_size":0,"ondisk_log_size":5541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45382278,"num_objects":5540,"num_object_clones":0,"num_object_copies":22160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5540,"num_write_kb":44319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3412}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5601","reported_seq":5675,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.375902+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:14.375902+0000","last_peered":"2024-09-16T04:44:14.375902+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:14.375902+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45873798,"num_objects":5600,"num_object_clones":0,"num_object_copies":22400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5600,"num_write_kb":44799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3375}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5719","reported_seq":5773,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.576728+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:14.576728+0000","last_peered":"2024-09-16T04:44:14.576728+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:14.576728+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5719,"log_dups_size":0,"ondisk_log_size":5719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46814476,"num_objects":5715,"num_object_clones":0,"num_object_copies":22860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5717,"num_write_kb":45732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3481}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5562","reported_seq":5641,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.549865+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:14.549865+0000","last_peered":"2024-09-16T04:44:14.549865+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:14.549865+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45536524,"num_objects":5559,"num_object_clones":0,"num_object_copies":22236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5560,"num_write_kb":44477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3427}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5520","reported_seq":5585,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:14.902756+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:14.902756+0000","last_peered":"2024-09-16T04:44:14.902756+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:14.902756+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45176076,"num_objects":5515,"num_object_clones":0,"num_object_copies":22060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5518,"num_write_kb":44132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3276}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":740378280,"num_objects":90383,"num_object_clones":0,"num_object_copies":361532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90410,"num_write_kb":723162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2305,"num_bytes_recovered":18644992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1346093056,"data_stored":1346093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90437,"ondisk_log_size":90437,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703412,"kb_used_data":360736,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93668428,"statfs":{"total":96636764160,"available":95916470272,"internally_reserved":0,"allocated":369393664,"data_stored":369213695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350875343},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,2,5,10,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684432,"kb_used_data":360316,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93687408,"statfs":{"total":96636764160,"available":95935905792,"internally_reserved":0,"allocated":368963584,"data_stored":368783615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439008,"kb_used_data":230992,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93932832,"statfs":{"total":96636764160,"available":96187219968,"internally_reserved":0,"allocated":236535808,"data_stored":236355839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":212987599},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,5,5,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695604,"kb_used_data":361312,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93676236,"statfs":{"total":96636764160,"available":95924465664,"internally_reserved":0,"allocated":369983488,"data_stored":369803519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":342290126},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":370139136,"data_stored":370139136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":236265472,"data_stored":236265472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":369770496,"data_stored":369770496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":369917952,"data_stored":369917952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:20.148 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:20.315 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 206 20 91862 91842 3.48186 2.07812 0.104922 0.044863 2024-09-16T04:44:20.729 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:20.729 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:20.760 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-16T04:44:20.620039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":751052456,"num_objects":91686,"num_object_clones":0,"num_object_copies":366744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91713,"num_write_kb":733586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2339,"num_bytes_recovered":18923520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91740,"ondisk_log_size":91740,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2551640,"kb_used_data":1330068,"kb_used_omap":14,"kb_used_meta":1221489,"kb_avail":374935720,"statfs":{"total":386547056640,"available":383934177280,"internally_reserved":0,"allocated":1361989632,"data_stored":1361269756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1250805297},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,11,11,15,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22036480,"num_objects":2690,"num_object_clones":0,"num_object_copies":10760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2690,"num_write_kb":21520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":647168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.639185"},"pg_stats":[{"pgid":"1.f","version":"33'5770","reported_seq":5814,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.724396+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:19.724396+0000","last_peered":"2024-09-16T04:44:19.724396+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:19.724396+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47267840,"num_objects":5770,"num_object_clones":0,"num_object_copies":23080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":46160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3551}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5790","reported_seq":5856,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.772309+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:19.772309+0000","last_peered":"2024-09-16T04:44:19.772309+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:19.772309+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5790,"log_dups_size":0,"ondisk_log_size":5790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47396108,"num_objects":5786,"num_object_clones":0,"num_object_copies":23144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5788,"num_write_kb":46300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3575}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5808","reported_seq":5883,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.531919+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:19.531919+0000","last_peered":"2024-09-16T04:44:19.531919+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:19.531919+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5808,"log_dups_size":0,"ondisk_log_size":5808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47553158,"num_objects":5805,"num_object_clones":0,"num_object_copies":23220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5807,"num_write_kb":46446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3578}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5770","reported_seq":5836,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.930983+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:19.930983+0000","last_peered":"2024-09-16T04:44:19.930983+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:19.930983+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47232268,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5768,"num_write_kb":46133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3537}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5815","reported_seq":5871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.544162+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:19.544162+0000","last_peered":"2024-09-16T04:44:19.544162+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:19.544162+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5815,"log_dups_size":0,"ondisk_log_size":5815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47609100,"num_objects":5812,"num_object_clones":0,"num_object_copies":23248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5813,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5605","reported_seq":5649,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.665803+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:19.665803+0000","last_peered":"2024-09-16T04:44:19.665803+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:19.665803+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5605,"log_dups_size":0,"ondisk_log_size":5605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45916160,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5772","reported_seq":5837,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.917124+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:19.917124+0000","last_peered":"2024-09-16T04:44:19.917124+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:19.917124+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47232268,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":46140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3540}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5769","reported_seq":10325,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:16.752845+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:16.752845+0000","last_peered":"2024-09-16T04:44:16.752845+0000","last_clean":"2024-09-16T04:44:16.752845+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:16.752845+0000","last_undegraded":"2024-09-16T04:44:16.752845+0000","last_fullsized":"2024-09-16T04:44:16.752845+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5769,"log_dups_size":0,"ondisk_log_size":5769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:17.859161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47215884,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5767,"num_write_kb":46124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'5806","reported_seq":6008,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.761693+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:19.761693+0000","last_peered":"2024-09-16T04:44:19.761693+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:19.761693+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5806,"log_dups_size":0,"ondisk_log_size":5806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47491608,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5803,"num_write_kb":46401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5771","reported_seq":5848,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.496916+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:19.496916+0000","last_peered":"2024-09-16T04:44:19.496916+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:19.496916+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5771,"log_dups_size":0,"ondisk_log_size":5771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47256844,"num_objects":5769,"num_object_clones":0,"num_object_copies":23076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5769,"num_write_kb":46150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3569}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5691","reported_seq":5749,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.597205+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:19.597205+0000","last_peered":"2024-09-16T04:44:19.597205+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:19.597205+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5691,"log_dups_size":0,"ondisk_log_size":5691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46575506,"num_objects":5686,"num_object_clones":0,"num_object_copies":22744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5688,"num_write_kb":45492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5637","reported_seq":5677,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.717569+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:19.717569+0000","last_peered":"2024-09-16T04:44:19.717569+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:19.717569+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5637,"log_dups_size":0,"ondisk_log_size":5637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46168710,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5636,"num_write_kb":45087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3508}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5679","reported_seq":5753,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.376965+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:19.376965+0000","last_peered":"2024-09-16T04:44:19.376965+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:19.376965+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46512774,"num_objects":5678,"num_object_clones":0,"num_object_copies":22712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5678,"num_write_kb":45423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3453}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5803","reported_seq":5857,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.742810+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:19.742810+0000","last_peered":"2024-09-16T04:44:19.742810+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:19.742810+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47502604,"num_objects":5799,"num_object_clones":0,"num_object_copies":23196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5801,"num_write_kb":46404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3565}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5647","reported_seq":5726,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.464559+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:19.464559+0000","last_peered":"2024-09-16T04:44:19.464559+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:19.464559+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46232844,"num_objects":5644,"num_object_clones":0,"num_object_copies":22576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5645,"num_write_kb":45157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3512}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5607","reported_seq":5672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.920077+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:19.920077+0000","last_peered":"2024-09-16T04:44:19.920077+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:19.920077+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45888780,"num_objects":5602,"num_object_clones":0,"num_object_copies":22408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":751052456,"num_objects":91686,"num_object_clones":0,"num_object_copies":366744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91713,"num_write_kb":733586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2339,"num_bytes_recovered":18923520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1363296256,"data_stored":1363296256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91740,"ondisk_log_size":91740,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713116,"kb_used_data":366284,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93658724,"statfs":{"total":96636764160,"available":95906533376,"internally_reserved":0,"allocated":375074816,"data_stored":374894847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355135183},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694160,"kb_used_data":365884,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93677680,"statfs":{"total":96636764160,"available":95925944320,"internally_reserved":0,"allocated":374665216,"data_stored":374485247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439008,"kb_used_data":230992,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93932832,"statfs":{"total":96636764160,"available":96187219968,"internally_reserved":0,"allocated":236535808,"data_stored":236355839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":212987599},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,5,5,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705356,"kb_used_data":366908,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93666484,"statfs":{"total":96636764160,"available":95914479616,"internally_reserved":0,"allocated":375713792,"data_stored":375533823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":346549966},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":375848960,"data_stored":375848960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":236265472,"data_stored":236265472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":375488512,"data_stored":375488512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":375693312,"data_stored":375693312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:20.761 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:44:20.761 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5770", 'reported_seq': 5814, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.724396+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:44:19.724396+0000', 'last_peered': '2024-09-16T04:44:19.724396+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:44:19.724396+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5770, 'log_dups_size': 0, 'ondisk_log_size': 5770, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47267840, 'num_objects': 5770, 'num_object_clones': 0, 'num_object_copies': 23080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5770, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5770, 'num_write_kb': 46160, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3551}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:20.761 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:44:20.761 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5790", 'reported_seq': 5856, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.772309+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:44:19.772309+0000', 'last_peered': '2024-09-16T04:44:19.772309+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:44:19.772309+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5790, 'log_dups_size': 0, 'ondisk_log_size': 5790, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47396108, 'num_objects': 5786, 'num_object_clones': 0, 'num_object_copies': 23144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5786, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5788, 'num_write_kb': 46300, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3575}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:20.761 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:44:20.761 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5808", 'reported_seq': 5883, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.531919+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:44:19.531919+0000', 'last_peered': '2024-09-16T04:44:19.531919+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:44:19.531919+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5808, 'log_dups_size': 0, 'ondisk_log_size': 5808, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47553158, 'num_objects': 5805, 'num_object_clones': 0, 'num_object_copies': 23220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5805, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5807, 'num_write_kb': 46446, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3578}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:20.762 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:44:20.762 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5770", 'reported_seq': 5836, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.930983+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:44:19.930983+0000', 'last_peered': '2024-09-16T04:44:19.930983+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:44:19.930983+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5770, 'log_dups_size': 0, 'ondisk_log_size': 5770, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47232268, 'num_objects': 5766, 'num_object_clones': 0, 'num_object_copies': 23064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5768, 'num_write_kb': 46133, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3537}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:20.762 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:44:20.762 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5815", 'reported_seq': 5871, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.544162+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:44:19.544162+0000', 'last_peered': '2024-09-16T04:44:19.544162+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:44:19.544162+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5815, 'log_dups_size': 0, 'ondisk_log_size': 5815, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47609100, 'num_objects': 5812, 'num_object_clones': 0, 'num_object_copies': 23248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5812, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5813, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:20.762 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:44:20.762 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5605", 'reported_seq': 5649, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.665803+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:44:19.665803+0000', 'last_peered': '2024-09-16T04:44:19.665803+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:44:19.665803+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5605, 'log_dups_size': 0, 'ondisk_log_size': 5605, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45916160, 'num_objects': 5605, 'num_object_clones': 0, 'num_object_copies': 22420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5605, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5605, 'num_write_kb': 44840, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3397}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:20.762 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:44:20.762 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'5772", 'reported_seq': 5837, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.917124+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:44:19.917124+0000', 'last_peered': '2024-09-16T04:44:19.917124+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:44:19.917124+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5772, 'log_dups_size': 0, 'ondisk_log_size': 5772, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47232268, 'num_objects': 5766, 'num_object_clones': 0, 'num_object_copies': 23064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5770, 'num_write_kb': 46140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3540}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:20.762 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:44:20.762 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5806", 'reported_seq': 6008, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.761693+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:44:19.761693+0000', 'last_peered': '2024-09-16T04:44:19.761693+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:44:19.761693+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5806, 'log_dups_size': 0, 'ondisk_log_size': 5806, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47491608, 'num_objects': 5798, 'num_object_clones': 0, 'num_object_copies': 23192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2268, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5803, 'num_write_kb': 46401, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 75, 'num_bytes_recovered': 589824, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3530}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:20.763 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:44:20.763 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5771", 'reported_seq': 5848, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.496916+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:44:19.496916+0000', 'last_peered': '2024-09-16T04:44:19.496916+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:44:19.496916+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5771, 'log_dups_size': 0, 'ondisk_log_size': 5771, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47256844, 'num_objects': 5769, 'num_object_clones': 0, 'num_object_copies': 23076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5769, 'num_write_kb': 46150, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3569}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:20.763 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:44:20.763 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5691", 'reported_seq': 5749, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.597205+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:44:19.597205+0000', 'last_peered': '2024-09-16T04:44:19.597205+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:44:19.597205+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5691, 'log_dups_size': 0, 'ondisk_log_size': 5691, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46575506, 'num_objects': 5686, 'num_object_clones': 0, 'num_object_copies': 22744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5686, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5688, 'num_write_kb': 45492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3497}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:20.763 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:44:20.763 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5637", 'reported_seq': 5677, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.717569+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:44:19.717569+0000', 'last_peered': '2024-09-16T04:44:19.717569+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:44:19.717569+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5637, 'log_dups_size': 0, 'ondisk_log_size': 5637, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46168710, 'num_objects': 5636, 'num_object_clones': 0, 'num_object_copies': 22544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5636, 'num_write_kb': 45087, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3508}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:20.763 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:44:20.763 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5679", 'reported_seq': 5753, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.376965+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:44:19.376965+0000', 'last_peered': '2024-09-16T04:44:19.376965+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:44:19.376965+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5679, 'log_dups_size': 0, 'ondisk_log_size': 5679, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46512774, 'num_objects': 5678, 'num_object_clones': 0, 'num_object_copies': 22712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5678, 'num_write_kb': 45423, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3453}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:20.763 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:44:20.763 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5803", 'reported_seq': 5857, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.742810+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:44:19.742810+0000', 'last_peered': '2024-09-16T04:44:19.742810+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:44:19.742810+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5803, 'log_dups_size': 0, 'ondisk_log_size': 5803, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47502604, 'num_objects': 5799, 'num_object_clones': 0, 'num_object_copies': 23196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5799, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5801, 'num_write_kb': 46404, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3565}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:20.764 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:44:20.764 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'5647", 'reported_seq': 5726, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.464559+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:44:19.464559+0000', 'last_peered': '2024-09-16T04:44:19.464559+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:44:19.464559+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5647, 'log_dups_size': 0, 'ondisk_log_size': 5647, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46232844, 'num_objects': 5644, 'num_object_clones': 0, 'num_object_copies': 22576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5644, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5645, 'num_write_kb': 45157, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3512}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:20.764 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:44:20.764 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5607", 'reported_seq': 5672, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.920077+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:44:19.920077+0000', 'last_peered': '2024-09-16T04:44:19.920077+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:44:19.920077+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5607, 'log_dups_size': 0, 'ondisk_log_size': 5607, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45888780, 'num_objects': 5602, 'num_object_clones': 0, 'num_object_copies': 22408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5605, 'num_write_kb': 44828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3363}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:20.764 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:44:21.315 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 207 20 92107 92087 3.47428 1.91406 0.0419394 0.0449567 2024-09-16T04:44:21.508 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:21.508 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33240/366744 objects degraded (9.064%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":273,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":91686,"data_bytes":751052456,"bytes_used":2612879360,"bytes_avail":383934177280,"bytes_total":386547056640,"degraded_objects":33240,"degraded_total":366744,"degraded_ratio":0.090635429618480465,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":67139,"recovering_keys_per_sec":0,"num_objects_recovered":79,"num_bytes_recovered":647168,"num_keys_recovered":0,"write_bytes_sec":2286135,"read_op_per_sec":0,"write_op_per_sec":279},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:44:21.538 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:44:21.538 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:22.101 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:22.101 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:22.123 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-16T04:44:20.620039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":751052456,"num_objects":91686,"num_object_clones":0,"num_object_copies":366744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91713,"num_write_kb":733586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2339,"num_bytes_recovered":18923520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91740,"ondisk_log_size":91740,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2551640,"kb_used_data":1330068,"kb_used_omap":14,"kb_used_meta":1221489,"kb_avail":374935720,"statfs":{"total":386547056640,"available":383934177280,"internally_reserved":0,"allocated":1361989632,"data_stored":1361269756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1250805297},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,11,11,15,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22036480,"num_objects":2690,"num_object_clones":0,"num_object_copies":10760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2690,"num_write_kb":21520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":647168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.639185"},"pg_stats":[{"pgid":"1.f","version":"33'5770","reported_seq":5814,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.724396+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:19.724396+0000","last_peered":"2024-09-16T04:44:19.724396+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:19.724396+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47267840,"num_objects":5770,"num_object_clones":0,"num_object_copies":23080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":46160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3551}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5790","reported_seq":5856,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.772309+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:19.772309+0000","last_peered":"2024-09-16T04:44:19.772309+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:19.772309+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5790,"log_dups_size":0,"ondisk_log_size":5790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47396108,"num_objects":5786,"num_object_clones":0,"num_object_copies":23144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5788,"num_write_kb":46300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3575}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5808","reported_seq":5883,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.531919+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:19.531919+0000","last_peered":"2024-09-16T04:44:19.531919+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:19.531919+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5808,"log_dups_size":0,"ondisk_log_size":5808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47553158,"num_objects":5805,"num_object_clones":0,"num_object_copies":23220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5807,"num_write_kb":46446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3578}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5770","reported_seq":5836,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.930983+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:19.930983+0000","last_peered":"2024-09-16T04:44:19.930983+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:19.930983+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47232268,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5768,"num_write_kb":46133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3537}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5815","reported_seq":5871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.544162+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:19.544162+0000","last_peered":"2024-09-16T04:44:19.544162+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:19.544162+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5815,"log_dups_size":0,"ondisk_log_size":5815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47609100,"num_objects":5812,"num_object_clones":0,"num_object_copies":23248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5813,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5605","reported_seq":5649,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.665803+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:19.665803+0000","last_peered":"2024-09-16T04:44:19.665803+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:19.665803+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5605,"log_dups_size":0,"ondisk_log_size":5605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45916160,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5772","reported_seq":5837,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.917124+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:19.917124+0000","last_peered":"2024-09-16T04:44:19.917124+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:19.917124+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47232268,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":46140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3540}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5769","reported_seq":10325,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:16.752845+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:16.752845+0000","last_peered":"2024-09-16T04:44:16.752845+0000","last_clean":"2024-09-16T04:44:16.752845+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:16.752845+0000","last_undegraded":"2024-09-16T04:44:16.752845+0000","last_fullsized":"2024-09-16T04:44:16.752845+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5769,"log_dups_size":0,"ondisk_log_size":5769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:17.859161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47215884,"num_objects":5764,"num_object_clones":0,"num_object_copies":23056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5767,"num_write_kb":46124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'5806","reported_seq":6008,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.761693+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:19.761693+0000","last_peered":"2024-09-16T04:44:19.761693+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:19.761693+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5806,"log_dups_size":0,"ondisk_log_size":5806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47491608,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5803,"num_write_kb":46401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5771","reported_seq":5848,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.496916+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:19.496916+0000","last_peered":"2024-09-16T04:44:19.496916+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:19.496916+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5771,"log_dups_size":0,"ondisk_log_size":5771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47256844,"num_objects":5769,"num_object_clones":0,"num_object_copies":23076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5769,"num_write_kb":46150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3569}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5691","reported_seq":5749,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.597205+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:19.597205+0000","last_peered":"2024-09-16T04:44:19.597205+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:19.597205+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5691,"log_dups_size":0,"ondisk_log_size":5691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46575506,"num_objects":5686,"num_object_clones":0,"num_object_copies":22744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5688,"num_write_kb":45492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5637","reported_seq":5677,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.717569+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:19.717569+0000","last_peered":"2024-09-16T04:44:19.717569+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:19.717569+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5637,"log_dups_size":0,"ondisk_log_size":5637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46168710,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5636,"num_write_kb":45087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3508}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5679","reported_seq":5753,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.376965+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:19.376965+0000","last_peered":"2024-09-16T04:44:19.376965+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:19.376965+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46512774,"num_objects":5678,"num_object_clones":0,"num_object_copies":22712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5678,"num_write_kb":45423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3453}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5803","reported_seq":5857,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.742810+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:19.742810+0000","last_peered":"2024-09-16T04:44:19.742810+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:19.742810+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47502604,"num_objects":5799,"num_object_clones":0,"num_object_copies":23196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5801,"num_write_kb":46404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3565}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5647","reported_seq":5726,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.464559+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:19.464559+0000","last_peered":"2024-09-16T04:44:19.464559+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:19.464559+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46232844,"num_objects":5644,"num_object_clones":0,"num_object_copies":22576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5645,"num_write_kb":45157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3512}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5607","reported_seq":5672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.920077+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:19.920077+0000","last_peered":"2024-09-16T04:44:19.920077+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:19.920077+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45888780,"num_objects":5602,"num_object_clones":0,"num_object_copies":22408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":751052456,"num_objects":91686,"num_object_clones":0,"num_object_copies":366744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91713,"num_write_kb":733586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2339,"num_bytes_recovered":18923520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1363296256,"data_stored":1363296256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91740,"ondisk_log_size":91740,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713116,"kb_used_data":366284,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93658724,"statfs":{"total":96636764160,"available":95906533376,"internally_reserved":0,"allocated":375074816,"data_stored":374894847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355135183},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694160,"kb_used_data":365884,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93677680,"statfs":{"total":96636764160,"available":95925944320,"internally_reserved":0,"allocated":374665216,"data_stored":374485247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439008,"kb_used_data":230992,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93932832,"statfs":{"total":96636764160,"available":96187219968,"internally_reserved":0,"allocated":236535808,"data_stored":236355839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":212987599},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,5,5,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705356,"kb_used_data":366908,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93666484,"statfs":{"total":96636764160,"available":95914479616,"internally_reserved":0,"allocated":375713792,"data_stored":375533823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":346549966},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":375848960,"data_stored":375848960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":236265472,"data_stored":236265472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":375488512,"data_stored":375488512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":375693312,"data_stored":375693312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:22.315 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 208 20 92368 92348 3.46738 2.03906 0.0334002 0.04505 2024-09-16T04:44:23.315 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 209 20 92655 92635 3.46152 2.24219 0.0479088 0.0451264 2024-09-16T04:44:24.315 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 210 20 92945 92925 3.45583 2.26562 0.069681 0.0452036 2024-09-16T04:44:25.124 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:25.315 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 211 20 93217 93197 3.44952 2.125 0.0641284 0.0452854 2024-09-16T04:44:25.750 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:25.751 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:25.780 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-16T04:44:24.621038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":751797928,"num_objects":91777,"num_object_clones":0,"num_object_copies":367108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91804,"num_write_kb":734314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2339,"num_bytes_recovered":18923520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91831,"ondisk_log_size":91831,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2579496,"kb_used_data":1335716,"kb_used_omap":14,"kb_used_meta":1243697,"kb_avail":374907864,"statfs":{"total":386547056640,"available":383905652736,"internally_reserved":0,"allocated":1367773184,"data_stored":1367053308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1273546289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,8,7,7,13,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22618112,"num_objects":2761,"num_object_clones":0,"num_object_copies":11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2761,"num_write_kb":22088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003353"},"pg_stats":[{"pgid":"1.f","version":"33'5770","reported_seq":5814,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.724396+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:19.724396+0000","last_peered":"2024-09-16T04:44:19.724396+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:19.724396+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47267840,"num_objects":5770,"num_object_clones":0,"num_object_copies":23080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":46160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3551}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5790","reported_seq":5856,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.772309+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:19.772309+0000","last_peered":"2024-09-16T04:44:19.772309+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:19.772309+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5790,"log_dups_size":0,"ondisk_log_size":5790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47396108,"num_objects":5786,"num_object_clones":0,"num_object_copies":23144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5788,"num_write_kb":46300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3575}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5808","reported_seq":5883,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.531919+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:19.531919+0000","last_peered":"2024-09-16T04:44:19.531919+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:19.531919+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5808,"log_dups_size":0,"ondisk_log_size":5808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47553158,"num_objects":5805,"num_object_clones":0,"num_object_copies":23220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5807,"num_write_kb":46446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3578}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5770","reported_seq":5836,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.930983+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:19.930983+0000","last_peered":"2024-09-16T04:44:19.930983+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:19.930983+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47232268,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5768,"num_write_kb":46133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3537}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5815","reported_seq":5871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.544162+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:19.544162+0000","last_peered":"2024-09-16T04:44:19.544162+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:19.544162+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5815,"log_dups_size":0,"ondisk_log_size":5815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47609100,"num_objects":5812,"num_object_clones":0,"num_object_copies":23248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5813,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5605","reported_seq":5649,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.665803+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:19.665803+0000","last_peered":"2024-09-16T04:44:19.665803+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:19.665803+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5605,"log_dups_size":0,"ondisk_log_size":5605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45916160,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5772","reported_seq":5837,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.917124+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:19.917124+0000","last_peered":"2024-09-16T04:44:19.917124+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:19.917124+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47232268,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":46140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3540}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5860","reported_seq":10415,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:22.012347+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:22.012347+0000","last_peered":"2024-09-16T04:44:22.012347+0000","last_clean":"2024-09-16T04:44:22.012347+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:22.012347+0000","last_undegraded":"2024-09-16T04:44:22.012347+0000","last_fullsized":"2024-09-16T04:44:22.012347+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5860,"log_dups_size":0,"ondisk_log_size":5860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:22.983512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47961356,"num_objects":5855,"num_object_clones":0,"num_object_copies":23420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5858,"num_write_kb":46852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'5806","reported_seq":6008,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.761693+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:19.761693+0000","last_peered":"2024-09-16T04:44:19.761693+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:19.761693+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5806,"log_dups_size":0,"ondisk_log_size":5806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47491608,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5803,"num_write_kb":46401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5771","reported_seq":5848,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.496916+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:19.496916+0000","last_peered":"2024-09-16T04:44:19.496916+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:19.496916+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5771,"log_dups_size":0,"ondisk_log_size":5771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47256844,"num_objects":5769,"num_object_clones":0,"num_object_copies":23076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5769,"num_write_kb":46150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3569}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5691","reported_seq":5749,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.597205+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:19.597205+0000","last_peered":"2024-09-16T04:44:19.597205+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:19.597205+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5691,"log_dups_size":0,"ondisk_log_size":5691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46575506,"num_objects":5686,"num_object_clones":0,"num_object_copies":22744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5688,"num_write_kb":45492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5637","reported_seq":5677,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.717569+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:19.717569+0000","last_peered":"2024-09-16T04:44:19.717569+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:19.717569+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5637,"log_dups_size":0,"ondisk_log_size":5637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46168710,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5636,"num_write_kb":45087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3508}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5679","reported_seq":5753,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.376965+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:19.376965+0000","last_peered":"2024-09-16T04:44:19.376965+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:19.376965+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46512774,"num_objects":5678,"num_object_clones":0,"num_object_copies":22712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5678,"num_write_kb":45423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3453}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5803","reported_seq":5857,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.742810+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:19.742810+0000","last_peered":"2024-09-16T04:44:19.742810+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:19.742810+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47502604,"num_objects":5799,"num_object_clones":0,"num_object_copies":23196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5801,"num_write_kb":46404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3565}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5647","reported_seq":5726,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.464559+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:19.464559+0000","last_peered":"2024-09-16T04:44:19.464559+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:19.464559+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46232844,"num_objects":5644,"num_object_clones":0,"num_object_copies":22576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5645,"num_write_kb":45157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3512}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5607","reported_seq":5672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.920077+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:19.920077+0000","last_peered":"2024-09-16T04:44:19.920077+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:19.920077+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45888780,"num_objects":5602,"num_object_clones":0,"num_object_copies":22408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":751797928,"num_objects":91777,"num_object_clones":0,"num_object_copies":367108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91804,"num_write_kb":734314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2339,"num_bytes_recovered":18923520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1369133056,"data_stored":1369133056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91831,"ondisk_log_size":91831,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713116,"kb_used_data":366284,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93658724,"statfs":{"total":96636764160,"available":95906533376,"internally_reserved":0,"allocated":375074816,"data_stored":374894847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355135183},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694160,"kb_used_data":365884,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93677680,"statfs":{"total":96636764160,"available":95925944320,"internally_reserved":0,"allocated":374665216,"data_stored":374485247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051620,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466864,"kb_used_data":236640,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93904976,"statfs":{"total":96636764160,"available":96158695424,"internally_reserved":0,"allocated":242319360,"data_stored":242139391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235728591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,1,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705356,"kb_used_data":366908,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93666484,"statfs":{"total":96636764160,"available":95914479616,"internally_reserved":0,"allocated":375713792,"data_stored":375533823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":346549966},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":375848960,"data_stored":375848960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":242102272,"data_stored":242102272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":375488512,"data_stored":375488512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":375693312,"data_stored":375693312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:25.782 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:26.316 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 212 20 93460 93440 3.44221 1.89844 0.0967347 0.0453743 2024-09-16T04:44:26.359 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:26.360 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:26.391 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-16T04:44:24.621038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":751797928,"num_objects":91777,"num_object_clones":0,"num_object_copies":367108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91804,"num_write_kb":734314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2339,"num_bytes_recovered":18923520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91831,"ondisk_log_size":91831,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2579496,"kb_used_data":1335716,"kb_used_omap":14,"kb_used_meta":1243697,"kb_avail":374907864,"statfs":{"total":386547056640,"available":383905652736,"internally_reserved":0,"allocated":1367773184,"data_stored":1367053308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1273546289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,8,7,7,13,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22618112,"num_objects":2761,"num_object_clones":0,"num_object_copies":11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2761,"num_write_kb":22088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003353"},"pg_stats":[{"pgid":"1.f","version":"33'5770","reported_seq":5814,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.724396+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:19.724396+0000","last_peered":"2024-09-16T04:44:19.724396+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:19.724396+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47267840,"num_objects":5770,"num_object_clones":0,"num_object_copies":23080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":46160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3551}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5790","reported_seq":5856,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.772309+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:19.772309+0000","last_peered":"2024-09-16T04:44:19.772309+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:19.772309+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5790,"log_dups_size":0,"ondisk_log_size":5790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47396108,"num_objects":5786,"num_object_clones":0,"num_object_copies":23144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5788,"num_write_kb":46300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3575}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5808","reported_seq":5883,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.531919+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:19.531919+0000","last_peered":"2024-09-16T04:44:19.531919+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:19.531919+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5808,"log_dups_size":0,"ondisk_log_size":5808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47553158,"num_objects":5805,"num_object_clones":0,"num_object_copies":23220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5807,"num_write_kb":46446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3578}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5770","reported_seq":5836,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.930983+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:19.930983+0000","last_peered":"2024-09-16T04:44:19.930983+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:19.930983+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47232268,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5768,"num_write_kb":46133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3537}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5815","reported_seq":5871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.544162+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:19.544162+0000","last_peered":"2024-09-16T04:44:19.544162+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:19.544162+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5815,"log_dups_size":0,"ondisk_log_size":5815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47609100,"num_objects":5812,"num_object_clones":0,"num_object_copies":23248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5813,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5605","reported_seq":5649,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.665803+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:19.665803+0000","last_peered":"2024-09-16T04:44:19.665803+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:19.665803+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5605,"log_dups_size":0,"ondisk_log_size":5605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45916160,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5772","reported_seq":5837,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.917124+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:19.917124+0000","last_peered":"2024-09-16T04:44:19.917124+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:19.917124+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47232268,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":46140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3540}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5860","reported_seq":10415,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:22.012347+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:22.012347+0000","last_peered":"2024-09-16T04:44:22.012347+0000","last_clean":"2024-09-16T04:44:22.012347+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:22.012347+0000","last_undegraded":"2024-09-16T04:44:22.012347+0000","last_fullsized":"2024-09-16T04:44:22.012347+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5860,"log_dups_size":0,"ondisk_log_size":5860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:22.983512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47961356,"num_objects":5855,"num_object_clones":0,"num_object_copies":23420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5858,"num_write_kb":46852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'5806","reported_seq":6008,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.761693+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:19.761693+0000","last_peered":"2024-09-16T04:44:19.761693+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:19.761693+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5806,"log_dups_size":0,"ondisk_log_size":5806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47491608,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5803,"num_write_kb":46401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5771","reported_seq":5848,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.496916+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:19.496916+0000","last_peered":"2024-09-16T04:44:19.496916+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:19.496916+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5771,"log_dups_size":0,"ondisk_log_size":5771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47256844,"num_objects":5769,"num_object_clones":0,"num_object_copies":23076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5769,"num_write_kb":46150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3569}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5691","reported_seq":5749,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.597205+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:19.597205+0000","last_peered":"2024-09-16T04:44:19.597205+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:19.597205+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5691,"log_dups_size":0,"ondisk_log_size":5691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46575506,"num_objects":5686,"num_object_clones":0,"num_object_copies":22744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5688,"num_write_kb":45492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5637","reported_seq":5677,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.717569+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:19.717569+0000","last_peered":"2024-09-16T04:44:19.717569+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:19.717569+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5637,"log_dups_size":0,"ondisk_log_size":5637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46168710,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5636,"num_write_kb":45087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3508}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5679","reported_seq":5753,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.376965+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:19.376965+0000","last_peered":"2024-09-16T04:44:19.376965+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:19.376965+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46512774,"num_objects":5678,"num_object_clones":0,"num_object_copies":22712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5678,"num_write_kb":45423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3453}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5803","reported_seq":5857,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.742810+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:19.742810+0000","last_peered":"2024-09-16T04:44:19.742810+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:19.742810+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47502604,"num_objects":5799,"num_object_clones":0,"num_object_copies":23196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5801,"num_write_kb":46404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3565}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5647","reported_seq":5726,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.464559+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:19.464559+0000","last_peered":"2024-09-16T04:44:19.464559+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:19.464559+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46232844,"num_objects":5644,"num_object_clones":0,"num_object_copies":22576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5645,"num_write_kb":45157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3512}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5607","reported_seq":5672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:19.920077+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:19.920077+0000","last_peered":"2024-09-16T04:44:19.920077+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:19.920077+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45888780,"num_objects":5602,"num_object_clones":0,"num_object_copies":22408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":751797928,"num_objects":91777,"num_object_clones":0,"num_object_copies":367108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91804,"num_write_kb":734314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2339,"num_bytes_recovered":18923520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1369133056,"data_stored":1369133056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91831,"ondisk_log_size":91831,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713116,"kb_used_data":366284,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93658724,"statfs":{"total":96636764160,"available":95906533376,"internally_reserved":0,"allocated":375074816,"data_stored":374894847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355135183},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694160,"kb_used_data":365884,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93677680,"statfs":{"total":96636764160,"available":95925944320,"internally_reserved":0,"allocated":374665216,"data_stored":374485247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051620,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466864,"kb_used_data":236640,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93904976,"statfs":{"total":96636764160,"available":96158695424,"internally_reserved":0,"allocated":242319360,"data_stored":242139391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235728591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,1,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705356,"kb_used_data":366908,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93666484,"statfs":{"total":96636764160,"available":95914479616,"internally_reserved":0,"allocated":375713792,"data_stored":375533823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":346549966},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":375848960,"data_stored":375848960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":242102272,"data_stored":242102272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":375488512,"data_stored":375488512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":375693312,"data_stored":375693312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:26.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:44:26.393 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5770", 'reported_seq': 5814, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.724396+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:44:19.724396+0000', 'last_peered': '2024-09-16T04:44:19.724396+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:44:19.724396+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5770, 'log_dups_size': 0, 'ondisk_log_size': 5770, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47267840, 'num_objects': 5770, 'num_object_clones': 0, 'num_object_copies': 23080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5770, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5770, 'num_write_kb': 46160, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3551}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:26.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:44:26.393 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5790", 'reported_seq': 5856, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.772309+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:44:19.772309+0000', 'last_peered': '2024-09-16T04:44:19.772309+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:44:19.772309+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5790, 'log_dups_size': 0, 'ondisk_log_size': 5790, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47396108, 'num_objects': 5786, 'num_object_clones': 0, 'num_object_copies': 23144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5786, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5788, 'num_write_kb': 46300, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3575}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:26.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:44:26.393 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5808", 'reported_seq': 5883, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.531919+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:44:19.531919+0000', 'last_peered': '2024-09-16T04:44:19.531919+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:44:19.531919+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5808, 'log_dups_size': 0, 'ondisk_log_size': 5808, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47553158, 'num_objects': 5805, 'num_object_clones': 0, 'num_object_copies': 23220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5805, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5807, 'num_write_kb': 46446, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3578}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:26.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:44:26.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5770", 'reported_seq': 5836, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.930983+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:44:19.930983+0000', 'last_peered': '2024-09-16T04:44:19.930983+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:44:19.930983+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5770, 'log_dups_size': 0, 'ondisk_log_size': 5770, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47232268, 'num_objects': 5766, 'num_object_clones': 0, 'num_object_copies': 23064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5768, 'num_write_kb': 46133, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3537}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:26.394 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:44:26.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5815", 'reported_seq': 5871, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.544162+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:44:19.544162+0000', 'last_peered': '2024-09-16T04:44:19.544162+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:44:19.544162+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5815, 'log_dups_size': 0, 'ondisk_log_size': 5815, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47609100, 'num_objects': 5812, 'num_object_clones': 0, 'num_object_copies': 23248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5812, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5813, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:26.394 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:44:26.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5605", 'reported_seq': 5649, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.665803+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:44:19.665803+0000', 'last_peered': '2024-09-16T04:44:19.665803+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:44:19.665803+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5605, 'log_dups_size': 0, 'ondisk_log_size': 5605, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45916160, 'num_objects': 5605, 'num_object_clones': 0, 'num_object_copies': 22420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5605, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5605, 'num_write_kb': 44840, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3397}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:26.394 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:44:26.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'5772", 'reported_seq': 5837, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.917124+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:44:19.917124+0000', 'last_peered': '2024-09-16T04:44:19.917124+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:44:19.917124+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5772, 'log_dups_size': 0, 'ondisk_log_size': 5772, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47232268, 'num_objects': 5766, 'num_object_clones': 0, 'num_object_copies': 23064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5770, 'num_write_kb': 46140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3540}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:26.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:44:26.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5806", 'reported_seq': 6008, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.761693+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:44:19.761693+0000', 'last_peered': '2024-09-16T04:44:19.761693+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:44:19.761693+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5806, 'log_dups_size': 0, 'ondisk_log_size': 5806, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47491608, 'num_objects': 5798, 'num_object_clones': 0, 'num_object_copies': 23192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2268, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5803, 'num_write_kb': 46401, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 75, 'num_bytes_recovered': 589824, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3530}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:26.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:44:26.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5771", 'reported_seq': 5848, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.496916+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:44:19.496916+0000', 'last_peered': '2024-09-16T04:44:19.496916+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:44:19.496916+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5771, 'log_dups_size': 0, 'ondisk_log_size': 5771, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47256844, 'num_objects': 5769, 'num_object_clones': 0, 'num_object_copies': 23076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5769, 'num_write_kb': 46150, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3569}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:26.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:44:26.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5691", 'reported_seq': 5749, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.597205+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:44:19.597205+0000', 'last_peered': '2024-09-16T04:44:19.597205+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:44:19.597205+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5691, 'log_dups_size': 0, 'ondisk_log_size': 5691, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46575506, 'num_objects': 5686, 'num_object_clones': 0, 'num_object_copies': 22744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5686, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5688, 'num_write_kb': 45492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3497}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:26.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:44:26.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5637", 'reported_seq': 5677, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.717569+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:44:19.717569+0000', 'last_peered': '2024-09-16T04:44:19.717569+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:44:19.717569+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5637, 'log_dups_size': 0, 'ondisk_log_size': 5637, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46168710, 'num_objects': 5636, 'num_object_clones': 0, 'num_object_copies': 22544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5636, 'num_write_kb': 45087, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3508}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:26.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:44:26.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5679", 'reported_seq': 5753, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.376965+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:44:19.376965+0000', 'last_peered': '2024-09-16T04:44:19.376965+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:44:19.376965+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5679, 'log_dups_size': 0, 'ondisk_log_size': 5679, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46512774, 'num_objects': 5678, 'num_object_clones': 0, 'num_object_copies': 22712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5678, 'num_write_kb': 45423, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3453}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:26.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:44:26.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5803", 'reported_seq': 5857, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.742810+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:44:19.742810+0000', 'last_peered': '2024-09-16T04:44:19.742810+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:44:19.742810+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5803, 'log_dups_size': 0, 'ondisk_log_size': 5803, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47502604, 'num_objects': 5799, 'num_object_clones': 0, 'num_object_copies': 23196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5799, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5801, 'num_write_kb': 46404, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3565}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:26.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:44:26.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'5647", 'reported_seq': 5726, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.464559+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:44:19.464559+0000', 'last_peered': '2024-09-16T04:44:19.464559+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:44:19.464559+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5647, 'log_dups_size': 0, 'ondisk_log_size': 5647, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46232844, 'num_objects': 5644, 'num_object_clones': 0, 'num_object_copies': 22576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5644, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5645, 'num_write_kb': 45157, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3512}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:26.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:44:26.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5607", 'reported_seq': 5672, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:19.920077+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:44:19.920077+0000', 'last_peered': '2024-09-16T04:44:19.920077+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:44:19.920077+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5607, 'log_dups_size': 0, 'ondisk_log_size': 5607, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45888780, 'num_objects': 5602, 'num_object_clones': 0, 'num_object_copies': 22408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5605, 'num_write_kb': 44828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3363}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:26.397 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:44:27.117 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:27.118 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33240/367108 objects degraded (9.055%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":279,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":91777,"data_bytes":751797928,"bytes_used":2641403904,"bytes_avail":383905652736,"bytes_total":386547056640,"degraded_objects":33240,"degraded_total":367108,"degraded_ratio":0.090545561524129139,"recovering_objects_per_sec":5,"recovering_bytes_per_sec":49138,"recovering_keys_per_sec":0,"num_objects_recovered":72,"num_bytes_recovered":589824,"num_keys_recovered":0,"write_bytes_sec":1884316,"read_op_per_sec":0,"write_op_per_sec":230},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:44:27.145 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:44:27.145 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:27.316 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 213 20 93713 93693 3.43533 1.97656 0.0645782 0.0454683 2024-09-16T04:44:27.727 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:27.727 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:27.759 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-16T04:44:26.621523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":762144424,"num_objects":93040,"num_object_clones":0,"num_object_copies":372160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93067,"num_write_kb":744418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2382,"num_bytes_recovered":19275776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93094,"ondisk_log_size":93094,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2587456,"kb_used_data":1351996,"kb_used_omap":14,"kb_used_meta":1235377,"kb_avail":374899904,"statfs":{"total":386547056640,"available":383897501696,"internally_reserved":0,"allocated":1384443904,"data_stored":1383724028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1265026609},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,10,7,14,15,13,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30392320,"num_objects":3710,"num_object_clones":0,"num_object_copies":14840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3710,"num_write_kb":29680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003198"},"pg_stats":[{"pgid":"1.f","version":"33'5848","reported_seq":5892,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:24.714920+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:24.714920+0000","last_peered":"2024-09-16T04:44:24.714920+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:24.714920+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5848,"log_dups_size":0,"ondisk_log_size":5848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47906816,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5848,"num_write_kb":46784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3629}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5872","reported_seq":5938,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:24.879022+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:24.879022+0000","last_peered":"2024-09-16T04:44:24.879022+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:24.879022+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5872,"log_dups_size":0,"ondisk_log_size":5872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48067852,"num_objects":5868,"num_object_clones":0,"num_object_copies":23472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5870,"num_write_kb":46956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3657}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5873","reported_seq":5948,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:24.491436+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:24.491436+0000","last_peered":"2024-09-16T04:44:24.491436+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:24.491436+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5873,"log_dups_size":0,"ondisk_log_size":5873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48085638,"num_objects":5870,"num_object_clones":0,"num_object_copies":23480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5872,"num_write_kb":46966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3643}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5854","reported_seq":5920,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:24.695193+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:24.695193+0000","last_peered":"2024-09-16T04:44:24.695193+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:24.695193+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5854,"log_dups_size":0,"ondisk_log_size":5854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47920396,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5852,"num_write_kb":46805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3621}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5899","reported_seq":5954,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:24.795525+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:24.795525+0000","last_peered":"2024-09-16T04:44:24.795525+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:24.795525+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5899,"log_dups_size":0,"ondisk_log_size":5899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48297228,"num_objects":5896,"num_object_clones":0,"num_object_copies":23584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5897,"num_write_kb":47173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3613}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5681","reported_seq":5725,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:24.628334+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:24.628334+0000","last_peered":"2024-09-16T04:44:24.628334+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:24.628334+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46538752,"num_objects":5681,"num_object_clones":0,"num_object_copies":22724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5681,"num_write_kb":45448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3473}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5865","reported_seq":5927,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:24.943884+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:24.943884+0000","last_peered":"2024-09-16T04:44:24.943884+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:24.943884+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5865,"log_dups_size":0,"ondisk_log_size":5865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47994124,"num_objects":5859,"num_object_clones":0,"num_object_copies":23436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5863,"num_write_kb":46884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3633}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5860","reported_seq":10415,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:22.012347+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:22.012347+0000","last_peered":"2024-09-16T04:44:22.012347+0000","last_clean":"2024-09-16T04:44:22.012347+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:22.012347+0000","last_undegraded":"2024-09-16T04:44:22.012347+0000","last_fullsized":"2024-09-16T04:44:22.012347+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5860,"log_dups_size":0,"ondisk_log_size":5860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:22.983512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47961356,"num_objects":5855,"num_object_clones":0,"num_object_copies":23420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5858,"num_write_kb":46852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'5887","reported_seq":6173,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:24.773228+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:24.773228+0000","last_peered":"2024-09-16T04:44:24.773228+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:24.773228+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5887,"log_dups_size":0,"ondisk_log_size":5887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48155160,"num_objects":5879,"num_object_clones":0,"num_object_copies":23516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5884,"num_write_kb":47049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3654}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5859","reported_seq":5935,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:24.584303+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:24.584303+0000","last_peered":"2024-09-16T04:44:24.584303+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:24.584303+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5859,"log_dups_size":0,"ondisk_log_size":5859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47977740,"num_objects":5857,"num_object_clones":0,"num_object_copies":23428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5857,"num_write_kb":46854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3657}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5786","reported_seq":5844,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:24.722628+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:24.722628+0000","last_peered":"2024-09-16T04:44:24.722628+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:24.722628+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5786,"log_dups_size":0,"ondisk_log_size":5786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47353746,"num_objects":5781,"num_object_clones":0,"num_object_copies":23124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5783,"num_write_kb":46252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5722","reported_seq":5762,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:24.723349+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:24.723349+0000","last_peered":"2024-09-16T04:44:24.723349+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:24.723349+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5722,"log_dups_size":0,"ondisk_log_size":5722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46865030,"num_objects":5721,"num_object_clones":0,"num_object_copies":22884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5721,"num_write_kb":45767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5763","reported_seq":5837,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:24.547760+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:24.547760+0000","last_peered":"2024-09-16T04:44:24.547760+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:24.547760+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5763,"log_dups_size":0,"ondisk_log_size":5763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47200902,"num_objects":5762,"num_object_clones":0,"num_object_copies":23048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5762,"num_write_kb":46095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3537}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5901","reported_seq":5954,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:24.767302+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:24.767302+0000","last_peered":"2024-09-16T04:44:24.767302+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:24.767302+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5901,"log_dups_size":0,"ondisk_log_size":5901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48305420,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5899,"num_write_kb":47188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3663}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5746","reported_seq":5823,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:24.561468+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:24.561468+0000","last_peered":"2024-09-16T04:44:24.561468+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:24.561468+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47043852,"num_objects":5743,"num_object_clones":0,"num_object_copies":22972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":45949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3611}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5678","reported_seq":5744,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:24.840815+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:24.840815+0000","last_peered":"2024-09-16T04:44:24.840815+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:24.840815+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5678,"log_dups_size":0,"ondisk_log_size":5678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46470412,"num_objects":5673,"num_object_clones":0,"num_object_copies":22692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5676,"num_write_kb":45396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":762144424,"num_objects":93040,"num_object_clones":0,"num_object_copies":372160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93067,"num_write_kb":744418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2382,"num_bytes_recovered":19275776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1385775104,"data_stored":1385775104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93094,"ondisk_log_size":93094,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706156,"kb_used_data":371804,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93665684,"statfs":{"total":96636764160,"available":95913660416,"internally_reserved":0,"allocated":380727296,"data_stored":380547327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":342355663},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,11,3,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699580,"kb_used_data":371308,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93672260,"statfs":{"total":96636764160,"available":95920394240,"internally_reserved":0,"allocated":380219392,"data_stored":380039423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051620,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466864,"kb_used_data":236640,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93904976,"statfs":{"total":96636764160,"available":96158695424,"internally_reserved":0,"allocated":242319360,"data_stored":242139391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235728591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,1,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714856,"kb_used_data":372244,"kb_used_omap":4,"kb_used_meta":342587,"kb_avail":93656984,"statfs":{"total":96636764160,"available":95904751616,"internally_reserved":0,"allocated":381177856,"data_stored":380997887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":350809806},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,2,1,8,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":381394944,"data_stored":381394944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":242102272,"data_stored":242102272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381026304,"data_stored":381026304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":381251584,"data_stored":381251584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:28.316 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 214 20 93981 93961 3.42906 2.09375 0.0928882 0.0455543 2024-09-16T04:44:29.316 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 215 20 94262 94242 3.42332 2.19531 0.0737384 0.0456299 2024-09-16T04:44:30.316 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 216 20 94538 94518 3.41746 2.15625 0.0453249 0.0457078 2024-09-16T04:44:30.761 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:31.316 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 217 20 94798 94778 3.41107 2.03125 0.0422402 0.0457914 2024-09-16T04:44:31.329 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:31.329 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:31.351 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-09-16T04:44:30.622552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":773260968,"num_objects":94397,"num_object_clones":0,"num_object_copies":377588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94424,"num_write_kb":755274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2418,"num_bytes_recovered":19570688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94451,"ondisk_log_size":94451,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2608848,"kb_used_data":1373392,"kb_used_omap":14,"kb_used_meta":1235377,"kb_avail":374878512,"statfs":{"total":386547056640,"available":383875596288,"internally_reserved":0,"allocated":1406353408,"data_stored":1405633532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1265026609},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,0,2,13,16,14,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32882688,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4014,"num_write_kb":32112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002995"},"pg_stats":[{"pgid":"1.f","version":"33'5945","reported_seq":5989,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.761726+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:29.761726+0000","last_peered":"2024-09-16T04:44:29.761726+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:29.761726+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5945,"log_dups_size":0,"ondisk_log_size":5945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48701440,"num_objects":5945,"num_object_clones":0,"num_object_copies":23780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5945,"num_write_kb":47560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5960","reported_seq":6026,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.927613+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:29.927613+0000","last_peered":"2024-09-16T04:44:29.927613+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:29.927613+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5960,"log_dups_size":0,"ondisk_log_size":5960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48788748,"num_objects":5956,"num_object_clones":0,"num_object_copies":23824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5958,"num_write_kb":47660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5950","reported_seq":6025,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.595388+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:29.595388+0000","last_peered":"2024-09-16T04:44:29.595388+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:29.595388+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5950,"log_dups_size":0,"ondisk_log_size":5950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48716422,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5949,"num_write_kb":47582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3720}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5953","reported_seq":6019,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.773532+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:29.773532+0000","last_peered":"2024-09-16T04:44:29.773532+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:29.773532+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5953,"log_dups_size":0,"ondisk_log_size":5953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48731404,"num_objects":5949,"num_object_clones":0,"num_object_copies":23796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5951,"num_write_kb":47597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3720}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5990","reported_seq":6046,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.740561+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:29.740561+0000","last_peered":"2024-09-16T04:44:29.740561+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:29.740561+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5990,"log_dups_size":0,"ondisk_log_size":5990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49042700,"num_objects":5987,"num_object_clones":0,"num_object_copies":23948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5988,"num_write_kb":47901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3704}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5780","reported_seq":5824,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.746518+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:29.746518+0000","last_peered":"2024-09-16T04:44:29.746518+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:29.746518+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47349760,"num_objects":5780,"num_object_clones":0,"num_object_copies":23120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5780,"num_write_kb":46240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3572}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5948","reported_seq":6013,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.948737+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:29.948737+0000","last_peered":"2024-09-16T04:44:29.948737+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:29.948737+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5948,"log_dups_size":0,"ondisk_log_size":5948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48674060,"num_objects":5942,"num_object_clones":0,"num_object_copies":23768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":47548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3716}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5957","reported_seq":10512,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:26.907579+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:26.907579+0000","last_peered":"2024-09-16T04:44:26.907579+0000","last_clean":"2024-09-16T04:44:26.907579+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:26.907579+0000","last_undegraded":"2024-09-16T04:44:26.907579+0000","last_fullsized":"2024-09-16T04:44:26.907579+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:27.989164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48755980,"num_objects":5952,"num_object_clones":0,"num_object_copies":23808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5955,"num_write_kb":47628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'5961","reported_seq":6320,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.861612+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:29.861612+0000","last_peered":"2024-09-16T04:44:29.861612+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:29.861612+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5961,"log_dups_size":0,"ondisk_log_size":5961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48761368,"num_objects":5953,"num_object_clones":0,"num_object_copies":23812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5958,"num_write_kb":47641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3764}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5932","reported_seq":6009,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.332677+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:29.332677+0000","last_peered":"2024-09-16T04:44:29.332677+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:29.332677+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5932,"log_dups_size":0,"ondisk_log_size":5932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48575756,"num_objects":5930,"num_object_clones":0,"num_object_copies":23720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5930,"num_write_kb":47438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3730}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5858","reported_seq":5916,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.685417+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:29.685417+0000","last_peered":"2024-09-16T04:44:29.685417+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:29.685417+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47943570,"num_objects":5853,"num_object_clones":0,"num_object_copies":23412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5855,"num_write_kb":46828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5799","reported_seq":5839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.798858+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:29.798858+0000","last_peered":"2024-09-16T04:44:29.798858+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:29.798858+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5799,"log_dups_size":0,"ondisk_log_size":5799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47495814,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5798,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3670}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5841","reported_seq":5915,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.436178+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:29.436178+0000","last_peered":"2024-09-16T04:44:29.436178+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:29.436178+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5841,"log_dups_size":0,"ondisk_log_size":5841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47839878,"num_objects":5840,"num_object_clones":0,"num_object_copies":23360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5840,"num_write_kb":46719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3615}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5982","reported_seq":6036,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.811437+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:29.811437+0000","last_peered":"2024-09-16T04:44:29.811437+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:29.811437+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5982,"log_dups_size":0,"ondisk_log_size":5982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48968972,"num_objects":5978,"num_object_clones":0,"num_object_copies":23912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5980,"num_write_kb":47836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3744}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5831","reported_seq":5910,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.585412+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:29.585412+0000","last_peered":"2024-09-16T04:44:29.585412+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:29.585412+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5831,"log_dups_size":0,"ondisk_log_size":5831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47740172,"num_objects":5828,"num_object_clones":0,"num_object_copies":23312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5829,"num_write_kb":46629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3696}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5764","reported_seq":5830,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.943676+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:29.943676+0000","last_peered":"2024-09-16T04:44:29.943676+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:29.943676+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5764,"log_dups_size":0,"ondisk_log_size":5764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47174924,"num_objects":5759,"num_object_clones":0,"num_object_copies":23036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5762,"num_write_kb":46084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":773260968,"num_objects":94397,"num_object_clones":0,"num_object_copies":377588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94424,"num_write_kb":755274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2418,"num_bytes_recovered":19570688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1407954944,"data_stored":1407954944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94451,"ondisk_log_size":94451,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715520,"kb_used_data":377008,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93656320,"statfs":{"total":96636764160,"available":95904071680,"internally_reserved":0,"allocated":386056192,"data_stored":385876223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346615503},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,7,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692408,"kb_used_data":376616,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93679432,"statfs":{"total":96636764160,"available":95927738368,"internally_reserved":0,"allocated":385654784,"data_stored":385474815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051621,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476584,"kb_used_data":242200,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93895256,"statfs":{"total":96636764160,"available":96148742144,"internally_reserved":0,"allocated":248012800,"data_stored":247832831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239988431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724336,"kb_used_data":377568,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93647504,"statfs":{"total":96636764160,"available":95895044096,"internally_reserved":0,"allocated":386629632,"data_stored":386449663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":355069646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":386875392,"data_stored":386875392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":247840768,"data_stored":247840768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386502656,"data_stored":386502656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":386736128,"data_stored":386736128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:31.352 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:31.985 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:31.986 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:32.012 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-09-16T04:44:30.622552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":773260968,"num_objects":94397,"num_object_clones":0,"num_object_copies":377588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94424,"num_write_kb":755274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2418,"num_bytes_recovered":19570688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94451,"ondisk_log_size":94451,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2608848,"kb_used_data":1373392,"kb_used_omap":14,"kb_used_meta":1235377,"kb_avail":374878512,"statfs":{"total":386547056640,"available":383875596288,"internally_reserved":0,"allocated":1406353408,"data_stored":1405633532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1265026609},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,0,2,13,16,14,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32882688,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4014,"num_write_kb":32112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002995"},"pg_stats":[{"pgid":"1.f","version":"33'5945","reported_seq":5989,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.761726+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:29.761726+0000","last_peered":"2024-09-16T04:44:29.761726+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:29.761726+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5945,"log_dups_size":0,"ondisk_log_size":5945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48701440,"num_objects":5945,"num_object_clones":0,"num_object_copies":23780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5945,"num_write_kb":47560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5960","reported_seq":6026,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.927613+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:29.927613+0000","last_peered":"2024-09-16T04:44:29.927613+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:29.927613+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5960,"log_dups_size":0,"ondisk_log_size":5960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48788748,"num_objects":5956,"num_object_clones":0,"num_object_copies":23824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5958,"num_write_kb":47660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5950","reported_seq":6025,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.595388+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:29.595388+0000","last_peered":"2024-09-16T04:44:29.595388+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:29.595388+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5950,"log_dups_size":0,"ondisk_log_size":5950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48716422,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5949,"num_write_kb":47582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3720}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5953","reported_seq":6019,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.773532+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:29.773532+0000","last_peered":"2024-09-16T04:44:29.773532+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:29.773532+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5953,"log_dups_size":0,"ondisk_log_size":5953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48731404,"num_objects":5949,"num_object_clones":0,"num_object_copies":23796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5951,"num_write_kb":47597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3720}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5990","reported_seq":6046,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.740561+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:29.740561+0000","last_peered":"2024-09-16T04:44:29.740561+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:29.740561+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5990,"log_dups_size":0,"ondisk_log_size":5990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49042700,"num_objects":5987,"num_object_clones":0,"num_object_copies":23948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5988,"num_write_kb":47901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3704}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5780","reported_seq":5824,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.746518+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:29.746518+0000","last_peered":"2024-09-16T04:44:29.746518+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:29.746518+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47349760,"num_objects":5780,"num_object_clones":0,"num_object_copies":23120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5780,"num_write_kb":46240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3572}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5948","reported_seq":6013,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.948737+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:29.948737+0000","last_peered":"2024-09-16T04:44:29.948737+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:29.948737+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5948,"log_dups_size":0,"ondisk_log_size":5948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48674060,"num_objects":5942,"num_object_clones":0,"num_object_copies":23768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":47548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3716}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'5957","reported_seq":10512,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:26.907579+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:26.907579+0000","last_peered":"2024-09-16T04:44:26.907579+0000","last_clean":"2024-09-16T04:44:26.907579+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:26.907579+0000","last_undegraded":"2024-09-16T04:44:26.907579+0000","last_fullsized":"2024-09-16T04:44:26.907579+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:27.989164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48755980,"num_objects":5952,"num_object_clones":0,"num_object_copies":23808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5955,"num_write_kb":47628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'5961","reported_seq":6320,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.861612+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:29.861612+0000","last_peered":"2024-09-16T04:44:29.861612+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:29.861612+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5961,"log_dups_size":0,"ondisk_log_size":5961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48761368,"num_objects":5953,"num_object_clones":0,"num_object_copies":23812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5958,"num_write_kb":47641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3764}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5932","reported_seq":6009,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.332677+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:29.332677+0000","last_peered":"2024-09-16T04:44:29.332677+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:29.332677+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5932,"log_dups_size":0,"ondisk_log_size":5932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48575756,"num_objects":5930,"num_object_clones":0,"num_object_copies":23720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5930,"num_write_kb":47438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3730}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5858","reported_seq":5916,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.685417+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:29.685417+0000","last_peered":"2024-09-16T04:44:29.685417+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:29.685417+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47943570,"num_objects":5853,"num_object_clones":0,"num_object_copies":23412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5855,"num_write_kb":46828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5799","reported_seq":5839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.798858+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:29.798858+0000","last_peered":"2024-09-16T04:44:29.798858+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:29.798858+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5799,"log_dups_size":0,"ondisk_log_size":5799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47495814,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5798,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3670}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5841","reported_seq":5915,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.436178+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:29.436178+0000","last_peered":"2024-09-16T04:44:29.436178+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:29.436178+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5841,"log_dups_size":0,"ondisk_log_size":5841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47839878,"num_objects":5840,"num_object_clones":0,"num_object_copies":23360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5840,"num_write_kb":46719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3615}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5982","reported_seq":6036,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.811437+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:29.811437+0000","last_peered":"2024-09-16T04:44:29.811437+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:29.811437+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5982,"log_dups_size":0,"ondisk_log_size":5982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48968972,"num_objects":5978,"num_object_clones":0,"num_object_copies":23912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5980,"num_write_kb":47836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3744}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5831","reported_seq":5910,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.585412+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:29.585412+0000","last_peered":"2024-09-16T04:44:29.585412+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:29.585412+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5831,"log_dups_size":0,"ondisk_log_size":5831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47740172,"num_objects":5828,"num_object_clones":0,"num_object_copies":23312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5829,"num_write_kb":46629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3696}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5764","reported_seq":5830,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.943676+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:29.943676+0000","last_peered":"2024-09-16T04:44:29.943676+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:29.943676+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5764,"log_dups_size":0,"ondisk_log_size":5764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47174924,"num_objects":5759,"num_object_clones":0,"num_object_copies":23036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5762,"num_write_kb":46084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":773260968,"num_objects":94397,"num_object_clones":0,"num_object_copies":377588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94424,"num_write_kb":755274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2418,"num_bytes_recovered":19570688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1407954944,"data_stored":1407954944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94451,"ondisk_log_size":94451,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715520,"kb_used_data":377008,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93656320,"statfs":{"total":96636764160,"available":95904071680,"internally_reserved":0,"allocated":386056192,"data_stored":385876223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346615503},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,7,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692408,"kb_used_data":376616,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93679432,"statfs":{"total":96636764160,"available":95927738368,"internally_reserved":0,"allocated":385654784,"data_stored":385474815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051621,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476584,"kb_used_data":242200,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93895256,"statfs":{"total":96636764160,"available":96148742144,"internally_reserved":0,"allocated":248012800,"data_stored":247832831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239988431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724336,"kb_used_data":377568,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93647504,"statfs":{"total":96636764160,"available":95895044096,"internally_reserved":0,"allocated":386629632,"data_stored":386449663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":355069646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":386875392,"data_stored":386875392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":247840768,"data_stored":247840768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386502656,"data_stored":386502656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":386736128,"data_stored":386736128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:32.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:44:32.014 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5945", 'reported_seq': 5989, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:29.761726+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:44:29.761726+0000', 'last_peered': '2024-09-16T04:44:29.761726+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:44:29.761726+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5945, 'log_dups_size': 0, 'ondisk_log_size': 5945, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48701440, 'num_objects': 5945, 'num_object_clones': 0, 'num_object_copies': 23780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5945, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5945, 'num_write_kb': 47560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3726}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:32.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:44:32.014 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5960", 'reported_seq': 6026, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:29.927613+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:44:29.927613+0000', 'last_peered': '2024-09-16T04:44:29.927613+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:44:29.927613+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5960, 'log_dups_size': 0, 'ondisk_log_size': 5960, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48788748, 'num_objects': 5956, 'num_object_clones': 0, 'num_object_copies': 23824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5958, 'num_write_kb': 47660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3745}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:32.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:44:32.014 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5950", 'reported_seq': 6025, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:29.595388+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:44:29.595388+0000', 'last_peered': '2024-09-16T04:44:29.595388+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:44:29.595388+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5950, 'log_dups_size': 0, 'ondisk_log_size': 5950, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48716422, 'num_objects': 5947, 'num_object_clones': 0, 'num_object_copies': 23788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5949, 'num_write_kb': 47582, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3720}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:32.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:44:32.014 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5953", 'reported_seq': 6019, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:29.773532+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:44:29.773532+0000', 'last_peered': '2024-09-16T04:44:29.773532+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:44:29.773532+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5953, 'log_dups_size': 0, 'ondisk_log_size': 5953, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48731404, 'num_objects': 5949, 'num_object_clones': 0, 'num_object_copies': 23796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5949, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5951, 'num_write_kb': 47597, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3720}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:32.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:44:32.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'5990", 'reported_seq': 6046, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:29.740561+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:44:29.740561+0000', 'last_peered': '2024-09-16T04:44:29.740561+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:44:29.740561+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5990, 'log_dups_size': 0, 'ondisk_log_size': 5990, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49042700, 'num_objects': 5987, 'num_object_clones': 0, 'num_object_copies': 23948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5987, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5988, 'num_write_kb': 47901, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3704}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:32.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:44:32.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5780", 'reported_seq': 5824, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:29.746518+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:44:29.746518+0000', 'last_peered': '2024-09-16T04:44:29.746518+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:44:29.746518+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5780, 'log_dups_size': 0, 'ondisk_log_size': 5780, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47349760, 'num_objects': 5780, 'num_object_clones': 0, 'num_object_copies': 23120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5780, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5780, 'num_write_kb': 46240, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3572}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:32.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:44:32.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'5948", 'reported_seq': 6013, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:29.948737+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:44:29.948737+0000', 'last_peered': '2024-09-16T04:44:29.948737+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:44:29.948737+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5948, 'log_dups_size': 0, 'ondisk_log_size': 5948, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48674060, 'num_objects': 5942, 'num_object_clones': 0, 'num_object_copies': 23768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5942, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5946, 'num_write_kb': 47548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3716}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:32.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:44:32.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5961", 'reported_seq': 6320, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:29.861612+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:44:29.861612+0000', 'last_peered': '2024-09-16T04:44:29.861612+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:44:29.861612+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5961, 'log_dups_size': 0, 'ondisk_log_size': 5961, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48761368, 'num_objects': 5953, 'num_object_clones': 0, 'num_object_copies': 23812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2189, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5953, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5958, 'num_write_kb': 47641, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 154, 'num_bytes_recovered': 1236992, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3764}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:32.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:44:32.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5932", 'reported_seq': 6009, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:29.332677+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:44:29.332677+0000', 'last_peered': '2024-09-16T04:44:29.332677+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:44:29.332677+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5932, 'log_dups_size': 0, 'ondisk_log_size': 5932, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48575756, 'num_objects': 5930, 'num_object_clones': 0, 'num_object_copies': 23720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5930, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5930, 'num_write_kb': 47438, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3730}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:32.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:44:32.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5858", 'reported_seq': 5916, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:29.685417+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:44:29.685417+0000', 'last_peered': '2024-09-16T04:44:29.685417+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:44:29.685417+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5858, 'log_dups_size': 0, 'ondisk_log_size': 5858, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47943570, 'num_objects': 5853, 'num_object_clones': 0, 'num_object_copies': 23412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5853, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5855, 'num_write_kb': 46828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3664}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:32.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:44:32.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5799", 'reported_seq': 5839, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:29.798858+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:44:29.798858+0000', 'last_peered': '2024-09-16T04:44:29.798858+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:44:29.798858+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5799, 'log_dups_size': 0, 'ondisk_log_size': 5799, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47495814, 'num_objects': 5798, 'num_object_clones': 0, 'num_object_copies': 23192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5798, 'num_write_kb': 46383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3670}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:32.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:44:32.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5841", 'reported_seq': 5915, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:29.436178+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:44:29.436178+0000', 'last_peered': '2024-09-16T04:44:29.436178+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:44:29.436178+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5841, 'log_dups_size': 0, 'ondisk_log_size': 5841, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47839878, 'num_objects': 5840, 'num_object_clones': 0, 'num_object_copies': 23360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5840, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5840, 'num_write_kb': 46719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3615}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:32.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:44:32.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5982", 'reported_seq': 6036, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:29.811437+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:44:29.811437+0000', 'last_peered': '2024-09-16T04:44:29.811437+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:44:29.811437+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5982, 'log_dups_size': 0, 'ondisk_log_size': 5982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48968972, 'num_objects': 5978, 'num_object_clones': 0, 'num_object_copies': 23912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5980, 'num_write_kb': 47836, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3744}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:32.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:44:32.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'5831", 'reported_seq': 5910, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:29.585412+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:44:29.585412+0000', 'last_peered': '2024-09-16T04:44:29.585412+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:44:29.585412+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5831, 'log_dups_size': 0, 'ondisk_log_size': 5831, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47740172, 'num_objects': 5828, 'num_object_clones': 0, 'num_object_copies': 23312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5828, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5829, 'num_write_kb': 46629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3696}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:32.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:44:32.017 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5764", 'reported_seq': 5830, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:29.943676+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:44:29.943676+0000', 'last_peered': '2024-09-16T04:44:29.943676+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:44:29.943676+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5764, 'log_dups_size': 0, 'ondisk_log_size': 5764, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47174924, 'num_objects': 5759, 'num_object_clones': 0, 'num_object_copies': 23036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5762, 'num_write_kb': 46084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3520}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:32.017 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:44:32.316 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 218 20 95034 95014 3.40388 1.84375 0.056613 0.04589 2024-09-16T04:44:32.784 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:32.785 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33161/377588 objects degraded (8.782%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":285,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":94397,"data_bytes":773260968,"bytes_used":2671460352,"bytes_avail":383875596288,"bytes_total":386547056640,"degraded_objects":33161,"degraded_total":377588,"degraded_ratio":0.087823235907920794,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":77122,"recovering_keys_per_sec":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"write_bytes_sec":2739540,"read_op_per_sec":0,"write_op_per_sec":334},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:44:32.807 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:44:32.808 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:33.316 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 219 20 95282 95262 3.39719 1.9375 0.0587202 0.0459784 2024-09-16T04:44:33.417 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:33.418 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:33.448 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-16T04:44:32.623079+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":773727912,"num_objects":94454,"num_object_clones":0,"num_object_copies":377816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94481,"num_write_kb":755730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2418,"num_bytes_recovered":19570688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94508,"ondisk_log_size":94508,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2618436,"kb_used_data":1378820,"kb_used_omap":14,"kb_used_meta":1239537,"kb_avail":374868924,"statfs":{"total":386547056640,"available":383865778176,"internally_reserved":0,"allocated":1411911680,"data_stored":1411191804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1269286449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,0,4,13,12,16,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22675456,"num_objects":2768,"num_object_clones":0,"num_object_copies":11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2768,"num_write_kb":22144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":647168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003040"},"pg_stats":[{"pgid":"1.f","version":"33'5945","reported_seq":5989,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.761726+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:29.761726+0000","last_peered":"2024-09-16T04:44:29.761726+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:29.761726+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5945,"log_dups_size":0,"ondisk_log_size":5945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48701440,"num_objects":5945,"num_object_clones":0,"num_object_copies":23780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5945,"num_write_kb":47560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5960","reported_seq":6026,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.927613+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:29.927613+0000","last_peered":"2024-09-16T04:44:29.927613+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:29.927613+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5960,"log_dups_size":0,"ondisk_log_size":5960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48788748,"num_objects":5956,"num_object_clones":0,"num_object_copies":23824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5958,"num_write_kb":47660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5950","reported_seq":6025,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.595388+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:29.595388+0000","last_peered":"2024-09-16T04:44:29.595388+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:29.595388+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5950,"log_dups_size":0,"ondisk_log_size":5950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48716422,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5949,"num_write_kb":47582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3720}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5953","reported_seq":6019,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.773532+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:29.773532+0000","last_peered":"2024-09-16T04:44:29.773532+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:29.773532+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5953,"log_dups_size":0,"ondisk_log_size":5953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48731404,"num_objects":5949,"num_object_clones":0,"num_object_copies":23796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5951,"num_write_kb":47597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3720}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'5990","reported_seq":6046,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.740561+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:29.740561+0000","last_peered":"2024-09-16T04:44:29.740561+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:29.740561+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5990,"log_dups_size":0,"ondisk_log_size":5990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49042700,"num_objects":5987,"num_object_clones":0,"num_object_copies":23948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5988,"num_write_kb":47901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3704}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5780","reported_seq":5824,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.746518+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:29.746518+0000","last_peered":"2024-09-16T04:44:29.746518+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:29.746518+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47349760,"num_objects":5780,"num_object_clones":0,"num_object_copies":23120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5780,"num_write_kb":46240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3572}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'5948","reported_seq":6013,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.948737+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:29.948737+0000","last_peered":"2024-09-16T04:44:29.948737+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:29.948737+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5948,"log_dups_size":0,"ondisk_log_size":5948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48674060,"num_objects":5942,"num_object_clones":0,"num_object_copies":23768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":47548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3716}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6014","reported_seq":10570,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:31.915309+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:31.915309+0000","last_peered":"2024-09-16T04:44:31.915309+0000","last_clean":"2024-09-16T04:44:31.915309+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:31.915309+0000","last_undegraded":"2024-09-16T04:44:31.915309+0000","last_fullsized":"2024-09-16T04:44:31.915309+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6014,"log_dups_size":0,"ondisk_log_size":6014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:33.967800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49222924,"num_objects":6009,"num_object_clones":0,"num_object_copies":24036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6012,"num_write_kb":48084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'5961","reported_seq":6320,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.861612+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:29.861612+0000","last_peered":"2024-09-16T04:44:29.861612+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:29.861612+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5961,"log_dups_size":0,"ondisk_log_size":5961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48761368,"num_objects":5953,"num_object_clones":0,"num_object_copies":23812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5958,"num_write_kb":47641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3764}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5932","reported_seq":6009,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.332677+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:29.332677+0000","last_peered":"2024-09-16T04:44:29.332677+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:29.332677+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5932,"log_dups_size":0,"ondisk_log_size":5932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48575756,"num_objects":5930,"num_object_clones":0,"num_object_copies":23720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5930,"num_write_kb":47438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3730}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5858","reported_seq":5916,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.685417+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:29.685417+0000","last_peered":"2024-09-16T04:44:29.685417+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:29.685417+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47943570,"num_objects":5853,"num_object_clones":0,"num_object_copies":23412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5855,"num_write_kb":46828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5799","reported_seq":5839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.798858+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:29.798858+0000","last_peered":"2024-09-16T04:44:29.798858+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:29.798858+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5799,"log_dups_size":0,"ondisk_log_size":5799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47495814,"num_objects":5798,"num_object_clones":0,"num_object_copies":23192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5798,"num_write_kb":46383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3670}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5841","reported_seq":5915,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.436178+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:29.436178+0000","last_peered":"2024-09-16T04:44:29.436178+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:29.436178+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5841,"log_dups_size":0,"ondisk_log_size":5841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47839878,"num_objects":5840,"num_object_clones":0,"num_object_copies":23360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5840,"num_write_kb":46719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3615}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5982","reported_seq":6036,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.811437+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:29.811437+0000","last_peered":"2024-09-16T04:44:29.811437+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:29.811437+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5982,"log_dups_size":0,"ondisk_log_size":5982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48968972,"num_objects":5978,"num_object_clones":0,"num_object_copies":23912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5980,"num_write_kb":47836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3744}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5831","reported_seq":5910,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.585412+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:29.585412+0000","last_peered":"2024-09-16T04:44:29.585412+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:29.585412+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5831,"log_dups_size":0,"ondisk_log_size":5831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47740172,"num_objects":5828,"num_object_clones":0,"num_object_copies":23312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5829,"num_write_kb":46629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3696}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5764","reported_seq":5830,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:29.943676+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:29.943676+0000","last_peered":"2024-09-16T04:44:29.943676+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:29.943676+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5764,"log_dups_size":0,"ondisk_log_size":5764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47174924,"num_objects":5759,"num_object_clones":0,"num_object_copies":23036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5762,"num_write_kb":46084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":773727912,"num_objects":94454,"num_object_clones":0,"num_object_copies":377816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94481,"num_write_kb":755730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2418,"num_bytes_recovered":19570688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413529600,"data_stored":1413529600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94508,"ondisk_log_size":94508,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715520,"kb_used_data":377008,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93656320,"statfs":{"total":96636764160,"available":95904071680,"internally_reserved":0,"allocated":386056192,"data_stored":385876223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":346615503},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,7,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692408,"kb_used_data":376616,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93679432,"statfs":{"total":96636764160,"available":95927738368,"internally_reserved":0,"allocated":385654784,"data_stored":385474815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051622,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486172,"kb_used_data":247628,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93885668,"statfs":{"total":96636764160,"available":96138924032,"internally_reserved":0,"allocated":253571072,"data_stored":253391103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244248271},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724336,"kb_used_data":377568,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93647504,"statfs":{"total":96636764160,"available":95895044096,"internally_reserved":0,"allocated":386629632,"data_stored":386449663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":355069646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":386875392,"data_stored":386875392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":253415424,"data_stored":253415424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386502656,"data_stored":386502656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":386736128,"data_stored":386736128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:34.316 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:2024-09-16T04:44:34.315812+0000 min lat: 0.00269411 max lat: 1.73107 avg lat: 0.0460603 2024-09-16T04:44:34.316 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T04:44:34.316 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 220 20 95553 95533 3.39137 2.11719 0.121473 0.0460603 2024-09-16T04:44:35.316 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 221 20 95828 95808 3.38574 2.14844 0.0383408 0.0461355 2024-09-16T04:44:36.316 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 222 20 96104 96084 3.38021 2.15625 0.0288373 0.0462112 2024-09-16T04:44:36.450 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:37.068 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:37.069 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:37.091 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-09-16T04:44:36.624046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":783771304,"num_objects":95680,"num_object_clones":0,"num_object_copies":382720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95707,"num_write_kb":765538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2452,"num_bytes_recovered":19849216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95734,"ondisk_log_size":95734,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2653012,"kb_used_data":1395348,"kb_used_omap":14,"kb_used_meta":1257585,"kb_avail":374834348,"statfs":{"total":386547056640,"available":383830372352,"internally_reserved":0,"allocated":1428836352,"data_stored":1428116476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1287767601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,7,5,10,17,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31973376,"num_objects":3903,"num_object_clones":0,"num_object_copies":15612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3903,"num_write_kb":31224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003007"},"pg_stats":[{"pgid":"1.f","version":"33'6008","reported_seq":6052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.791600+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:34.791600+0000","last_peered":"2024-09-16T04:44:34.791600+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:34.791600+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6008,"log_dups_size":0,"ondisk_log_size":6008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49217536,"num_objects":6008,"num_object_clones":0,"num_object_copies":24032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6008,"num_write_kb":48064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6051","reported_seq":6117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:35.041518+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:35.041518+0000","last_peered":"2024-09-16T04:44:35.041518+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:35.041518+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6051,"log_dups_size":0,"ondisk_log_size":6051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49534220,"num_objects":6047,"num_object_clones":0,"num_object_copies":24188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":48388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3836}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6042","reported_seq":6117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.627700+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:34.627700+0000","last_peered":"2024-09-16T04:44:34.627700+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:34.627700+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6042,"log_dups_size":0,"ondisk_log_size":6042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49470086,"num_objects":6039,"num_object_clones":0,"num_object_copies":24156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6041,"num_write_kb":48318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3812}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6036","reported_seq":6102,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.853061+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:34.853061+0000","last_peered":"2024-09-16T04:44:34.853061+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:34.853061+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6036,"log_dups_size":0,"ondisk_log_size":6036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49411340,"num_objects":6032,"num_object_clones":0,"num_object_copies":24128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6034,"num_write_kb":48261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6065","reported_seq":6119,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.834422+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:34.834422+0000","last_peered":"2024-09-16T04:44:34.834422+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:34.834422+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6065,"log_dups_size":0,"ondisk_log_size":6065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49657100,"num_objects":6062,"num_object_clones":0,"num_object_copies":24248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6063,"num_write_kb":48501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5858","reported_seq":5902,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.758834+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:34.758834+0000","last_peered":"2024-09-16T04:44:34.758834+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:34.758834+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47988736,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5858,"num_write_kb":46864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6007","reported_seq":6072,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.897505+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:34.897505+0000","last_peered":"2024-09-16T04:44:34.897505+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:34.897505+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6007,"log_dups_size":0,"ondisk_log_size":6007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49157388,"num_objects":6001,"num_object_clones":0,"num_object_copies":24004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6005,"num_write_kb":48020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3775}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6014","reported_seq":10570,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:31.915309+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:31.915309+0000","last_peered":"2024-09-16T04:44:31.915309+0000","last_clean":"2024-09-16T04:44:31.915309+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:31.915309+0000","last_undegraded":"2024-09-16T04:44:31.915309+0000","last_fullsized":"2024-09-16T04:44:31.915309+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6014,"log_dups_size":0,"ondisk_log_size":6014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:33.967800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49222924,"num_objects":6009,"num_object_clones":0,"num_object_copies":24036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6012,"num_write_kb":48084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6056","reported_seq":6482,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.862074+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:34.862074+0000","last_peered":"2024-09-16T04:44:34.862074+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:34.862074+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6056,"log_dups_size":0,"ondisk_log_size":6056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49539608,"num_objects":6048,"num_object_clones":0,"num_object_copies":24192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6053,"num_write_kb":48401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":1515520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3893}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6033","reported_seq":6112,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.686840+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:34.686840+0000","last_peered":"2024-09-16T04:44:34.686840+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:34.686840+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49403148,"num_objects":6031,"num_object_clones":0,"num_object_copies":24124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6031,"num_write_kb":48246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3831}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5931","reported_seq":5989,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.846934+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:34.846934+0000","last_peered":"2024-09-16T04:44:34.846934+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:34.846934+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5931,"log_dups_size":0,"ondisk_log_size":5931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48541586,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5928,"num_write_kb":47412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3737}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5872","reported_seq":5910,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.725379+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:34.725379+0000","last_peered":"2024-09-16T04:44:34.725379+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:34.725379+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5872,"log_dups_size":0,"ondisk_log_size":5872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48093830,"num_objects":5871,"num_object_clones":0,"num_object_copies":23484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5871,"num_write_kb":46967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3743}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5928","reported_seq":6002,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.529155+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:34.529155+0000","last_peered":"2024-09-16T04:44:34.529155+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:34.529155+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48552582,"num_objects":5927,"num_object_clones":0,"num_object_copies":23708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5927,"num_write_kb":47415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3702}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6067","reported_seq":6121,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.723960+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:34.723960+0000","last_peered":"2024-09-16T04:44:34.723960+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:34.723960+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6067,"log_dups_size":0,"ondisk_log_size":6067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49665292,"num_objects":6063,"num_object_clones":0,"num_object_copies":24252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6065,"num_write_kb":48516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3829}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5911","reported_seq":5990,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.476276+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:34.476276+0000","last_peered":"2024-09-16T04:44:34.476276+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:34.476276+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5911,"log_dups_size":0,"ondisk_log_size":5911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48395532,"num_objects":5908,"num_object_clones":0,"num_object_copies":23632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5909,"num_write_kb":47269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3776}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5855","reported_seq":5921,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:35.011177+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:35.011177+0000","last_peered":"2024-09-16T04:44:35.011177+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:35.011177+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5855,"log_dups_size":0,"ondisk_log_size":5855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47920396,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5853,"num_write_kb":46812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3611}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":783771304,"num_objects":95680,"num_object_clones":0,"num_object_copies":382720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95707,"num_write_kb":765538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2452,"num_bytes_recovered":19849216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1429467136,"data_stored":1429467136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95734,"ondisk_log_size":95734,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724896,"kb_used_data":382224,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93646944,"statfs":{"total":96636764160,"available":95894470656,"internally_reserved":0,"allocated":391397376,"data_stored":391217407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350875343},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,8,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724960,"kb_used_data":382800,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93646880,"statfs":{"total":96636764160,"available":95894405120,"internally_reserved":0,"allocated":391987200,"data_stored":391807231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,0,1,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051622,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486172,"kb_used_data":247628,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93885668,"statfs":{"total":96636764160,"available":96138924032,"internally_reserved":0,"allocated":253571072,"data_stored":253391103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244248271},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716984,"kb_used_data":382696,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93654856,"statfs":{"total":96636764160,"available":95902572544,"internally_reserved":0,"allocated":391880704,"data_stored":391700735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":342290126},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":392241152,"data_stored":392241152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":253415424,"data_stored":253415424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391802880,"data_stored":391802880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":392007680,"data_stored":392007680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:37.092 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:37.316 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 223 20 96349 96329 3.37363 1.91406 0.126843 0.0463011 2024-09-16T04:44:37.713 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:37.714 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:37.741 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-09-16T04:44:36.624046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":783771304,"num_objects":95680,"num_object_clones":0,"num_object_copies":382720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95707,"num_write_kb":765538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2452,"num_bytes_recovered":19849216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95734,"ondisk_log_size":95734,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2653012,"kb_used_data":1395348,"kb_used_omap":14,"kb_used_meta":1257585,"kb_avail":374834348,"statfs":{"total":386547056640,"available":383830372352,"internally_reserved":0,"allocated":1428836352,"data_stored":1428116476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1287767601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,7,5,10,17,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31973376,"num_objects":3903,"num_object_clones":0,"num_object_copies":15612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3903,"num_write_kb":31224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003007"},"pg_stats":[{"pgid":"1.f","version":"33'6008","reported_seq":6052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.791600+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:34.791600+0000","last_peered":"2024-09-16T04:44:34.791600+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:34.791600+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6008,"log_dups_size":0,"ondisk_log_size":6008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49217536,"num_objects":6008,"num_object_clones":0,"num_object_copies":24032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6008,"num_write_kb":48064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6051","reported_seq":6117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:35.041518+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:35.041518+0000","last_peered":"2024-09-16T04:44:35.041518+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:35.041518+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6051,"log_dups_size":0,"ondisk_log_size":6051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49534220,"num_objects":6047,"num_object_clones":0,"num_object_copies":24188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":48388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3836}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6042","reported_seq":6117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.627700+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:34.627700+0000","last_peered":"2024-09-16T04:44:34.627700+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:34.627700+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6042,"log_dups_size":0,"ondisk_log_size":6042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49470086,"num_objects":6039,"num_object_clones":0,"num_object_copies":24156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6041,"num_write_kb":48318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3812}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6036","reported_seq":6102,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.853061+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:34.853061+0000","last_peered":"2024-09-16T04:44:34.853061+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:34.853061+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6036,"log_dups_size":0,"ondisk_log_size":6036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49411340,"num_objects":6032,"num_object_clones":0,"num_object_copies":24128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6034,"num_write_kb":48261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6065","reported_seq":6119,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.834422+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:34.834422+0000","last_peered":"2024-09-16T04:44:34.834422+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:34.834422+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6065,"log_dups_size":0,"ondisk_log_size":6065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49657100,"num_objects":6062,"num_object_clones":0,"num_object_copies":24248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6063,"num_write_kb":48501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5858","reported_seq":5902,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.758834+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:34.758834+0000","last_peered":"2024-09-16T04:44:34.758834+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:34.758834+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47988736,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5858,"num_write_kb":46864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6007","reported_seq":6072,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.897505+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:34.897505+0000","last_peered":"2024-09-16T04:44:34.897505+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:34.897505+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6007,"log_dups_size":0,"ondisk_log_size":6007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49157388,"num_objects":6001,"num_object_clones":0,"num_object_copies":24004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6005,"num_write_kb":48020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3775}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6014","reported_seq":10570,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:31.915309+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:31.915309+0000","last_peered":"2024-09-16T04:44:31.915309+0000","last_clean":"2024-09-16T04:44:31.915309+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:31.915309+0000","last_undegraded":"2024-09-16T04:44:31.915309+0000","last_fullsized":"2024-09-16T04:44:31.915309+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6014,"log_dups_size":0,"ondisk_log_size":6014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:33.967800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49222924,"num_objects":6009,"num_object_clones":0,"num_object_copies":24036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6012,"num_write_kb":48084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6056","reported_seq":6482,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.862074+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:34.862074+0000","last_peered":"2024-09-16T04:44:34.862074+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:34.862074+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6056,"log_dups_size":0,"ondisk_log_size":6056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49539608,"num_objects":6048,"num_object_clones":0,"num_object_copies":24192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6053,"num_write_kb":48401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":1515520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3893}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6033","reported_seq":6112,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.686840+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:34.686840+0000","last_peered":"2024-09-16T04:44:34.686840+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:34.686840+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49403148,"num_objects":6031,"num_object_clones":0,"num_object_copies":24124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6031,"num_write_kb":48246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3831}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5931","reported_seq":5989,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.846934+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:34.846934+0000","last_peered":"2024-09-16T04:44:34.846934+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:34.846934+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5931,"log_dups_size":0,"ondisk_log_size":5931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48541586,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5928,"num_write_kb":47412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3737}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5872","reported_seq":5910,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.725379+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:34.725379+0000","last_peered":"2024-09-16T04:44:34.725379+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:34.725379+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5872,"log_dups_size":0,"ondisk_log_size":5872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48093830,"num_objects":5871,"num_object_clones":0,"num_object_copies":23484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5871,"num_write_kb":46967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3743}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5928","reported_seq":6002,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.529155+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:34.529155+0000","last_peered":"2024-09-16T04:44:34.529155+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:34.529155+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48552582,"num_objects":5927,"num_object_clones":0,"num_object_copies":23708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5927,"num_write_kb":47415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3702}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6067","reported_seq":6121,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.723960+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:34.723960+0000","last_peered":"2024-09-16T04:44:34.723960+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:34.723960+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6067,"log_dups_size":0,"ondisk_log_size":6067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49665292,"num_objects":6063,"num_object_clones":0,"num_object_copies":24252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6065,"num_write_kb":48516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3829}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5911","reported_seq":5990,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.476276+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:34.476276+0000","last_peered":"2024-09-16T04:44:34.476276+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:34.476276+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5911,"log_dups_size":0,"ondisk_log_size":5911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48395532,"num_objects":5908,"num_object_clones":0,"num_object_copies":23632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5909,"num_write_kb":47269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3776}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5855","reported_seq":5921,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:35.011177+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:35.011177+0000","last_peered":"2024-09-16T04:44:35.011177+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:35.011177+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5855,"log_dups_size":0,"ondisk_log_size":5855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47920396,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5853,"num_write_kb":46812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3611}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":783771304,"num_objects":95680,"num_object_clones":0,"num_object_copies":382720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95707,"num_write_kb":765538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2452,"num_bytes_recovered":19849216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1429467136,"data_stored":1429467136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95734,"ondisk_log_size":95734,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724896,"kb_used_data":382224,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93646944,"statfs":{"total":96636764160,"available":95894470656,"internally_reserved":0,"allocated":391397376,"data_stored":391217407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350875343},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,8,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724960,"kb_used_data":382800,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93646880,"statfs":{"total":96636764160,"available":95894405120,"internally_reserved":0,"allocated":391987200,"data_stored":391807231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,0,1,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051622,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486172,"kb_used_data":247628,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93885668,"statfs":{"total":96636764160,"available":96138924032,"internally_reserved":0,"allocated":253571072,"data_stored":253391103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244248271},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716984,"kb_used_data":382696,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93654856,"statfs":{"total":96636764160,"available":95902572544,"internally_reserved":0,"allocated":391880704,"data_stored":391700735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":342290126},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":392241152,"data_stored":392241152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":253415424,"data_stored":253415424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391802880,"data_stored":391802880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":392007680,"data_stored":392007680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:37.743 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:44:37.743 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6008", 'reported_seq': 6052, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:34.791600+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:44:34.791600+0000', 'last_peered': '2024-09-16T04:44:34.791600+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:44:34.791600+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6008, 'log_dups_size': 0, 'ondisk_log_size': 6008, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49217536, 'num_objects': 6008, 'num_object_clones': 0, 'num_object_copies': 24032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6008, 'num_write_kb': 48064, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3789}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:37.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:44:37.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6051", 'reported_seq': 6117, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:35.041518+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:44:35.041518+0000', 'last_peered': '2024-09-16T04:44:35.041518+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:44:35.041518+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6051, 'log_dups_size': 0, 'ondisk_log_size': 6051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49534220, 'num_objects': 6047, 'num_object_clones': 0, 'num_object_copies': 24188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6047, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6049, 'num_write_kb': 48388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3836}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:37.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:44:37.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6042", 'reported_seq': 6117, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:34.627700+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:44:34.627700+0000', 'last_peered': '2024-09-16T04:44:34.627700+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:44:34.627700+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6042, 'log_dups_size': 0, 'ondisk_log_size': 6042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49470086, 'num_objects': 6039, 'num_object_clones': 0, 'num_object_copies': 24156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6041, 'num_write_kb': 48318, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3812}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:37.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:44:37.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6036", 'reported_seq': 6102, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:34.853061+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:44:34.853061+0000', 'last_peered': '2024-09-16T04:44:34.853061+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:44:34.853061+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6036, 'log_dups_size': 0, 'ondisk_log_size': 6036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49411340, 'num_objects': 6032, 'num_object_clones': 0, 'num_object_copies': 24128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6034, 'num_write_kb': 48261, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3803}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:37.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:44:37.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6065", 'reported_seq': 6119, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:34.834422+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:44:34.834422+0000', 'last_peered': '2024-09-16T04:44:34.834422+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:44:34.834422+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6065, 'log_dups_size': 0, 'ondisk_log_size': 6065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49657100, 'num_objects': 6062, 'num_object_clones': 0, 'num_object_copies': 24248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6063, 'num_write_kb': 48501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3779}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:37.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:44:37.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5858", 'reported_seq': 5902, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:34.758834+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:44:34.758834+0000', 'last_peered': '2024-09-16T04:44:34.758834+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:44:34.758834+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5858, 'log_dups_size': 0, 'ondisk_log_size': 5858, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47988736, 'num_objects': 5858, 'num_object_clones': 0, 'num_object_copies': 23432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5858, 'num_write_kb': 46864, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3650}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:37.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:44:37.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6007", 'reported_seq': 6072, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:34.897505+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:44:34.897505+0000', 'last_peered': '2024-09-16T04:44:34.897505+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:44:34.897505+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6007, 'log_dups_size': 0, 'ondisk_log_size': 6007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49157388, 'num_objects': 6001, 'num_object_clones': 0, 'num_object_copies': 24004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6005, 'num_write_kb': 48020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3775}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:37.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:44:37.746 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6056", 'reported_seq': 6482, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:34.862074+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:44:34.862074+0000', 'last_peered': '2024-09-16T04:44:34.862074+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:44:34.862074+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6056, 'log_dups_size': 0, 'ondisk_log_size': 6056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49539608, 'num_objects': 6048, 'num_object_clones': 0, 'num_object_copies': 24192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2155, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6048, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6053, 'num_write_kb': 48401, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 188, 'num_bytes_recovered': 1515520, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3893}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:37.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:44:37.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6033", 'reported_seq': 6112, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:34.686840+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:44:34.686840+0000', 'last_peered': '2024-09-16T04:44:34.686840+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:44:34.686840+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6033, 'log_dups_size': 0, 'ondisk_log_size': 6033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49403148, 'num_objects': 6031, 'num_object_clones': 0, 'num_object_copies': 24124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6031, 'num_write_kb': 48246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3831}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:37.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:44:37.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5931", 'reported_seq': 5989, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:34.846934+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:44:34.846934+0000', 'last_peered': '2024-09-16T04:44:34.846934+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:44:34.846934+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5931, 'log_dups_size': 0, 'ondisk_log_size': 5931, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48541586, 'num_objects': 5926, 'num_object_clones': 0, 'num_object_copies': 23704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5928, 'num_write_kb': 47412, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3737}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:37.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:44:37.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5872", 'reported_seq': 5910, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:34.725379+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:44:34.725379+0000', 'last_peered': '2024-09-16T04:44:34.725379+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:44:34.725379+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5872, 'log_dups_size': 0, 'ondisk_log_size': 5872, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48093830, 'num_objects': 5871, 'num_object_clones': 0, 'num_object_copies': 23484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5871, 'num_write_kb': 46967, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3743}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:37.748 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:44:37.748 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5928", 'reported_seq': 6002, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:34.529155+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:44:34.529155+0000', 'last_peered': '2024-09-16T04:44:34.529155+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:44:34.529155+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5928, 'log_dups_size': 0, 'ondisk_log_size': 5928, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48552582, 'num_objects': 5927, 'num_object_clones': 0, 'num_object_copies': 23708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5927, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5927, 'num_write_kb': 47415, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3702}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:37.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:44:37.750 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6067", 'reported_seq': 6121, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:34.723960+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:44:34.723960+0000', 'last_peered': '2024-09-16T04:44:34.723960+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:44:34.723960+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6067, 'log_dups_size': 0, 'ondisk_log_size': 6067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49665292, 'num_objects': 6063, 'num_object_clones': 0, 'num_object_copies': 24252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6063, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6065, 'num_write_kb': 48516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3829}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:37.750 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:44:37.750 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'5911", 'reported_seq': 5990, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:34.476276+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:44:34.476276+0000', 'last_peered': '2024-09-16T04:44:34.476276+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:44:34.476276+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5911, 'log_dups_size': 0, 'ondisk_log_size': 5911, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48395532, 'num_objects': 5908, 'num_object_clones': 0, 'num_object_copies': 23632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5909, 'num_write_kb': 47269, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3776}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:37.750 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:44:37.750 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5855", 'reported_seq': 5921, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:35.011177+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:44:35.011177+0000', 'last_peered': '2024-09-16T04:44:35.011177+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:44:35.011177+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5855, 'log_dups_size': 0, 'ondisk_log_size': 5855, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47920396, 'num_objects': 5850, 'num_object_clones': 0, 'num_object_copies': 23400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5853, 'num_write_kb': 46812, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3611}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:37.750 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:44:38.317 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 224 20 96595 96575 3.36715 1.92188 0.0653173 0.0463935 2024-09-16T04:44:38.432 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:38.432 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33127/382720 objects degraded (8.656%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":290,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":95680,"data_bytes":783771304,"bytes_used":2716684288,"bytes_avail":383830372352,"bytes_total":386547056640,"degraded_objects":33127,"degraded_total":382720,"degraded_ratio":0.086556751672240806,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":77122,"recovering_keys_per_sec":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"write_bytes_sec":2663780,"read_op_per_sec":0,"write_op_per_sec":325},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:44:38.455 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:44:38.455 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:39.044 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:39.044 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:39.067 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-09-16T04:44:38.624558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":784426664,"num_objects":95760,"num_object_clones":0,"num_object_copies":383040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95787,"num_write_kb":766178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2452,"num_bytes_recovered":19849216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95814,"ondisk_log_size":95814,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2645864,"kb_used_data":1400680,"kb_used_omap":14,"kb_used_meta":1245105,"kb_avail":374841496,"statfs":{"total":386547056640,"available":383837691904,"internally_reserved":0,"allocated":1434296320,"data_stored":1433576444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1274988081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,5,4,14,18,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22282240,"num_objects":2720,"num_object_clones":0,"num_object_copies":10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-70,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":21760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":573440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003035"},"pg_stats":[{"pgid":"1.f","version":"33'6008","reported_seq":6052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.791600+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:34.791600+0000","last_peered":"2024-09-16T04:44:34.791600+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:34.791600+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6008,"log_dups_size":0,"ondisk_log_size":6008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49217536,"num_objects":6008,"num_object_clones":0,"num_object_copies":24032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6008,"num_write_kb":48064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6051","reported_seq":6117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:35.041518+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:35.041518+0000","last_peered":"2024-09-16T04:44:35.041518+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:35.041518+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6051,"log_dups_size":0,"ondisk_log_size":6051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49534220,"num_objects":6047,"num_object_clones":0,"num_object_copies":24188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":48388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3836}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6042","reported_seq":6117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.627700+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:34.627700+0000","last_peered":"2024-09-16T04:44:34.627700+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:34.627700+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6042,"log_dups_size":0,"ondisk_log_size":6042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49470086,"num_objects":6039,"num_object_clones":0,"num_object_copies":24156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6041,"num_write_kb":48318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3812}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6036","reported_seq":6102,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.853061+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:34.853061+0000","last_peered":"2024-09-16T04:44:34.853061+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:34.853061+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6036,"log_dups_size":0,"ondisk_log_size":6036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49411340,"num_objects":6032,"num_object_clones":0,"num_object_copies":24128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6034,"num_write_kb":48261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6065","reported_seq":6119,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.834422+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:34.834422+0000","last_peered":"2024-09-16T04:44:34.834422+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:34.834422+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6065,"log_dups_size":0,"ondisk_log_size":6065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49657100,"num_objects":6062,"num_object_clones":0,"num_object_copies":24248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6063,"num_write_kb":48501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5858","reported_seq":5902,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.758834+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:34.758834+0000","last_peered":"2024-09-16T04:44:34.758834+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:34.758834+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47988736,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5858,"num_write_kb":46864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6007","reported_seq":6072,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.897505+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:34.897505+0000","last_peered":"2024-09-16T04:44:34.897505+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:34.897505+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6007,"log_dups_size":0,"ondisk_log_size":6007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49157388,"num_objects":6001,"num_object_clones":0,"num_object_copies":24004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6005,"num_write_kb":48020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3775}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6094","reported_seq":10650,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:37.054041+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:37.054041+0000","last_peered":"2024-09-16T04:44:37.054041+0000","last_clean":"2024-09-16T04:44:37.054041+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:37.054041+0000","last_undegraded":"2024-09-16T04:44:37.054041+0000","last_fullsized":"2024-09-16T04:44:37.054041+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6094,"log_dups_size":0,"ondisk_log_size":6094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:39.941704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49878284,"num_objects":6089,"num_object_clones":0,"num_object_copies":24356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6092,"num_write_kb":48724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6056","reported_seq":6482,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.862074+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:34.862074+0000","last_peered":"2024-09-16T04:44:34.862074+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:34.862074+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6056,"log_dups_size":0,"ondisk_log_size":6056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49539608,"num_objects":6048,"num_object_clones":0,"num_object_copies":24192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6053,"num_write_kb":48401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":1515520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3893}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6033","reported_seq":6112,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.686840+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:34.686840+0000","last_peered":"2024-09-16T04:44:34.686840+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:34.686840+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49403148,"num_objects":6031,"num_object_clones":0,"num_object_copies":24124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6031,"num_write_kb":48246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3831}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5931","reported_seq":5989,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.846934+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:34.846934+0000","last_peered":"2024-09-16T04:44:34.846934+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:34.846934+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5931,"log_dups_size":0,"ondisk_log_size":5931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48541586,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5928,"num_write_kb":47412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3737}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5872","reported_seq":5910,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.725379+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:34.725379+0000","last_peered":"2024-09-16T04:44:34.725379+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:34.725379+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5872,"log_dups_size":0,"ondisk_log_size":5872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48093830,"num_objects":5871,"num_object_clones":0,"num_object_copies":23484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5871,"num_write_kb":46967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3743}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5928","reported_seq":6002,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.529155+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:34.529155+0000","last_peered":"2024-09-16T04:44:34.529155+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:34.529155+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48552582,"num_objects":5927,"num_object_clones":0,"num_object_copies":23708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5927,"num_write_kb":47415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3702}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6067","reported_seq":6121,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.723960+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:34.723960+0000","last_peered":"2024-09-16T04:44:34.723960+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:34.723960+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6067,"log_dups_size":0,"ondisk_log_size":6067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49665292,"num_objects":6063,"num_object_clones":0,"num_object_copies":24252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6065,"num_write_kb":48516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3829}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5911","reported_seq":5990,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:34.476276+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:34.476276+0000","last_peered":"2024-09-16T04:44:34.476276+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:34.476276+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5911,"log_dups_size":0,"ondisk_log_size":5911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48395532,"num_objects":5908,"num_object_clones":0,"num_object_copies":23632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5909,"num_write_kb":47269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3776}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5855","reported_seq":5921,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:35.011177+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:35.011177+0000","last_peered":"2024-09-16T04:44:35.011177+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:35.011177+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5855,"log_dups_size":0,"ondisk_log_size":5855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47920396,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5853,"num_write_kb":46812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3611}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":784426664,"num_objects":95760,"num_object_clones":0,"num_object_copies":383040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95787,"num_write_kb":766178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2452,"num_bytes_recovered":19849216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1435074560,"data_stored":1435074560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95814,"ondisk_log_size":95814,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724896,"kb_used_data":382224,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93646944,"statfs":{"total":96636764160,"available":95894470656,"internally_reserved":0,"allocated":391397376,"data_stored":391217407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":350875343},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,8,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724960,"kb_used_data":382800,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93646880,"statfs":{"total":96636764160,"available":95894405120,"internally_reserved":0,"allocated":391987200,"data_stored":391807231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,0,1,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051623,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479024,"kb_used_data":252960,"kb_used_omap":4,"kb_used_meta":226043,"kb_avail":93892816,"statfs":{"total":96636764160,"available":96146243584,"internally_reserved":0,"allocated":259031040,"data_stored":258851071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":231468751},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,4,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716984,"kb_used_data":382696,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93654856,"statfs":{"total":96636764160,"available":95902572544,"internally_reserved":0,"allocated":391880704,"data_stored":391700735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":342290126},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":392241152,"data_stored":392241152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259022848,"data_stored":259022848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391802880,"data_stored":391802880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":392007680,"data_stored":392007680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:39.317 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 225 20 96841 96821 3.36073 1.92188 0.0634274 0.0464787 2024-09-16T04:44:40.317 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 226 20 97117 97097 3.3554 2.15625 0.135232 0.0465526 2024-09-16T04:44:41.317 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 227 20 97382 97362 3.34974 2.07031 0.0724022 0.0466342 2024-09-16T04:44:42.069 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:42.317 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 228 20 97647 97627 3.34413 2.07031 0.0773865 0.0467095 2024-09-16T04:44:42.623 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:42.624 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:42.651 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-09-16T04:44:40.625112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":794379944,"num_objects":96975,"num_object_clones":0,"num_object_copies":387900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97002,"num_write_kb":775898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2490,"num_bytes_recovered":20160512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97029,"ondisk_log_size":97029,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2687676,"kb_used_data":1416124,"kb_used_omap":14,"kb_used_meta":1271473,"kb_avail":374799684,"statfs":{"total":386547056640,"available":383794876416,"internally_reserved":0,"allocated":1450110976,"data_stored":1449391100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1301988913},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,3,1,3,13,16,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31440896,"num_objects":3838,"num_object_clones":0,"num_object_copies":15352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3838,"num_write_kb":30704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003055"},"pg_stats":[{"pgid":"1.f","version":"33'6080","reported_seq":6124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.631782+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:39.631782+0000","last_peered":"2024-09-16T04:44:39.631782+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:39.631782+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6080,"log_dups_size":0,"ondisk_log_size":6080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49807360,"num_objects":6080,"num_object_clones":0,"num_object_copies":24320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6080,"num_write_kb":48640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6125","reported_seq":6191,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.937619+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:39.937619+0000","last_peered":"2024-09-16T04:44:39.937619+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:39.937619+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6125,"log_dups_size":0,"ondisk_log_size":6125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50140428,"num_objects":6121,"num_object_clones":0,"num_object_copies":24484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6123,"num_write_kb":48980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3910}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6107","reported_seq":6182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.712418+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:39.712418+0000","last_peered":"2024-09-16T04:44:39.712418+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:39.712418+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6107,"log_dups_size":0,"ondisk_log_size":6107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50002566,"num_objects":6104,"num_object_clones":0,"num_object_copies":24416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6106,"num_write_kb":48838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3877}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6115","reported_seq":6182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:40.061906+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:40.061906+0000","last_peered":"2024-09-16T04:44:40.061906+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:40.061906+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6115,"log_dups_size":0,"ondisk_log_size":6115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50058508,"num_objects":6111,"num_object_clones":0,"num_object_copies":24444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6113,"num_write_kb":48893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3882}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6149","reported_seq":6205,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.737581+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:39.737581+0000","last_peered":"2024-09-16T04:44:39.737581+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:39.737581+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6149,"log_dups_size":0,"ondisk_log_size":6149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50345228,"num_objects":6146,"num_object_clones":0,"num_object_copies":24584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6147,"num_write_kb":49173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3863}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5942","reported_seq":5986,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.894096+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:39.894096+0000","last_peered":"2024-09-16T04:44:39.894096+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:39.894096+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5942,"log_dups_size":0,"ondisk_log_size":5942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48676864,"num_objects":5942,"num_object_clones":0,"num_object_copies":23768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5942,"num_write_kb":47536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3734}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6079","reported_seq":6144,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:40.122967+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:40.122967+0000","last_peered":"2024-09-16T04:44:40.122967+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:40.122967+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6079,"log_dups_size":0,"ondisk_log_size":6079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49747212,"num_objects":6073,"num_object_clones":0,"num_object_copies":24292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6077,"num_write_kb":48596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3847}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6094","reported_seq":10650,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:37.054041+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:37.054041+0000","last_peered":"2024-09-16T04:44:37.054041+0000","last_clean":"2024-09-16T04:44:37.054041+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:37.054041+0000","last_undegraded":"2024-09-16T04:44:37.054041+0000","last_fullsized":"2024-09-16T04:44:37.054041+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6094,"log_dups_size":0,"ondisk_log_size":6094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:39.941704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49878284,"num_objects":6089,"num_object_clones":0,"num_object_copies":24356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6092,"num_write_kb":48724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6140","reported_seq":6644,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.891140+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:39.891140+0000","last_peered":"2024-09-16T04:44:39.891140+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:39.891140+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6140,"log_dups_size":0,"ondisk_log_size":6140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50227736,"num_objects":6132,"num_object_clones":0,"num_object_copies":24528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6137,"num_write_kb":49073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":1826816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4015}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6131","reported_seq":6210,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.714221+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:39.714221+0000","last_peered":"2024-09-16T04:44:39.714221+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:39.714221+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6131,"log_dups_size":0,"ondisk_log_size":6131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50205964,"num_objects":6129,"num_object_clones":0,"num_object_copies":24516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6129,"num_write_kb":49030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3929}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6009","reported_seq":6067,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.777141+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:39.777141+0000","last_peered":"2024-09-16T04:44:39.777141+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:39.777141+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49180562,"num_objects":6004,"num_object_clones":0,"num_object_copies":24016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6006,"num_write_kb":48036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3815}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5961","reported_seq":6001,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.908155+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:39.908155+0000","last_peered":"2024-09-16T04:44:39.908155+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:39.908155+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5961,"log_dups_size":0,"ondisk_log_size":5961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48822918,"num_objects":5960,"num_object_clones":0,"num_object_copies":23840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5960,"num_write_kb":47679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6016","reported_seq":6090,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.670930+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:39.670930+0000","last_peered":"2024-09-16T04:44:39.670930+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:39.670930+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6016,"log_dups_size":0,"ondisk_log_size":6016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49273478,"num_objects":6015,"num_object_clones":0,"num_object_copies":24060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6015,"num_write_kb":48119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3790}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6146","reported_seq":6200,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.799393+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:39.799393+0000","last_peered":"2024-09-16T04:44:39.799393+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:39.799393+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6146,"log_dups_size":0,"ondisk_log_size":6146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50312460,"num_objects":6142,"num_object_clones":0,"num_object_copies":24568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6144,"num_write_kb":49148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3908}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5997","reported_seq":6075,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.725548+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:39.725548+0000","last_peered":"2024-09-16T04:44:39.725548+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:39.725548+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49100044,"num_objects":5994,"num_object_clones":0,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5995,"num_write_kb":47957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3862}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5938","reported_seq":6004,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:40.073718+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:40.073718+0000","last_peered":"2024-09-16T04:44:40.073718+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:40.073718+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5938,"log_dups_size":0,"ondisk_log_size":5938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48600332,"num_objects":5933,"num_object_clones":0,"num_object_copies":23732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5936,"num_write_kb":47476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":794379944,"num_objects":96975,"num_object_clones":0,"num_object_copies":387900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97002,"num_write_kb":775898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2490,"num_bytes_recovered":20160512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1451143168,"data_stored":1451143168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97029,"ondisk_log_size":97029,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734188,"kb_used_data":387356,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93637652,"statfs":{"total":96636764160,"available":95884955648,"internally_reserved":0,"allocated":396652544,"data_stored":396472575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355135183},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,0,0,0,4,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748196,"kb_used_data":387988,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93623644,"statfs":{"total":96636764160,"available":95870611456,"internally_reserved":0,"allocated":397299712,"data_stored":397119743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051623,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479024,"kb_used_data":252960,"kb_used_omap":4,"kb_used_meta":226043,"kb_avail":93892816,"statfs":{"total":96636764160,"available":96146243584,"internally_reserved":0,"allocated":259031040,"data_stored":258851071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":231468751},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,4,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726268,"kb_used_data":387820,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93645572,"statfs":{"total":96636764160,"available":95893065728,"internally_reserved":0,"allocated":397127680,"data_stored":396947711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":346549966},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":397578240,"data_stored":397578240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259022848,"data_stored":259022848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397156352,"data_stored":397156352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":397385728,"data_stored":397385728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:42.653 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:43.233 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:43.234 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:43.264 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-16T04:44:42.625644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":795035304,"num_objects":97055,"num_object_clones":0,"num_object_copies":388220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97082,"num_write_kb":776538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2490,"num_bytes_recovered":20160512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97109,"ondisk_log_size":97109,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2697212,"kb_used_data":1421500,"kb_used_omap":14,"kb_used_meta":1275633,"kb_avail":374790148,"statfs":{"total":386547056640,"available":383785111552,"internally_reserved":0,"allocated":1455616000,"data_stored":1454896124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1306248753},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,3,0,3,13,19,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21774336,"num_objects":2658,"num_object_clones":0,"num_object_copies":10632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-72,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2658,"num_write_kb":21264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003092"},"pg_stats":[{"pgid":"1.f","version":"33'6080","reported_seq":6124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.631782+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:39.631782+0000","last_peered":"2024-09-16T04:44:39.631782+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:39.631782+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6080,"log_dups_size":0,"ondisk_log_size":6080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49807360,"num_objects":6080,"num_object_clones":0,"num_object_copies":24320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6080,"num_write_kb":48640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6125","reported_seq":6191,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.937619+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:39.937619+0000","last_peered":"2024-09-16T04:44:39.937619+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:39.937619+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6125,"log_dups_size":0,"ondisk_log_size":6125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50140428,"num_objects":6121,"num_object_clones":0,"num_object_copies":24484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6123,"num_write_kb":48980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3910}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6107","reported_seq":6182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.712418+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:39.712418+0000","last_peered":"2024-09-16T04:44:39.712418+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:39.712418+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6107,"log_dups_size":0,"ondisk_log_size":6107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50002566,"num_objects":6104,"num_object_clones":0,"num_object_copies":24416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6106,"num_write_kb":48838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3877}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6115","reported_seq":6182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:40.061906+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:40.061906+0000","last_peered":"2024-09-16T04:44:40.061906+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:40.061906+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6115,"log_dups_size":0,"ondisk_log_size":6115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50058508,"num_objects":6111,"num_object_clones":0,"num_object_copies":24444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6113,"num_write_kb":48893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3882}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6149","reported_seq":6205,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.737581+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:39.737581+0000","last_peered":"2024-09-16T04:44:39.737581+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:39.737581+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6149,"log_dups_size":0,"ondisk_log_size":6149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50345228,"num_objects":6146,"num_object_clones":0,"num_object_copies":24584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6147,"num_write_kb":49173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3863}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5942","reported_seq":5986,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.894096+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:39.894096+0000","last_peered":"2024-09-16T04:44:39.894096+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:39.894096+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5942,"log_dups_size":0,"ondisk_log_size":5942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48676864,"num_objects":5942,"num_object_clones":0,"num_object_copies":23768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5942,"num_write_kb":47536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3734}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6079","reported_seq":6144,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:40.122967+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:40.122967+0000","last_peered":"2024-09-16T04:44:40.122967+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:40.122967+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6079,"log_dups_size":0,"ondisk_log_size":6079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49747212,"num_objects":6073,"num_object_clones":0,"num_object_copies":24292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6077,"num_write_kb":48596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3847}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6174","reported_seq":10730,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:42.026527+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:42.026527+0000","last_peered":"2024-09-16T04:44:42.026527+0000","last_clean":"2024-09-16T04:44:42.026527+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:42.026527+0000","last_undegraded":"2024-09-16T04:44:42.026527+0000","last_fullsized":"2024-09-16T04:44:42.026527+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6174,"log_dups_size":0,"ondisk_log_size":6174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:45.867845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50533644,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6172,"num_write_kb":49364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6140","reported_seq":6644,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.891140+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:39.891140+0000","last_peered":"2024-09-16T04:44:39.891140+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:39.891140+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6140,"log_dups_size":0,"ondisk_log_size":6140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50227736,"num_objects":6132,"num_object_clones":0,"num_object_copies":24528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6137,"num_write_kb":49073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":1826816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4015}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6131","reported_seq":6210,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.714221+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:39.714221+0000","last_peered":"2024-09-16T04:44:39.714221+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:39.714221+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6131,"log_dups_size":0,"ondisk_log_size":6131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50205964,"num_objects":6129,"num_object_clones":0,"num_object_copies":24516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6129,"num_write_kb":49030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3929}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6009","reported_seq":6067,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.777141+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:39.777141+0000","last_peered":"2024-09-16T04:44:39.777141+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:39.777141+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49180562,"num_objects":6004,"num_object_clones":0,"num_object_copies":24016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6006,"num_write_kb":48036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3815}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5961","reported_seq":6001,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.908155+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:39.908155+0000","last_peered":"2024-09-16T04:44:39.908155+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:39.908155+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5961,"log_dups_size":0,"ondisk_log_size":5961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48822918,"num_objects":5960,"num_object_clones":0,"num_object_copies":23840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5960,"num_write_kb":47679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6016","reported_seq":6090,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.670930+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:39.670930+0000","last_peered":"2024-09-16T04:44:39.670930+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:39.670930+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6016,"log_dups_size":0,"ondisk_log_size":6016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49273478,"num_objects":6015,"num_object_clones":0,"num_object_copies":24060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6015,"num_write_kb":48119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3790}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6146","reported_seq":6200,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.799393+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:39.799393+0000","last_peered":"2024-09-16T04:44:39.799393+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:39.799393+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6146,"log_dups_size":0,"ondisk_log_size":6146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50312460,"num_objects":6142,"num_object_clones":0,"num_object_copies":24568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6144,"num_write_kb":49148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3908}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5997","reported_seq":6075,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.725548+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:39.725548+0000","last_peered":"2024-09-16T04:44:39.725548+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:39.725548+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49100044,"num_objects":5994,"num_object_clones":0,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5995,"num_write_kb":47957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3862}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5938","reported_seq":6004,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:40.073718+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:40.073718+0000","last_peered":"2024-09-16T04:44:40.073718+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:40.073718+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5938,"log_dups_size":0,"ondisk_log_size":5938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48600332,"num_objects":5933,"num_object_clones":0,"num_object_copies":23732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5936,"num_write_kb":47476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":795035304,"num_objects":97055,"num_object_clones":0,"num_object_copies":388220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97082,"num_write_kb":776538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2490,"num_bytes_recovered":20160512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1456685056,"data_stored":1456685056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97109,"ondisk_log_size":97109,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734188,"kb_used_data":387356,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93637652,"statfs":{"total":96636764160,"available":95884955648,"internally_reserved":0,"allocated":396652544,"data_stored":396472575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355135183},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,0,0,0,4,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748196,"kb_used_data":387988,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93623644,"statfs":{"total":96636764160,"available":95870611456,"internally_reserved":0,"allocated":397299712,"data_stored":397119743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051624,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488560,"kb_used_data":258336,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93883280,"statfs":{"total":96636764160,"available":96136478720,"internally_reserved":0,"allocated":264536064,"data_stored":264356095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235728591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726268,"kb_used_data":387820,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93645572,"statfs":{"total":96636764160,"available":95893065728,"internally_reserved":0,"allocated":397127680,"data_stored":396947711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":346549966},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":397578240,"data_stored":397578240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264564736,"data_stored":264564736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397156352,"data_stored":397156352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":397385728,"data_stored":397385728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:43.265 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:44:43.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6080", 'reported_seq': 6124, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:39.631782+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:44:39.631782+0000', 'last_peered': '2024-09-16T04:44:39.631782+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:44:39.631782+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6080, 'log_dups_size': 0, 'ondisk_log_size': 6080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49807360, 'num_objects': 6080, 'num_object_clones': 0, 'num_object_copies': 24320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6080, 'num_write_kb': 48640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3861}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:43.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:44:43.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6125", 'reported_seq': 6191, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:39.937619+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:44:39.937619+0000', 'last_peered': '2024-09-16T04:44:39.937619+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:44:39.937619+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6125, 'log_dups_size': 0, 'ondisk_log_size': 6125, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50140428, 'num_objects': 6121, 'num_object_clones': 0, 'num_object_copies': 24484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6121, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6123, 'num_write_kb': 48980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:43.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:44:43.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6107", 'reported_seq': 6182, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:39.712418+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:44:39.712418+0000', 'last_peered': '2024-09-16T04:44:39.712418+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:44:39.712418+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6107, 'log_dups_size': 0, 'ondisk_log_size': 6107, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50002566, 'num_objects': 6104, 'num_object_clones': 0, 'num_object_copies': 24416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6104, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6106, 'num_write_kb': 48838, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3877}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:43.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:44:43.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6115", 'reported_seq': 6182, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:40.061906+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:44:40.061906+0000', 'last_peered': '2024-09-16T04:44:40.061906+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:44:40.061906+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6115, 'log_dups_size': 0, 'ondisk_log_size': 6115, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50058508, 'num_objects': 6111, 'num_object_clones': 0, 'num_object_copies': 24444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6111, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6113, 'num_write_kb': 48893, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3882}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:43.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:44:43.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6149", 'reported_seq': 6205, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:39.737581+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:44:39.737581+0000', 'last_peered': '2024-09-16T04:44:39.737581+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:44:39.737581+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6149, 'log_dups_size': 0, 'ondisk_log_size': 6149, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50345228, 'num_objects': 6146, 'num_object_clones': 0, 'num_object_copies': 24584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6147, 'num_write_kb': 49173, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3863}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:43.266 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:44:43.266 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5942", 'reported_seq': 5986, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:39.894096+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:44:39.894096+0000', 'last_peered': '2024-09-16T04:44:39.894096+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:44:39.894096+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5942, 'log_dups_size': 0, 'ondisk_log_size': 5942, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48676864, 'num_objects': 5942, 'num_object_clones': 0, 'num_object_copies': 23768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5942, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5942, 'num_write_kb': 47536, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3734}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:43.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:44:43.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6079", 'reported_seq': 6144, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:40.122967+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:44:40.122967+0000', 'last_peered': '2024-09-16T04:44:40.122967+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:44:40.122967+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6079, 'log_dups_size': 0, 'ondisk_log_size': 6079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49747212, 'num_objects': 6073, 'num_object_clones': 0, 'num_object_copies': 24292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6073, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6077, 'num_write_kb': 48596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3847}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:43.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:44:43.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6140", 'reported_seq': 6644, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:39.891140+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:44:39.891140+0000', 'last_peered': '2024-09-16T04:44:39.891140+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:44:39.891140+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6140, 'log_dups_size': 0, 'ondisk_log_size': 6140, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50227736, 'num_objects': 6132, 'num_object_clones': 0, 'num_object_copies': 24528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2117, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6132, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6137, 'num_write_kb': 49073, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 226, 'num_bytes_recovered': 1826816, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4015}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:43.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:44:43.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6131", 'reported_seq': 6210, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:39.714221+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:44:39.714221+0000', 'last_peered': '2024-09-16T04:44:39.714221+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:44:39.714221+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6131, 'log_dups_size': 0, 'ondisk_log_size': 6131, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50205964, 'num_objects': 6129, 'num_object_clones': 0, 'num_object_copies': 24516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6129, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6129, 'num_write_kb': 49030, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3929}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:43.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:44:43.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6009", 'reported_seq': 6067, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:39.777141+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:44:39.777141+0000', 'last_peered': '2024-09-16T04:44:39.777141+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:44:39.777141+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6009, 'log_dups_size': 0, 'ondisk_log_size': 6009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49180562, 'num_objects': 6004, 'num_object_clones': 0, 'num_object_copies': 24016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6004, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6006, 'num_write_kb': 48036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3815}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:43.267 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:44:43.267 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5961", 'reported_seq': 6001, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:39.908155+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:44:39.908155+0000', 'last_peered': '2024-09-16T04:44:39.908155+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:44:39.908155+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5961, 'log_dups_size': 0, 'ondisk_log_size': 5961, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48822918, 'num_objects': 5960, 'num_object_clones': 0, 'num_object_copies': 23840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5960, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5960, 'num_write_kb': 47679, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3832}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:43.268 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:44:43.268 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6016", 'reported_seq': 6090, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:39.670930+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:44:39.670930+0000', 'last_peered': '2024-09-16T04:44:39.670930+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:44:39.670930+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6016, 'log_dups_size': 0, 'ondisk_log_size': 6016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49273478, 'num_objects': 6015, 'num_object_clones': 0, 'num_object_copies': 24060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6015, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6015, 'num_write_kb': 48119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3790}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:43.268 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:44:43.268 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6146", 'reported_seq': 6200, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:39.799393+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:44:39.799393+0000', 'last_peered': '2024-09-16T04:44:39.799393+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:44:39.799393+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6146, 'log_dups_size': 0, 'ondisk_log_size': 6146, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50312460, 'num_objects': 6142, 'num_object_clones': 0, 'num_object_copies': 24568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6144, 'num_write_kb': 49148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3908}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:43.268 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:44:43.268 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'5997", 'reported_seq': 6075, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:39.725548+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:44:39.725548+0000', 'last_peered': '2024-09-16T04:44:39.725548+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:44:39.725548+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5997, 'log_dups_size': 0, 'ondisk_log_size': 5997, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49100044, 'num_objects': 5994, 'num_object_clones': 0, 'num_object_copies': 23976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5995, 'num_write_kb': 47957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3862}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:43.268 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:44:43.268 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5938", 'reported_seq': 6004, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:40.073718+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:44:40.073718+0000', 'last_peered': '2024-09-16T04:44:40.073718+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:44:40.073718+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 5938, 'log_dups_size': 0, 'ondisk_log_size': 5938, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48600332, 'num_objects': 5933, 'num_object_clones': 0, 'num_object_copies': 23732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5933, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5936, 'num_write_kb': 47476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3694}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:43.268 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:44:43.317 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 229 20 97885 97865 3.33764 1.85938 0.0613319 0.0467964 2024-09-16T04:44:43.960 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:43.960 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33089/388220 objects degraded (8.523%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":296,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":97055,"data_bytes":795035304,"bytes_used":2761945088,"bytes_avail":383785111552,"bytes_total":386547056640,"degraded_objects":33089,"degraded_total":388220,"degraded_ratio":0.085232600072124048,"recovering_objects_per_sec":5,"recovering_bytes_per_sec":49139,"recovering_keys_per_sec":0,"num_objects_recovered":72,"num_bytes_recovered":589824,"num_keys_recovered":0,"write_bytes_sec":1814060,"read_op_per_sec":0,"write_op_per_sec":221},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:44:43.994 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:44:43.994 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:44.317 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 230 20 98119 98099 3.33108 1.82812 0.0151887 0.0468833 2024-09-16T04:44:44.559 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:44.560 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:44.577 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-16T04:44:42.625644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":795035304,"num_objects":97055,"num_object_clones":0,"num_object_copies":388220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97082,"num_write_kb":776538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2490,"num_bytes_recovered":20160512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97109,"ondisk_log_size":97109,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2697212,"kb_used_data":1421500,"kb_used_omap":14,"kb_used_meta":1275633,"kb_avail":374790148,"statfs":{"total":386547056640,"available":383785111552,"internally_reserved":0,"allocated":1455616000,"data_stored":1454896124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1306248753},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,3,0,3,13,19,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21774336,"num_objects":2658,"num_object_clones":0,"num_object_copies":10632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-72,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2658,"num_write_kb":21264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003092"},"pg_stats":[{"pgid":"1.f","version":"33'6080","reported_seq":6124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.631782+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:39.631782+0000","last_peered":"2024-09-16T04:44:39.631782+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:39.631782+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6080,"log_dups_size":0,"ondisk_log_size":6080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49807360,"num_objects":6080,"num_object_clones":0,"num_object_copies":24320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6080,"num_write_kb":48640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6125","reported_seq":6191,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.937619+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:39.937619+0000","last_peered":"2024-09-16T04:44:39.937619+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:39.937619+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6125,"log_dups_size":0,"ondisk_log_size":6125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50140428,"num_objects":6121,"num_object_clones":0,"num_object_copies":24484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6123,"num_write_kb":48980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3910}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6107","reported_seq":6182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.712418+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:39.712418+0000","last_peered":"2024-09-16T04:44:39.712418+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:39.712418+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6107,"log_dups_size":0,"ondisk_log_size":6107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50002566,"num_objects":6104,"num_object_clones":0,"num_object_copies":24416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6106,"num_write_kb":48838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3877}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6115","reported_seq":6182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:40.061906+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:40.061906+0000","last_peered":"2024-09-16T04:44:40.061906+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:40.061906+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6115,"log_dups_size":0,"ondisk_log_size":6115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50058508,"num_objects":6111,"num_object_clones":0,"num_object_copies":24444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6113,"num_write_kb":48893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3882}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6149","reported_seq":6205,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.737581+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:39.737581+0000","last_peered":"2024-09-16T04:44:39.737581+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:39.737581+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6149,"log_dups_size":0,"ondisk_log_size":6149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50345228,"num_objects":6146,"num_object_clones":0,"num_object_copies":24584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6147,"num_write_kb":49173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3863}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5942","reported_seq":5986,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.894096+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:39.894096+0000","last_peered":"2024-09-16T04:44:39.894096+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:39.894096+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5942,"log_dups_size":0,"ondisk_log_size":5942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48676864,"num_objects":5942,"num_object_clones":0,"num_object_copies":23768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5942,"num_write_kb":47536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3734}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6079","reported_seq":6144,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:40.122967+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:40.122967+0000","last_peered":"2024-09-16T04:44:40.122967+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:40.122967+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6079,"log_dups_size":0,"ondisk_log_size":6079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49747212,"num_objects":6073,"num_object_clones":0,"num_object_copies":24292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6077,"num_write_kb":48596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3847}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6174","reported_seq":10730,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:42.026527+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:42.026527+0000","last_peered":"2024-09-16T04:44:42.026527+0000","last_clean":"2024-09-16T04:44:42.026527+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:42.026527+0000","last_undegraded":"2024-09-16T04:44:42.026527+0000","last_fullsized":"2024-09-16T04:44:42.026527+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6174,"log_dups_size":0,"ondisk_log_size":6174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:45.867845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50533644,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6172,"num_write_kb":49364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6140","reported_seq":6644,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.891140+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:39.891140+0000","last_peered":"2024-09-16T04:44:39.891140+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:39.891140+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6140,"log_dups_size":0,"ondisk_log_size":6140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50227736,"num_objects":6132,"num_object_clones":0,"num_object_copies":24528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6137,"num_write_kb":49073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":1826816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4015}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6131","reported_seq":6210,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.714221+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:39.714221+0000","last_peered":"2024-09-16T04:44:39.714221+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:39.714221+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6131,"log_dups_size":0,"ondisk_log_size":6131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50205964,"num_objects":6129,"num_object_clones":0,"num_object_copies":24516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6129,"num_write_kb":49030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3929}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6009","reported_seq":6067,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.777141+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:39.777141+0000","last_peered":"2024-09-16T04:44:39.777141+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:39.777141+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49180562,"num_objects":6004,"num_object_clones":0,"num_object_copies":24016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6006,"num_write_kb":48036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3815}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5961","reported_seq":6001,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.908155+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:39.908155+0000","last_peered":"2024-09-16T04:44:39.908155+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:39.908155+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5961,"log_dups_size":0,"ondisk_log_size":5961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48822918,"num_objects":5960,"num_object_clones":0,"num_object_copies":23840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5960,"num_write_kb":47679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6016","reported_seq":6090,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.670930+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:39.670930+0000","last_peered":"2024-09-16T04:44:39.670930+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:39.670930+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6016,"log_dups_size":0,"ondisk_log_size":6016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49273478,"num_objects":6015,"num_object_clones":0,"num_object_copies":24060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6015,"num_write_kb":48119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3790}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6146","reported_seq":6200,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.799393+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:39.799393+0000","last_peered":"2024-09-16T04:44:39.799393+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:39.799393+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6146,"log_dups_size":0,"ondisk_log_size":6146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50312460,"num_objects":6142,"num_object_clones":0,"num_object_copies":24568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6144,"num_write_kb":49148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3908}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5997","reported_seq":6075,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:39.725548+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:39.725548+0000","last_peered":"2024-09-16T04:44:39.725548+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:39.725548+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49100044,"num_objects":5994,"num_object_clones":0,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5995,"num_write_kb":47957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3862}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5938","reported_seq":6004,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:40.073718+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:40.073718+0000","last_peered":"2024-09-16T04:44:40.073718+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:40.073718+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":5938,"log_dups_size":0,"ondisk_log_size":5938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48600332,"num_objects":5933,"num_object_clones":0,"num_object_copies":23732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5936,"num_write_kb":47476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":795035304,"num_objects":97055,"num_object_clones":0,"num_object_copies":388220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97082,"num_write_kb":776538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2490,"num_bytes_recovered":20160512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1456685056,"data_stored":1456685056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97109,"ondisk_log_size":97109,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734188,"kb_used_data":387356,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93637652,"statfs":{"total":96636764160,"available":95884955648,"internally_reserved":0,"allocated":396652544,"data_stored":396472575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355135183},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,0,0,0,4,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748196,"kb_used_data":387988,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93623644,"statfs":{"total":96636764160,"available":95870611456,"internally_reserved":0,"allocated":397299712,"data_stored":397119743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051624,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488560,"kb_used_data":258336,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93883280,"statfs":{"total":96636764160,"available":96136478720,"internally_reserved":0,"allocated":264536064,"data_stored":264356095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235728591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726268,"kb_used_data":387820,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93645572,"statfs":{"total":96636764160,"available":95893065728,"internally_reserved":0,"allocated":397127680,"data_stored":396947711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":346549966},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":397578240,"data_stored":397578240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264564736,"data_stored":264564736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397156352,"data_stored":397156352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":397385728,"data_stored":397385728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:45.317 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 231 20 98382 98362 3.32555 2.05469 0.0867466 0.0469748 2024-09-16T04:44:46.320 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 232 20 98660 98640 3.32058 2.17188 0.0678447 0.0470416 2024-09-16T04:44:47.317 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 233 20 98934 98914 3.31552 2.14062 0.0496233 0.0471138 2024-09-16T04:44:47.578 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:48.189 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:48.190 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:48.215 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-09-16T04:44:46.626684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":804890280,"num_objects":98258,"num_object_clones":0,"num_object_copies":393032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98285,"num_write_kb":786162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2530,"num_bytes_recovered":20488192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98312,"ondisk_log_size":98312,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745288,"kb_used_data":1437640,"kb_used_omap":14,"kb_used_meta":1307569,"kb_avail":374742072,"statfs":{"total":386547056640,"available":383735881728,"internally_reserved":0,"allocated":1472143360,"data_stored":1471423484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1338951217},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,19,6,9,8,22,15,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31162368,"num_objects":3804,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3804,"num_write_kb":30432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003107"},"pg_stats":[{"pgid":"1.f","version":"33'6163","reported_seq":6207,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.840645+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:44.840645+0000","last_peered":"2024-09-16T04:44:44.840645+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:44.840645+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6163,"log_dups_size":0,"ondisk_log_size":6163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50487296,"num_objects":6163,"num_object_clones":0,"num_object_copies":24652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":49304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6197","reported_seq":6263,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:45.140980+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:45.140980+0000","last_peered":"2024-09-16T04:44:45.140980+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:45.140980+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6197,"log_dups_size":0,"ondisk_log_size":6197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50730252,"num_objects":6193,"num_object_clones":0,"num_object_copies":24772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6195,"num_write_kb":49556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3982}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6186","reported_seq":6260,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.643410+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:44.643410+0000","last_peered":"2024-09-16T04:44:44.643410+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:44.643410+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6186,"log_dups_size":0,"ondisk_log_size":6186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50649734,"num_objects":6183,"num_object_clones":0,"num_object_copies":24732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6185,"num_write_kb":49470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3956}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6183","reported_seq":6249,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:45.015037+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:45.015037+0000","last_peered":"2024-09-16T04:44:45.015037+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:45.015037+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6183,"log_dups_size":0,"ondisk_log_size":6183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50615564,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6181,"num_write_kb":49437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6230","reported_seq":6286,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.901363+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:44.901363+0000","last_peered":"2024-09-16T04:44:44.901363+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:44.901363+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6230,"log_dups_size":0,"ondisk_log_size":6230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51008780,"num_objects":6227,"num_object_clones":0,"num_object_copies":24908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6228,"num_write_kb":49821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6012","reported_seq":6056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.967327+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:44.967327+0000","last_peered":"2024-09-16T04:44:44.967327+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:44.967327+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6012,"log_dups_size":0,"ondisk_log_size":6012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49250304,"num_objects":6012,"num_object_clones":0,"num_object_copies":24048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6012,"num_write_kb":48096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3804}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6154","reported_seq":6219,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:45.170144+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:45.170144+0000","last_peered":"2024-09-16T04:44:45.170144+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:45.170144+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6154,"log_dups_size":0,"ondisk_log_size":6154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50361612,"num_objects":6148,"num_object_clones":0,"num_object_copies":24592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6152,"num_write_kb":49196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3922}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6174","reported_seq":10730,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:42.026527+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:42.026527+0000","last_peered":"2024-09-16T04:44:42.026527+0000","last_clean":"2024-09-16T04:44:42.026527+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:42.026527+0000","last_undegraded":"2024-09-16T04:44:42.026527+0000","last_fullsized":"2024-09-16T04:44:42.026527+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6174,"log_dups_size":0,"ondisk_log_size":6174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:45.867845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50533644,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6172,"num_write_kb":49364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6221","reported_seq":6805,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.944798+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:44.944798+0000","last_peered":"2024-09-16T04:44:44.944798+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:44.944798+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50891288,"num_objects":6213,"num_object_clones":0,"num_object_copies":24852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6218,"num_write_kb":49721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":2154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6209","reported_seq":6288,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.639951+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:44.639951+0000","last_peered":"2024-09-16T04:44:44.639951+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:44.639951+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50844940,"num_objects":6207,"num_object_clones":0,"num_object_copies":24828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6207,"num_write_kb":49654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4007}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6090","reported_seq":6148,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.871560+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:44.871560+0000","last_peered":"2024-09-16T04:44:44.871560+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:44.871560+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6090,"log_dups_size":0,"ondisk_log_size":6090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49844114,"num_objects":6085,"num_object_clones":0,"num_object_copies":24340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6087,"num_write_kb":48684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6047","reported_seq":6087,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.835687+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:44.835687+0000","last_peered":"2024-09-16T04:44:44.835687+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:44.835687+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6047,"log_dups_size":0,"ondisk_log_size":6047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49527430,"num_objects":6046,"num_object_clones":0,"num_object_copies":24184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6046,"num_write_kb":48367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3918}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6105","reported_seq":6179,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.598190+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:44.598190+0000","last_peered":"2024-09-16T04:44:44.598190+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:44.598190+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50002566,"num_objects":6104,"num_object_clones":0,"num_object_copies":24416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6104,"num_write_kb":48831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6244","reported_seq":6298,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.913540+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:44.913540+0000","last_peered":"2024-09-16T04:44:44.913540+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:44.913540+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6244,"log_dups_size":0,"ondisk_log_size":6244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51115276,"num_objects":6240,"num_object_clones":0,"num_object_copies":24960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6242,"num_write_kb":49932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4006}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6078","reported_seq":6157,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.590358+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:44.590358+0000","last_peered":"2024-09-16T04:44:44.590358+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:44.590358+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6078,"log_dups_size":0,"ondisk_log_size":6078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49763596,"num_objects":6075,"num_object_clones":0,"num_object_copies":24300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6076,"num_write_kb":48605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3943}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6019","reported_seq":6085,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:45.083321+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:45.083321+0000","last_peered":"2024-09-16T04:44:45.083321+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:45.083321+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263884,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":48124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3775}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":804890280,"num_objects":98258,"num_object_clones":0,"num_object_copies":393032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98285,"num_write_kb":786162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2530,"num_bytes_recovered":20488192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1472286720,"data_stored":1472286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98312,"ondisk_log_size":98312,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745736,"kb_used_data":393336,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93626104,"statfs":{"total":96636764160,"available":95873130496,"internally_reserved":0,"allocated":402776064,"data_stored":402596095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360836815},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,18,4,2,2,6,4,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757380,"kb_used_data":393012,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93614460,"statfs":{"total":96636764160,"available":95861207040,"internally_reserved":0,"allocated":402444288,"data_stored":402264319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051624,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488560,"kb_used_data":258336,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93883280,"statfs":{"total":96636764160,"available":96136478720,"internally_reserved":0,"allocated":264536064,"data_stored":264356095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235728591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753612,"kb_used_data":392956,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93618228,"statfs":{"total":96636764160,"available":95865065472,"internally_reserved":0,"allocated":402386944,"data_stored":402206975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":369290958},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,0,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":402800640,"data_stored":402800640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264564736,"data_stored":264564736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402329600,"data_stored":402329600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":402591744,"data_stored":402591744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:48.217 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:48.318 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 234 20 99179 99159 3.30953 1.91406 0.0592912 0.047197 2024-09-16T04:44:48.798 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:48.833 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-09-16T04:44:48.627234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":805586600,"num_objects":98343,"num_object_clones":0,"num_object_copies":393372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98370,"num_write_kb":786842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2530,"num_bytes_recovered":20488192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98397,"ondisk_log_size":98397,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754696,"kb_used_data":1442888,"kb_used_omap":14,"kb_used_meta":1311729,"kb_avail":374732664,"statfs":{"total":386547056640,"available":383726247936,"internally_reserved":0,"allocated":1477517312,"data_stored":1476797436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1343211057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,20,7,11,9,16,18,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21815296,"num_objects":2663,"num_object_clones":0,"num_object_copies":10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-78,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2663,"num_write_kb":21304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003188"},"pg_stats":[{"pgid":"1.f","version":"33'6163","reported_seq":6207,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.840645+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:44.840645+0000","last_peered":"2024-09-16T04:44:44.840645+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:44.840645+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6163,"log_dups_size":0,"ondisk_log_size":6163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50487296,"num_objects":6163,"num_object_clones":0,"num_object_copies":24652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":49304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6197","reported_seq":6263,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:45.140980+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:45.140980+0000","last_peered":"2024-09-16T04:44:45.140980+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:45.140980+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6197,"log_dups_size":0,"ondisk_log_size":6197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50730252,"num_objects":6193,"num_object_clones":0,"num_object_copies":24772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6195,"num_write_kb":49556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3982}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6186","reported_seq":6260,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.643410+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:44.643410+0000","last_peered":"2024-09-16T04:44:44.643410+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:44.643410+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6186,"log_dups_size":0,"ondisk_log_size":6186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50649734,"num_objects":6183,"num_object_clones":0,"num_object_copies":24732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6185,"num_write_kb":49470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3956}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6183","reported_seq":6249,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:45.015037+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:45.015037+0000","last_peered":"2024-09-16T04:44:45.015037+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:45.015037+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6183,"log_dups_size":0,"ondisk_log_size":6183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50615564,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6181,"num_write_kb":49437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6230","reported_seq":6286,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.901363+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:44.901363+0000","last_peered":"2024-09-16T04:44:44.901363+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:44.901363+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6230,"log_dups_size":0,"ondisk_log_size":6230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51008780,"num_objects":6227,"num_object_clones":0,"num_object_copies":24908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6228,"num_write_kb":49821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6012","reported_seq":6056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.967327+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:44.967327+0000","last_peered":"2024-09-16T04:44:44.967327+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:44.967327+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6012,"log_dups_size":0,"ondisk_log_size":6012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49250304,"num_objects":6012,"num_object_clones":0,"num_object_copies":24048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6012,"num_write_kb":48096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3804}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6154","reported_seq":6219,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:45.170144+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:45.170144+0000","last_peered":"2024-09-16T04:44:45.170144+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:45.170144+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6154,"log_dups_size":0,"ondisk_log_size":6154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50361612,"num_objects":6148,"num_object_clones":0,"num_object_copies":24592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6152,"num_write_kb":49196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3922}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6259","reported_seq":10815,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:47.193054+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:47.193054+0000","last_peered":"2024-09-16T04:44:47.193054+0000","last_clean":"2024-09-16T04:44:47.193054+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:47.193054+0000","last_undegraded":"2024-09-16T04:44:47.193054+0000","last_fullsized":"2024-09-16T04:44:47.193054+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6259,"log_dups_size":0,"ondisk_log_size":6259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:51.888276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51229964,"num_objects":6254,"num_object_clones":0,"num_object_copies":25016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6257,"num_write_kb":50044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6221","reported_seq":6805,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.944798+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:44.944798+0000","last_peered":"2024-09-16T04:44:44.944798+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:44.944798+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50891288,"num_objects":6213,"num_object_clones":0,"num_object_copies":24852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6218,"num_write_kb":49721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":2154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6209","reported_seq":6288,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.639951+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:44.639951+0000","last_peered":"2024-09-16T04:44:44.639951+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:44.639951+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50844940,"num_objects":6207,"num_object_clones":0,"num_object_copies":24828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6207,"num_write_kb":49654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4007}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6090","reported_seq":6148,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.871560+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:44.871560+0000","last_peered":"2024-09-16T04:44:44.871560+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:44.871560+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6090,"log_dups_size":0,"ondisk_log_size":6090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49844114,"num_objects":6085,"num_object_clones":0,"num_object_copies":24340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6087,"num_write_kb":48684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6047","reported_seq":6087,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.835687+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:44.835687+0000","last_peered":"2024-09-16T04:44:44.835687+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:44.835687+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6047,"log_dups_size":0,"ondisk_log_size":6047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49527430,"num_objects":6046,"num_object_clones":0,"num_object_copies":24184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6046,"num_write_kb":48367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3918}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6105","reported_seq":6179,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.598190+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:44.598190+0000","last_peered":"2024-09-16T04:44:44.598190+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:44.598190+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50002566,"num_objects":6104,"num_object_clones":0,"num_object_copies":24416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6104,"num_write_kb":48831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6244","reported_seq":6298,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.913540+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:44.913540+0000","last_peered":"2024-09-16T04:44:44.913540+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:44.913540+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6244,"log_dups_size":0,"ondisk_log_size":6244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51115276,"num_objects":6240,"num_object_clones":0,"num_object_copies":24960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6242,"num_write_kb":49932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4006}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6078","reported_seq":6157,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.590358+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:44.590358+0000","last_peered":"2024-09-16T04:44:44.590358+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:44.590358+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6078,"log_dups_size":0,"ondisk_log_size":6078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49763596,"num_objects":6075,"num_object_clones":0,"num_object_copies":24300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6076,"num_write_kb":48605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3943}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6019","reported_seq":6085,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:45.083321+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:45.083321+0000","last_peered":"2024-09-16T04:44:45.083321+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:45.083321+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263884,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":48124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3775}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":805586600,"num_objects":98343,"num_object_clones":0,"num_object_copies":393372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98370,"num_write_kb":786842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2530,"num_bytes_recovered":20488192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1477783552,"data_stored":1477783552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98397,"ondisk_log_size":98397,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745736,"kb_used_data":393336,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93626104,"statfs":{"total":96636764160,"available":95873130496,"internally_reserved":0,"allocated":402776064,"data_stored":402596095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360836815},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,18,4,2,2,6,4,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757380,"kb_used_data":393012,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93614460,"statfs":{"total":96636764160,"available":95861207040,"internally_reserved":0,"allocated":402444288,"data_stored":402264319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051625,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497968,"kb_used_data":263584,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93873872,"statfs":{"total":96636764160,"available":96126844928,"internally_reserved":0,"allocated":269910016,"data_stored":269730047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239988431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2,4,1,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753612,"kb_used_data":392956,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93618228,"statfs":{"total":96636764160,"available":95865065472,"internally_reserved":0,"allocated":402386944,"data_stored":402206975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":369290958},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,0,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":402800640,"data_stored":402800640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270061568,"data_stored":270061568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402329600,"data_stored":402329600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":402591744,"data_stored":402591744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:48.834 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:44:48.834 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6163", 'reported_seq': 6207, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:44.840645+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:44:44.840645+0000', 'last_peered': '2024-09-16T04:44:44.840645+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:44:44.840645+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6163, 'log_dups_size': 0, 'ondisk_log_size': 6163, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50487296, 'num_objects': 6163, 'num_object_clones': 0, 'num_object_copies': 24652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6163, 'num_write_kb': 49304, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3944}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:48.834 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:44:48.834 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6197", 'reported_seq': 6263, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:45.140980+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:44:45.140980+0000', 'last_peered': '2024-09-16T04:44:45.140980+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:44:45.140980+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6197, 'log_dups_size': 0, 'ondisk_log_size': 6197, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50730252, 'num_objects': 6193, 'num_object_clones': 0, 'num_object_copies': 24772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6195, 'num_write_kb': 49556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3982}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:48.834 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:44:48.834 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6186", 'reported_seq': 6260, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:44.643410+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:44:44.643410+0000', 'last_peered': '2024-09-16T04:44:44.643410+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:44:44.643410+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6186, 'log_dups_size': 0, 'ondisk_log_size': 6186, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50649734, 'num_objects': 6183, 'num_object_clones': 0, 'num_object_copies': 24732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6185, 'num_write_kb': 49470, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3956}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:48.834 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:44:48.834 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6183", 'reported_seq': 6249, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:45.015037+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:44:45.015037+0000', 'last_peered': '2024-09-16T04:44:45.015037+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:44:45.015037+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6183, 'log_dups_size': 0, 'ondisk_log_size': 6183, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50615564, 'num_objects': 6179, 'num_object_clones': 0, 'num_object_copies': 24716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6181, 'num_write_kb': 49437, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3950}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:48.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:44:48.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6230", 'reported_seq': 6286, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:44.901363+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:44:44.901363+0000', 'last_peered': '2024-09-16T04:44:44.901363+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:44:44.901363+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6230, 'log_dups_size': 0, 'ondisk_log_size': 6230, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51008780, 'num_objects': 6227, 'num_object_clones': 0, 'num_object_copies': 24908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6228, 'num_write_kb': 49821, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3944}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:48.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:44:48.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6012", 'reported_seq': 6056, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:44.967327+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:44:44.967327+0000', 'last_peered': '2024-09-16T04:44:44.967327+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:44:44.967327+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6012, 'log_dups_size': 0, 'ondisk_log_size': 6012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49250304, 'num_objects': 6012, 'num_object_clones': 0, 'num_object_copies': 24048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6012, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6012, 'num_write_kb': 48096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3804}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:48.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:44:48.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6154", 'reported_seq': 6219, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:45.170144+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:44:45.170144+0000', 'last_peered': '2024-09-16T04:44:45.170144+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:44:45.170144+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6154, 'log_dups_size': 0, 'ondisk_log_size': 6154, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50361612, 'num_objects': 6148, 'num_object_clones': 0, 'num_object_copies': 24592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6148, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6152, 'num_write_kb': 49196, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3922}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:48.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:44:48.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6221", 'reported_seq': 6805, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:44.944798+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:44:44.944798+0000', 'last_peered': '2024-09-16T04:44:44.944798+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:44:44.944798+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6221, 'log_dups_size': 0, 'ondisk_log_size': 6221, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50891288, 'num_objects': 6213, 'num_object_clones': 0, 'num_object_copies': 24852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2077, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6213, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6218, 'num_write_kb': 49721, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 266, 'num_bytes_recovered': 2154496, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4136}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:48.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:44:48.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6209", 'reported_seq': 6288, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:44.639951+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:44:44.639951+0000', 'last_peered': '2024-09-16T04:44:44.639951+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:44:44.639951+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6209, 'log_dups_size': 0, 'ondisk_log_size': 6209, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50844940, 'num_objects': 6207, 'num_object_clones': 0, 'num_object_copies': 24828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6207, 'num_write_kb': 49654, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4007}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:48.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:44:48.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6090", 'reported_seq': 6148, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:44.871560+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:44:44.871560+0000', 'last_peered': '2024-09-16T04:44:44.871560+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:44:44.871560+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6090, 'log_dups_size': 0, 'ondisk_log_size': 6090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49844114, 'num_objects': 6085, 'num_object_clones': 0, 'num_object_copies': 24340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6087, 'num_write_kb': 48684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3896}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:48.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:44:48.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6047", 'reported_seq': 6087, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:44.835687+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:44:44.835687+0000', 'last_peered': '2024-09-16T04:44:44.835687+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:44:44.835687+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6047, 'log_dups_size': 0, 'ondisk_log_size': 6047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49527430, 'num_objects': 6046, 'num_object_clones': 0, 'num_object_copies': 24184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6046, 'num_write_kb': 48367, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3918}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:48.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:44:48.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6105", 'reported_seq': 6179, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:44.598190+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:44:44.598190+0000', 'last_peered': '2024-09-16T04:44:44.598190+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:44:44.598190+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6105, 'log_dups_size': 0, 'ondisk_log_size': 6105, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50002566, 'num_objects': 6104, 'num_object_clones': 0, 'num_object_copies': 24416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6104, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6104, 'num_write_kb': 48831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:48.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:44:48.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6244", 'reported_seq': 6298, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:44.913540+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:44:44.913540+0000', 'last_peered': '2024-09-16T04:44:44.913540+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:44:44.913540+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6244, 'log_dups_size': 0, 'ondisk_log_size': 6244, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51115276, 'num_objects': 6240, 'num_object_clones': 0, 'num_object_copies': 24960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6240, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6242, 'num_write_kb': 49932, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4006}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:48.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:44:48.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6078", 'reported_seq': 6157, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:44.590358+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:44:44.590358+0000', 'last_peered': '2024-09-16T04:44:44.590358+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:44:44.590358+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6078, 'log_dups_size': 0, 'ondisk_log_size': 6078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49763596, 'num_objects': 6075, 'num_object_clones': 0, 'num_object_copies': 24300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6076, 'num_write_kb': 48605, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3943}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:48.837 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:44:48.837 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6019", 'reported_seq': 6085, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:45.083321+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:44:45.083321+0000', 'last_peered': '2024-09-16T04:44:45.083321+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:44:45.083321+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6019, 'log_dups_size': 0, 'ondisk_log_size': 6019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49263884, 'num_objects': 6014, 'num_object_clones': 0, 'num_object_copies': 24056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6017, 'num_write_kb': 48124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3775}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:48.837 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:44:49.318 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 235 20 99416 99396 3.30333 1.85156 0.0592806 0.0472863 2024-09-16T04:44:49.519 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:49.519 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33049/393372 objects degraded (8.401%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":301,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":98343,"data_bytes":805586600,"bytes_used":2820808704,"bytes_avail":383726247936,"bytes_total":386547056640,"degraded_objects":33049,"degraded_total":393372,"degraded_ratio":0.084014622291367969,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":53233,"recovering_keys_per_sec":0,"num_objects_recovered":78,"num_bytes_recovered":638976,"num_keys_recovered":0,"write_bytes_sec":1817458,"read_op_per_sec":0,"write_op_per_sec":221},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:44:49.543 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:44:49.544 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:50.146 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:50.147 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:50.169 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-09-16T04:44:48.627234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":805586600,"num_objects":98343,"num_object_clones":0,"num_object_copies":393372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98370,"num_write_kb":786842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2530,"num_bytes_recovered":20488192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98397,"ondisk_log_size":98397,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754696,"kb_used_data":1442888,"kb_used_omap":14,"kb_used_meta":1311729,"kb_avail":374732664,"statfs":{"total":386547056640,"available":383726247936,"internally_reserved":0,"allocated":1477517312,"data_stored":1476797436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1343211057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,20,7,11,9,16,18,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21815296,"num_objects":2663,"num_object_clones":0,"num_object_copies":10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-78,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2663,"num_write_kb":21304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003188"},"pg_stats":[{"pgid":"1.f","version":"33'6163","reported_seq":6207,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.840645+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:44.840645+0000","last_peered":"2024-09-16T04:44:44.840645+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:44.840645+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6163,"log_dups_size":0,"ondisk_log_size":6163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50487296,"num_objects":6163,"num_object_clones":0,"num_object_copies":24652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":49304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6197","reported_seq":6263,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:45.140980+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:45.140980+0000","last_peered":"2024-09-16T04:44:45.140980+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:45.140980+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6197,"log_dups_size":0,"ondisk_log_size":6197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50730252,"num_objects":6193,"num_object_clones":0,"num_object_copies":24772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6195,"num_write_kb":49556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3982}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6186","reported_seq":6260,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.643410+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:44.643410+0000","last_peered":"2024-09-16T04:44:44.643410+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:44.643410+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6186,"log_dups_size":0,"ondisk_log_size":6186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50649734,"num_objects":6183,"num_object_clones":0,"num_object_copies":24732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6185,"num_write_kb":49470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3956}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6183","reported_seq":6249,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:45.015037+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:45.015037+0000","last_peered":"2024-09-16T04:44:45.015037+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:45.015037+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6183,"log_dups_size":0,"ondisk_log_size":6183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50615564,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6181,"num_write_kb":49437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6230","reported_seq":6286,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.901363+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:44.901363+0000","last_peered":"2024-09-16T04:44:44.901363+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:44.901363+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6230,"log_dups_size":0,"ondisk_log_size":6230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51008780,"num_objects":6227,"num_object_clones":0,"num_object_copies":24908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6228,"num_write_kb":49821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6012","reported_seq":6056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.967327+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:44.967327+0000","last_peered":"2024-09-16T04:44:44.967327+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:44.967327+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6012,"log_dups_size":0,"ondisk_log_size":6012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49250304,"num_objects":6012,"num_object_clones":0,"num_object_copies":24048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6012,"num_write_kb":48096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3804}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6154","reported_seq":6219,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:45.170144+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:45.170144+0000","last_peered":"2024-09-16T04:44:45.170144+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:45.170144+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6154,"log_dups_size":0,"ondisk_log_size":6154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50361612,"num_objects":6148,"num_object_clones":0,"num_object_copies":24592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6152,"num_write_kb":49196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3922}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6259","reported_seq":10815,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:47.193054+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:47.193054+0000","last_peered":"2024-09-16T04:44:47.193054+0000","last_clean":"2024-09-16T04:44:47.193054+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:47.193054+0000","last_undegraded":"2024-09-16T04:44:47.193054+0000","last_fullsized":"2024-09-16T04:44:47.193054+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6259,"log_dups_size":0,"ondisk_log_size":6259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:51.888276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51229964,"num_objects":6254,"num_object_clones":0,"num_object_copies":25016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6257,"num_write_kb":50044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6221","reported_seq":6805,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.944798+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:44.944798+0000","last_peered":"2024-09-16T04:44:44.944798+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:44.944798+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50891288,"num_objects":6213,"num_object_clones":0,"num_object_copies":24852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6218,"num_write_kb":49721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":2154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6209","reported_seq":6288,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.639951+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:44.639951+0000","last_peered":"2024-09-16T04:44:44.639951+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:44.639951+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50844940,"num_objects":6207,"num_object_clones":0,"num_object_copies":24828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6207,"num_write_kb":49654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4007}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6090","reported_seq":6148,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.871560+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:44.871560+0000","last_peered":"2024-09-16T04:44:44.871560+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:44.871560+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6090,"log_dups_size":0,"ondisk_log_size":6090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49844114,"num_objects":6085,"num_object_clones":0,"num_object_copies":24340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6087,"num_write_kb":48684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6047","reported_seq":6087,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.835687+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:44.835687+0000","last_peered":"2024-09-16T04:44:44.835687+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:44.835687+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6047,"log_dups_size":0,"ondisk_log_size":6047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49527430,"num_objects":6046,"num_object_clones":0,"num_object_copies":24184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6046,"num_write_kb":48367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3918}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6105","reported_seq":6179,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.598190+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:44.598190+0000","last_peered":"2024-09-16T04:44:44.598190+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:44.598190+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50002566,"num_objects":6104,"num_object_clones":0,"num_object_copies":24416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6104,"num_write_kb":48831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6244","reported_seq":6298,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.913540+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:44.913540+0000","last_peered":"2024-09-16T04:44:44.913540+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:44.913540+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6244,"log_dups_size":0,"ondisk_log_size":6244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51115276,"num_objects":6240,"num_object_clones":0,"num_object_copies":24960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6242,"num_write_kb":49932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4006}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6078","reported_seq":6157,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:44.590358+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:44.590358+0000","last_peered":"2024-09-16T04:44:44.590358+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:44.590358+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6078,"log_dups_size":0,"ondisk_log_size":6078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49763596,"num_objects":6075,"num_object_clones":0,"num_object_copies":24300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6076,"num_write_kb":48605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3943}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6019","reported_seq":6085,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:45.083321+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:45.083321+0000","last_peered":"2024-09-16T04:44:45.083321+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:45.083321+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49263884,"num_objects":6014,"num_object_clones":0,"num_object_copies":24056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":48124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3775}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":805586600,"num_objects":98343,"num_object_clones":0,"num_object_copies":393372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98370,"num_write_kb":786842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2530,"num_bytes_recovered":20488192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1477783552,"data_stored":1477783552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98397,"ondisk_log_size":98397,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745736,"kb_used_data":393336,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93626104,"statfs":{"total":96636764160,"available":95873130496,"internally_reserved":0,"allocated":402776064,"data_stored":402596095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360836815},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,18,4,2,2,6,4,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757380,"kb_used_data":393012,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93614460,"statfs":{"total":96636764160,"available":95861207040,"internally_reserved":0,"allocated":402444288,"data_stored":402264319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051625,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497968,"kb_used_data":263584,"kb_used_omap":4,"kb_used_meta":234363,"kb_avail":93873872,"statfs":{"total":96636764160,"available":96126844928,"internally_reserved":0,"allocated":269910016,"data_stored":269730047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239988431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2,4,1,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753612,"kb_used_data":392956,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93618228,"statfs":{"total":96636764160,"available":95865065472,"internally_reserved":0,"allocated":402386944,"data_stored":402206975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":369290958},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,0,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":402800640,"data_stored":402800640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270061568,"data_stored":270061568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402329600,"data_stored":402329600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":402591744,"data_stored":402591744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:50.318 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 236 20 99662 99642 3.29747 1.92188 0.0491649 0.0473718 2024-09-16T04:44:51.318 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 237 20 99937 99917 3.29263 2.14844 0.0522991 0.0474409 2024-09-16T04:44:52.318 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 238 20 100211 100191 3.28779 2.14062 0.073878 0.0475128 2024-09-16T04:44:53.171 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:53.318 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout: 239 20 100485 100465 3.28298 2.14062 0.0598947 0.04758 2024-09-16T04:44:53.789 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:53.789 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:53.810 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-09-16T04:44:52.628288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":816350888,"num_objects":99657,"num_object_clones":0,"num_object_copies":398628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99684,"num_write_kb":797354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2565,"num_bytes_recovered":20774912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99711,"ondisk_log_size":99711,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2775448,"kb_used_data":1463640,"kb_used_omap":14,"kb_used_meta":1311729,"kb_avail":374711912,"statfs":{"total":386547056640,"available":383704997888,"internally_reserved":0,"allocated":1498767360,"data_stored":1498047484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1343211057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,15,19,19,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21970944,"num_objects":2682,"num_object_clones":0,"num_object_copies":10728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2682,"num_write_kb":21456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":614400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003176"},"pg_stats":[{"pgid":"1.f","version":"33'6250","reported_seq":6294,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.083915+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:50.083915+0000","last_peered":"2024-09-16T04:44:50.083915+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:50.083915+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6250,"log_dups_size":0,"ondisk_log_size":6250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51200000,"num_objects":6250,"num_object_clones":0,"num_object_copies":25000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6250,"num_write_kb":50000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6274","reported_seq":6339,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.261319+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:50.261319+0000","last_peered":"2024-09-16T04:44:50.261319+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:50.261319+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6274,"log_dups_size":0,"ondisk_log_size":6274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51361036,"num_objects":6270,"num_object_clones":0,"num_object_copies":25080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6272,"num_write_kb":50172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6261","reported_seq":6337,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:49.710955+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:49.710955+0000","last_peered":"2024-09-16T04:44:49.710955+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:49.710955+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51264134,"num_objects":6258,"num_object_clones":0,"num_object_copies":25032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":50070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4031}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6276","reported_seq":6343,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.135265+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:50.135265+0000","last_peered":"2024-09-16T04:44:50.135265+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:50.135265+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6276,"log_dups_size":0,"ondisk_log_size":6276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51377420,"num_objects":6272,"num_object_clones":0,"num_object_copies":25088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6274,"num_write_kb":50181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4043}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6303","reported_seq":6360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.108970+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:50.108970+0000","last_peered":"2024-09-16T04:44:50.108970+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:50.108970+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51606796,"num_objects":6300,"num_object_clones":0,"num_object_copies":25200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6301,"num_write_kb":50405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4017}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6087","reported_seq":6129,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.012915+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:50.012915+0000","last_peered":"2024-09-16T04:44:50.012915+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:50.012915+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49864704,"num_objects":6087,"num_object_clones":0,"num_object_copies":24348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6087,"num_write_kb":48696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3879}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6228","reported_seq":6294,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.171578+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:50.171578+0000","last_peered":"2024-09-16T04:44:50.171578+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:50.171578+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6228,"log_dups_size":0,"ondisk_log_size":6228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50967820,"num_objects":6222,"num_object_clones":0,"num_object_copies":24888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6226,"num_write_kb":49788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3996}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6363","reported_seq":10918,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:52.265881+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:52.265881+0000","last_peered":"2024-09-16T04:44:52.265881+0000","last_clean":"2024-09-16T04:44:52.265881+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:52.265881+0000","last_undegraded":"2024-09-16T04:44:52.265881+0000","last_fullsized":"2024-09-16T04:44:52.265881+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6363,"log_dups_size":0,"ondisk_log_size":6363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:56.943347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52081932,"num_objects":6358,"num_object_clones":0,"num_object_copies":25432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6361,"num_write_kb":50876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6311","reported_seq":6965,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.010555+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:50.010555+0000","last_peered":"2024-09-16T04:44:50.010555+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:50.010555+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6311,"log_dups_size":0,"ondisk_log_size":6311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51628568,"num_objects":6303,"num_object_clones":0,"num_object_copies":25212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6308,"num_write_kb":50441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":2441216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6318","reported_seq":6397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:49.708043+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:49.708043+0000","last_peered":"2024-09-16T04:44:49.708043+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:49.708043+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6318,"log_dups_size":0,"ondisk_log_size":6318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51737868,"num_objects":6316,"num_object_clones":0,"num_object_copies":25264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6316,"num_write_kb":50526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6170","reported_seq":6228,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.004124+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:50.004124+0000","last_peered":"2024-09-16T04:44:50.004124+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:50.004124+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6170,"log_dups_size":0,"ondisk_log_size":6170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50499474,"num_objects":6165,"num_object_clones":0,"num_object_copies":24660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6167,"num_write_kb":49324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3976}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6112","reported_seq":6152,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:49.698354+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:49.698354+0000","last_peered":"2024-09-16T04:44:49.698354+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:49.698354+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6112,"log_dups_size":0,"ondisk_log_size":6112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50059910,"num_objects":6111,"num_object_clones":0,"num_object_copies":24444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":48887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3983}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6189","reported_seq":6264,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:49.738253+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:49.738253+0000","last_peered":"2024-09-16T04:44:49.738253+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:49.738253+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6189,"log_dups_size":0,"ondisk_log_size":6189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50690694,"num_objects":6188,"num_object_clones":0,"num_object_copies":24752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6188,"num_write_kb":49503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3963}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6324","reported_seq":6376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.095391+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:50.095391+0000","last_peered":"2024-09-16T04:44:50.095391+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:50.095391+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6324,"log_dups_size":0,"ondisk_log_size":6324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51770636,"num_objects":6320,"num_object_clones":0,"num_object_copies":25280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6322,"num_write_kb":50572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4086}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6150","reported_seq":6230,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:49.681248+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:49.681248+0000","last_peered":"2024-09-16T04:44:49.681248+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:49.681248+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6150,"log_dups_size":0,"ondisk_log_size":6150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50353420,"num_objects":6147,"num_object_clones":0,"num_object_copies":24588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6148,"num_write_kb":49181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4015}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6095","reported_seq":6161,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.205629+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:50.205629+0000","last_peered":"2024-09-16T04:44:50.205629+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:50.205629+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6095,"log_dups_size":0,"ondisk_log_size":6095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49886476,"num_objects":6090,"num_object_clones":0,"num_object_copies":24360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6093,"num_write_kb":48732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3851}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":816350888,"num_objects":99657,"num_object_clones":0,"num_object_copies":398628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99684,"num_write_kb":797354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2565,"num_bytes_recovered":20774912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1499328512,"data_stored":1499328512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99711,"ondisk_log_size":99711,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755020,"kb_used_data":398460,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93616820,"statfs":{"total":96636764160,"available":95863623680,"internally_reserved":0,"allocated":408023040,"data_stored":407843071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365096655},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,3,6,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750040,"kb_used_data":398152,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93621800,"statfs":{"total":96636764160,"available":95868723200,"internally_reserved":0,"allocated":407707648,"data_stored":407527679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,9,1,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":507412,"kb_used_data":268868,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93864428,"statfs":{"total":96636764160,"available":96117174272,"internally_reserved":0,"allocated":275320832,"data_stored":275140863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244248271},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762976,"kb_used_data":398160,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93608864,"statfs":{"total":96636764160,"available":95855476736,"internally_reserved":0,"allocated":407715840,"data_stored":407535871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":373550798},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":408150016,"data_stored":408150016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275460096,"data_stored":275460096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":407719936,"data_stored":407719936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":407998464,"data_stored":407998464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:53.811 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:54.318 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Total time run: 240.037 2024-09-16T04:44:54.319 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Total writes made: 100701 2024-09-16T04:44:54.319 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Write size: 8192 2024-09-16T04:44:54.319 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Object size: 8192 2024-09-16T04:44:54.319 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Bandwidth (MB/sec): 3.27752 2024-09-16T04:44:54.319 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Stddev Bandwidth: 2.28315 2024-09-16T04:44:54.319 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Max bandwidth (MB/sec): 12.9141 2024-09-16T04:44:54.319 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Min bandwidth (MB/sec): 0 2024-09-16T04:44:54.319 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Average IOPS: 419 2024-09-16T04:44:54.319 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Stddev IOPS: 292.243 2024-09-16T04:44:54.319 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Max IOPS: 1653 2024-09-16T04:44:54.319 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Min IOPS: 0 2024-09-16T04:44:54.319 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Average Latency(s): 0.0476654 2024-09-16T04:44:54.319 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Stddev Latency(s): 0.0397934 2024-09-16T04:44:54.320 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Max latency(s): 1.73107 2024-09-16T04:44:54.320 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Min latency(s): 0.00269411 2024-09-16T04:44:54.334 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Cleaning up (deleting benchmark objects) 2024-09-16T04:44:54.392 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:54.392 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:54.418 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-09-16T04:44:52.628288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":816350888,"num_objects":99657,"num_object_clones":0,"num_object_copies":398628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99684,"num_write_kb":797354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2565,"num_bytes_recovered":20774912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99711,"ondisk_log_size":99711,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2775448,"kb_used_data":1463640,"kb_used_omap":14,"kb_used_meta":1311729,"kb_avail":374711912,"statfs":{"total":386547056640,"available":383704997888,"internally_reserved":0,"allocated":1498767360,"data_stored":1498047484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1343211057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,15,19,19,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21970944,"num_objects":2682,"num_object_clones":0,"num_object_copies":10728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2682,"num_write_kb":21456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":614400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003176"},"pg_stats":[{"pgid":"1.f","version":"33'6250","reported_seq":6294,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.083915+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:50.083915+0000","last_peered":"2024-09-16T04:44:50.083915+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:50.083915+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6250,"log_dups_size":0,"ondisk_log_size":6250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51200000,"num_objects":6250,"num_object_clones":0,"num_object_copies":25000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6250,"num_write_kb":50000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6274","reported_seq":6339,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.261319+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:50.261319+0000","last_peered":"2024-09-16T04:44:50.261319+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:50.261319+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6274,"log_dups_size":0,"ondisk_log_size":6274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51361036,"num_objects":6270,"num_object_clones":0,"num_object_copies":25080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6272,"num_write_kb":50172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6261","reported_seq":6337,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:49.710955+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:49.710955+0000","last_peered":"2024-09-16T04:44:49.710955+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:49.710955+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51264134,"num_objects":6258,"num_object_clones":0,"num_object_copies":25032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":50070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4031}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6276","reported_seq":6343,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.135265+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:50.135265+0000","last_peered":"2024-09-16T04:44:50.135265+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:50.135265+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6276,"log_dups_size":0,"ondisk_log_size":6276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51377420,"num_objects":6272,"num_object_clones":0,"num_object_copies":25088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6274,"num_write_kb":50181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4043}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6303","reported_seq":6360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.108970+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:50.108970+0000","last_peered":"2024-09-16T04:44:50.108970+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:50.108970+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51606796,"num_objects":6300,"num_object_clones":0,"num_object_copies":25200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6301,"num_write_kb":50405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4017}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6087","reported_seq":6129,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.012915+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:50.012915+0000","last_peered":"2024-09-16T04:44:50.012915+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:50.012915+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49864704,"num_objects":6087,"num_object_clones":0,"num_object_copies":24348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6087,"num_write_kb":48696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3879}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6228","reported_seq":6294,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.171578+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:50.171578+0000","last_peered":"2024-09-16T04:44:50.171578+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:50.171578+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6228,"log_dups_size":0,"ondisk_log_size":6228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50967820,"num_objects":6222,"num_object_clones":0,"num_object_copies":24888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6226,"num_write_kb":49788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3996}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6363","reported_seq":10918,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:52.265881+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:52.265881+0000","last_peered":"2024-09-16T04:44:52.265881+0000","last_clean":"2024-09-16T04:44:52.265881+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:52.265881+0000","last_undegraded":"2024-09-16T04:44:52.265881+0000","last_fullsized":"2024-09-16T04:44:52.265881+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6363,"log_dups_size":0,"ondisk_log_size":6363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:56.943347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52081932,"num_objects":6358,"num_object_clones":0,"num_object_copies":25432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6361,"num_write_kb":50876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6311","reported_seq":6965,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.010555+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:50.010555+0000","last_peered":"2024-09-16T04:44:50.010555+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:50.010555+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6311,"log_dups_size":0,"ondisk_log_size":6311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51628568,"num_objects":6303,"num_object_clones":0,"num_object_copies":25212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6308,"num_write_kb":50441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":2441216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6318","reported_seq":6397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:49.708043+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:49.708043+0000","last_peered":"2024-09-16T04:44:49.708043+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:49.708043+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6318,"log_dups_size":0,"ondisk_log_size":6318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51737868,"num_objects":6316,"num_object_clones":0,"num_object_copies":25264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6316,"num_write_kb":50526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6170","reported_seq":6228,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.004124+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:50.004124+0000","last_peered":"2024-09-16T04:44:50.004124+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:50.004124+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6170,"log_dups_size":0,"ondisk_log_size":6170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50499474,"num_objects":6165,"num_object_clones":0,"num_object_copies":24660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6167,"num_write_kb":49324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3976}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6112","reported_seq":6152,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:49.698354+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:49.698354+0000","last_peered":"2024-09-16T04:44:49.698354+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:49.698354+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6112,"log_dups_size":0,"ondisk_log_size":6112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50059910,"num_objects":6111,"num_object_clones":0,"num_object_copies":24444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":48887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3983}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6189","reported_seq":6264,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:49.738253+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:49.738253+0000","last_peered":"2024-09-16T04:44:49.738253+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:49.738253+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6189,"log_dups_size":0,"ondisk_log_size":6189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50690694,"num_objects":6188,"num_object_clones":0,"num_object_copies":24752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6188,"num_write_kb":49503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3963}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6324","reported_seq":6376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.095391+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:50.095391+0000","last_peered":"2024-09-16T04:44:50.095391+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:50.095391+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6324,"log_dups_size":0,"ondisk_log_size":6324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51770636,"num_objects":6320,"num_object_clones":0,"num_object_copies":25280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6322,"num_write_kb":50572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4086}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6150","reported_seq":6230,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:49.681248+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:49.681248+0000","last_peered":"2024-09-16T04:44:49.681248+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:49.681248+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6150,"log_dups_size":0,"ondisk_log_size":6150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50353420,"num_objects":6147,"num_object_clones":0,"num_object_copies":24588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6148,"num_write_kb":49181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4015}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6095","reported_seq":6161,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.205629+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:50.205629+0000","last_peered":"2024-09-16T04:44:50.205629+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:50.205629+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6095,"log_dups_size":0,"ondisk_log_size":6095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49886476,"num_objects":6090,"num_object_clones":0,"num_object_copies":24360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6093,"num_write_kb":48732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3851}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":816350888,"num_objects":99657,"num_object_clones":0,"num_object_copies":398628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99684,"num_write_kb":797354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2565,"num_bytes_recovered":20774912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1499328512,"data_stored":1499328512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99711,"ondisk_log_size":99711,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755020,"kb_used_data":398460,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93616820,"statfs":{"total":96636764160,"available":95863623680,"internally_reserved":0,"allocated":408023040,"data_stored":407843071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365096655},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,3,6,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750040,"kb_used_data":398152,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93621800,"statfs":{"total":96636764160,"available":95868723200,"internally_reserved":0,"allocated":407707648,"data_stored":407527679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,9,1,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":507412,"kb_used_data":268868,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93864428,"statfs":{"total":96636764160,"available":96117174272,"internally_reserved":0,"allocated":275320832,"data_stored":275140863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244248271},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762976,"kb_used_data":398160,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93608864,"statfs":{"total":96636764160,"available":95855476736,"internally_reserved":0,"allocated":407715840,"data_stored":407535871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":373550798},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":408150016,"data_stored":408150016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275460096,"data_stored":275460096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":407719936,"data_stored":407719936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":407998464,"data_stored":407998464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:54.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:44:54.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6250", 'reported_seq': 6294, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:50.083915+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:44:50.083915+0000', 'last_peered': '2024-09-16T04:44:50.083915+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:44:50.083915+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6250, 'log_dups_size': 0, 'ondisk_log_size': 6250, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51200000, 'num_objects': 6250, 'num_object_clones': 0, 'num_object_copies': 25000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6250, 'num_write_kb': 50000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4031}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:54.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:44:54.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6274", 'reported_seq': 6339, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:50.261319+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:44:50.261319+0000', 'last_peered': '2024-09-16T04:44:50.261319+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:44:50.261319+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6274, 'log_dups_size': 0, 'ondisk_log_size': 6274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51361036, 'num_objects': 6270, 'num_object_clones': 0, 'num_object_copies': 25080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6270, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6272, 'num_write_kb': 50172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4059}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:54.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:44:54.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6261", 'reported_seq': 6337, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:49.710955+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:44:49.710955+0000', 'last_peered': '2024-09-16T04:44:49.710955+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:44:49.710955+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6261, 'log_dups_size': 0, 'ondisk_log_size': 6261, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51264134, 'num_objects': 6258, 'num_object_clones': 0, 'num_object_copies': 25032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6260, 'num_write_kb': 50070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4031}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:54.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:44:54.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6276", 'reported_seq': 6343, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:50.135265+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:44:50.135265+0000', 'last_peered': '2024-09-16T04:44:50.135265+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:44:50.135265+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6276, 'log_dups_size': 0, 'ondisk_log_size': 6276, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51377420, 'num_objects': 6272, 'num_object_clones': 0, 'num_object_copies': 25088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6274, 'num_write_kb': 50181, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4043}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:54.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:44:54.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6303", 'reported_seq': 6360, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:50.108970+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:44:50.108970+0000', 'last_peered': '2024-09-16T04:44:50.108970+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:44:50.108970+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6303, 'log_dups_size': 0, 'ondisk_log_size': 6303, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51606796, 'num_objects': 6300, 'num_object_clones': 0, 'num_object_copies': 25200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6301, 'num_write_kb': 50405, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4017}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:54.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:44:54.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6087", 'reported_seq': 6129, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:50.012915+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:44:50.012915+0000', 'last_peered': '2024-09-16T04:44:50.012915+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:44:50.012915+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6087, 'log_dups_size': 0, 'ondisk_log_size': 6087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49864704, 'num_objects': 6087, 'num_object_clones': 0, 'num_object_copies': 24348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6087, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6087, 'num_write_kb': 48696, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3879}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:54.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:44:54.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6228", 'reported_seq': 6294, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:50.171578+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:44:50.171578+0000', 'last_peered': '2024-09-16T04:44:50.171578+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:44:50.171578+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6228, 'log_dups_size': 0, 'ondisk_log_size': 6228, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50967820, 'num_objects': 6222, 'num_object_clones': 0, 'num_object_copies': 24888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6226, 'num_write_kb': 49788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3996}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:54.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:44:54.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6311", 'reported_seq': 6965, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:50.010555+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:44:50.010555+0000', 'last_peered': '2024-09-16T04:44:50.010555+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:44:50.010555+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6311, 'log_dups_size': 0, 'ondisk_log_size': 6311, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51628568, 'num_objects': 6303, 'num_object_clones': 0, 'num_object_copies': 25212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2042, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6303, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6308, 'num_write_kb': 50441, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 301, 'num_bytes_recovered': 2441216, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4261}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:54.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:44:54.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6318", 'reported_seq': 6397, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:49.708043+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:44:49.708043+0000', 'last_peered': '2024-09-16T04:44:49.708043+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:44:49.708043+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6318, 'log_dups_size': 0, 'ondisk_log_size': 6318, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51737868, 'num_objects': 6316, 'num_object_clones': 0, 'num_object_copies': 25264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6316, 'num_write_kb': 50526, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4116}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:54.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:44:54.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6170", 'reported_seq': 6228, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:50.004124+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:44:50.004124+0000', 'last_peered': '2024-09-16T04:44:50.004124+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:44:50.004124+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6170, 'log_dups_size': 0, 'ondisk_log_size': 6170, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50499474, 'num_objects': 6165, 'num_object_clones': 0, 'num_object_copies': 24660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6165, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6167, 'num_write_kb': 49324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3976}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:54.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:44:54.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6112", 'reported_seq': 6152, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:49.698354+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:44:49.698354+0000', 'last_peered': '2024-09-16T04:44:49.698354+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:44:49.698354+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6112, 'log_dups_size': 0, 'ondisk_log_size': 6112, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50059910, 'num_objects': 6111, 'num_object_clones': 0, 'num_object_copies': 24444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6111, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6111, 'num_write_kb': 48887, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3983}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:54.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:44:54.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6189", 'reported_seq': 6264, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:49.738253+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:44:49.738253+0000', 'last_peered': '2024-09-16T04:44:49.738253+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:44:49.738253+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6189, 'log_dups_size': 0, 'ondisk_log_size': 6189, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50690694, 'num_objects': 6188, 'num_object_clones': 0, 'num_object_copies': 24752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6188, 'num_write_kb': 49503, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3963}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:54.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:44:54.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6324", 'reported_seq': 6376, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:50.095391+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:44:50.095391+0000', 'last_peered': '2024-09-16T04:44:50.095391+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:44:50.095391+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6324, 'log_dups_size': 0, 'ondisk_log_size': 6324, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51770636, 'num_objects': 6320, 'num_object_clones': 0, 'num_object_copies': 25280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6322, 'num_write_kb': 50572, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4086}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:44:54.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:44:54.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6150", 'reported_seq': 6230, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:49.681248+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:44:49.681248+0000', 'last_peered': '2024-09-16T04:44:49.681248+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:44:49.681248+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6150, 'log_dups_size': 0, 'ondisk_log_size': 6150, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50353420, 'num_objects': 6147, 'num_object_clones': 0, 'num_object_copies': 24588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6147, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6148, 'num_write_kb': 49181, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4015}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:44:54.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:44:54.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6095", 'reported_seq': 6161, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:50.205629+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:44:50.205629+0000', 'last_peered': '2024-09-16T04:44:50.205629+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:44:50.205629+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6095, 'log_dups_size': 0, 'ondisk_log_size': 6095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49886476, 'num_objects': 6090, 'num_object_clones': 0, 'num_object_copies': 24360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6090, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6093, 'num_write_kb': 48732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3851}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:44:54.422 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:44:55.176 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:55.176 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33014/398628 objects degraded (8.282%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":307,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":99657,"data_bytes":816350888,"bytes_used":2842058752,"bytes_avail":383704997888,"bytes_total":386547056640,"degraded_objects":33014,"degraded_total":398628,"degraded_ratio":0.08281906940806967,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":51186,"recovering_keys_per_sec":0,"num_objects_recovered":75,"num_bytes_recovered":614400,"num_keys_recovered":0,"write_bytes_sec":1830427,"read_op_per_sec":0,"write_op_per_sec":223},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:44:55.207 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:44:55.207 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:55.795 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:55.795 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:55.821 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-16T04:44:54.628772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":816350888,"num_objects":99657,"num_object_clones":0,"num_object_copies":398628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99684,"num_write_kb":797354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2565,"num_bytes_recovered":20774912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99711,"ondisk_log_size":99711,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2775448,"kb_used_data":1463640,"kb_used_omap":14,"kb_used_meta":1311729,"kb_avail":374711912,"statfs":{"total":386547056640,"available":383704997888,"internally_reserved":0,"allocated":1498767360,"data_stored":1498047484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1343211057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,15,19,19,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21315584,"num_objects":2602,"num_object_clones":0,"num_object_copies":10408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2602,"num_write_kb":20816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":614400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003128"},"pg_stats":[{"pgid":"1.f","version":"33'6250","reported_seq":6294,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.083915+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:50.083915+0000","last_peered":"2024-09-16T04:44:50.083915+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:50.083915+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6250,"log_dups_size":0,"ondisk_log_size":6250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51200000,"num_objects":6250,"num_object_clones":0,"num_object_copies":25000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6250,"num_write_kb":50000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6274","reported_seq":6339,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.261319+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:50.261319+0000","last_peered":"2024-09-16T04:44:50.261319+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:50.261319+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6274,"log_dups_size":0,"ondisk_log_size":6274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51361036,"num_objects":6270,"num_object_clones":0,"num_object_copies":25080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6272,"num_write_kb":50172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6261","reported_seq":6337,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:49.710955+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:49.710955+0000","last_peered":"2024-09-16T04:44:49.710955+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:49.710955+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51264134,"num_objects":6258,"num_object_clones":0,"num_object_copies":25032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":50070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4031}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6276","reported_seq":6343,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.135265+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:50.135265+0000","last_peered":"2024-09-16T04:44:50.135265+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:50.135265+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6276,"log_dups_size":0,"ondisk_log_size":6276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51377420,"num_objects":6272,"num_object_clones":0,"num_object_copies":25088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6274,"num_write_kb":50181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4043}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6303","reported_seq":6360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.108970+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:50.108970+0000","last_peered":"2024-09-16T04:44:50.108970+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:50.108970+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51606796,"num_objects":6300,"num_object_clones":0,"num_object_copies":25200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6301,"num_write_kb":50405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4017}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6087","reported_seq":6129,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.012915+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:50.012915+0000","last_peered":"2024-09-16T04:44:50.012915+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:50.012915+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49864704,"num_objects":6087,"num_object_clones":0,"num_object_copies":24348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6087,"num_write_kb":48696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3879}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6228","reported_seq":6294,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.171578+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:50.171578+0000","last_peered":"2024-09-16T04:44:50.171578+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:50.171578+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6228,"log_dups_size":0,"ondisk_log_size":6228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50967820,"num_objects":6222,"num_object_clones":0,"num_object_copies":24888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6226,"num_write_kb":49788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3996}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6363","reported_seq":10918,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:52.265881+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:52.265881+0000","last_peered":"2024-09-16T04:44:52.265881+0000","last_clean":"2024-09-16T04:44:52.265881+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:52.265881+0000","last_undegraded":"2024-09-16T04:44:52.265881+0000","last_fullsized":"2024-09-16T04:44:52.265881+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6363,"log_dups_size":0,"ondisk_log_size":6363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:56.943347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52081932,"num_objects":6358,"num_object_clones":0,"num_object_copies":25432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6361,"num_write_kb":50876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6311","reported_seq":6965,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.010555+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:50.010555+0000","last_peered":"2024-09-16T04:44:50.010555+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:50.010555+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6311,"log_dups_size":0,"ondisk_log_size":6311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51628568,"num_objects":6303,"num_object_clones":0,"num_object_copies":25212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6308,"num_write_kb":50441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":2441216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6318","reported_seq":6397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:49.708043+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:49.708043+0000","last_peered":"2024-09-16T04:44:49.708043+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:49.708043+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6318,"log_dups_size":0,"ondisk_log_size":6318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51737868,"num_objects":6316,"num_object_clones":0,"num_object_copies":25264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6316,"num_write_kb":50526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6170","reported_seq":6228,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.004124+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:50.004124+0000","last_peered":"2024-09-16T04:44:50.004124+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:50.004124+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6170,"log_dups_size":0,"ondisk_log_size":6170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50499474,"num_objects":6165,"num_object_clones":0,"num_object_copies":24660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6167,"num_write_kb":49324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3976}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6112","reported_seq":6152,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:49.698354+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:49.698354+0000","last_peered":"2024-09-16T04:44:49.698354+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:49.698354+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6112,"log_dups_size":0,"ondisk_log_size":6112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50059910,"num_objects":6111,"num_object_clones":0,"num_object_copies":24444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":48887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3983}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6189","reported_seq":6264,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:49.738253+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:49.738253+0000","last_peered":"2024-09-16T04:44:49.738253+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:49.738253+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6189,"log_dups_size":0,"ondisk_log_size":6189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50690694,"num_objects":6188,"num_object_clones":0,"num_object_copies":24752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6188,"num_write_kb":49503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3963}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6324","reported_seq":6376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.095391+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:50.095391+0000","last_peered":"2024-09-16T04:44:50.095391+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:50.095391+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6324,"log_dups_size":0,"ondisk_log_size":6324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51770636,"num_objects":6320,"num_object_clones":0,"num_object_copies":25280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6322,"num_write_kb":50572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4086}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6150","reported_seq":6230,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:49.681248+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:49.681248+0000","last_peered":"2024-09-16T04:44:49.681248+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:49.681248+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6150,"log_dups_size":0,"ondisk_log_size":6150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50353420,"num_objects":6147,"num_object_clones":0,"num_object_copies":24588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6148,"num_write_kb":49181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4015}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6095","reported_seq":6161,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:50.205629+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:50.205629+0000","last_peered":"2024-09-16T04:44:50.205629+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:50.205629+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6095,"log_dups_size":0,"ondisk_log_size":6095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49886476,"num_objects":6090,"num_object_clones":0,"num_object_copies":24360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6093,"num_write_kb":48732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3851}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":816350888,"num_objects":99657,"num_object_clones":0,"num_object_copies":398628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99684,"num_write_kb":797354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2565,"num_bytes_recovered":20774912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1499328512,"data_stored":1499328512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99711,"ondisk_log_size":99711,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755020,"kb_used_data":398460,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93616820,"statfs":{"total":96636764160,"available":95863623680,"internally_reserved":0,"allocated":408023040,"data_stored":407843071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365096655},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,3,6,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750040,"kb_used_data":398152,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93621800,"statfs":{"total":96636764160,"available":95868723200,"internally_reserved":0,"allocated":407707648,"data_stored":407527679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,9,1,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":507412,"kb_used_data":268868,"kb_used_omap":4,"kb_used_meta":238523,"kb_avail":93864428,"statfs":{"total":96636764160,"available":96117174272,"internally_reserved":0,"allocated":275320832,"data_stored":275140863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244248271},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762976,"kb_used_data":398160,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93608864,"statfs":{"total":96636764160,"available":95855476736,"internally_reserved":0,"allocated":407715840,"data_stored":407535871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":373550798},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":408150016,"data_stored":408150016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275460096,"data_stored":275460096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":407719936,"data_stored":407719936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":407998464,"data_stored":407998464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:58.823 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:59.410 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:44:59.410 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:44:59.438 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-16T04:44:58.629775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":823117504,"num_objects":100484,"num_object_clones":0,"num_object_copies":401936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100484,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101003,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2608,"num_bytes_recovered":21127168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101030,"ondisk_log_size":101030,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2809128,"kb_used_data":1479252,"kb_used_omap":14,"kb_used_meta":1329777,"kb_avail":374678232,"statfs":{"total":386547056640,"available":383670509568,"internally_reserved":0,"allocated":1514754048,"data_stored":1514034172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1361692209},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,8,8,19,9,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18227224,"num_objects":2226,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-78,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2226,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2718,"num_write_kb":19769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003090"},"pg_stats":[{"pgid":"1.f","version":"33'6337","reported_seq":6381,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.028611+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:55.028611+0000","last_peered":"2024-09-16T04:44:55.028611+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:55.028611+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51748864,"num_objects":6317,"num_object_clones":0,"num_object_copies":25268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6337,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6350","reported_seq":6416,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:54.843542+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:54.843542+0000","last_peered":"2024-09-16T04:44:54.843542+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:54.843542+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6350,"log_dups_size":0,"ondisk_log_size":6350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51803404,"num_objects":6324,"num_object_clones":0,"num_object_copies":25296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6348,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6343","reported_seq":6420,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:54.635493+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:54.635493+0000","last_peered":"2024-09-16T04:44:54.635493+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:54.635493+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6343,"log_dups_size":0,"ondisk_log_size":6343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51829406,"num_objects":6328,"num_object_clones":0,"num_object_copies":25312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6342,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4101}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6358","reported_seq":6423,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.293775+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:55.293775+0000","last_peered":"2024-09-16T04:44:55.293775+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:55.293775+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6358,"log_dups_size":0,"ondisk_log_size":6358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51705100,"num_objects":6312,"num_object_clones":0,"num_object_copies":25248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6356,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6382","reported_seq":6439,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:54.898029+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:54.898029+0000","last_peered":"2024-09-16T04:44:54.898029+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:54.898029+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6382,"log_dups_size":0,"ondisk_log_size":6382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52090124,"num_objects":6359,"num_object_clones":0,"num_object_copies":25436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6380,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4076}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6176","reported_seq":6219,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.133618+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:55.133618+0000","last_peered":"2024-09-16T04:44:55.133618+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:55.133618+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6176,"log_dups_size":0,"ondisk_log_size":6176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50266112,"num_objects":6136,"num_object_clones":0,"num_object_copies":24544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6176,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3928}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6322","reported_seq":6388,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.216907+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:55.216907+0000","last_peered":"2024-09-16T04:44:55.216907+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:55.216907+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51426572,"num_objects":6278,"num_object_clones":0,"num_object_copies":25112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6320,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4052}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6450","reported_seq":11006,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:57.274412+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:57.274412+0000","last_peered":"2024-09-16T04:44:57.274412+0000","last_clean":"2024-09-16T04:44:57.274412+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:57.274412+0000","last_undegraded":"2024-09-16T04:44:57.274412+0000","last_fullsized":"2024-09-16T04:44:57.274412+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:02.042021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51893516,"num_objects":6335,"num_object_clones":0,"num_object_copies":25340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6448,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6404","reported_seq":7145,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.112162+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:55.112162+0000","last_peered":"2024-09-16T04:44:55.112162+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:55.112162+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6404,"log_dups_size":0,"ondisk_log_size":6404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52029976,"num_objects":6352,"num_object_clones":0,"num_object_copies":25408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6401,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":344,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6390","reported_seq":6469,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:54.745425+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:54.745425+0000","last_peered":"2024-09-16T04:44:54.745425+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:54.745425+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52163852,"num_objects":6368,"num_object_clones":0,"num_object_copies":25472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6388,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6256","reported_seq":6314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.053852+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:55.053852+0000","last_peered":"2024-09-16T04:44:55.053852+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:55.053852+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51072914,"num_objects":6235,"num_object_clones":0,"num_object_copies":24940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6253,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4046}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6183","reported_seq":6223,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.047725+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:55.047725+0000","last_peered":"2024-09-16T04:44:55.047725+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:55.047725+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6183,"log_dups_size":0,"ondisk_log_size":6183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50363014,"num_objects":6148,"num_object_clones":0,"num_object_copies":24592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6182,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4020}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6275","reported_seq":6350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:54.818557+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:54.818557+0000","last_peered":"2024-09-16T04:44:54.818557+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:54.818557+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6275,"log_dups_size":0,"ondisk_log_size":6275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51264134,"num_objects":6258,"num_object_clones":0,"num_object_copies":25032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6274,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4033}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6405","reported_seq":6459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.015011+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:55.015011+0000","last_peered":"2024-09-16T04:44:55.015011+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:55.015011+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6405,"log_dups_size":0,"ondisk_log_size":6405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52237580,"num_objects":6377,"num_object_clones":0,"num_object_copies":25508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6403,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4143}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6231","reported_seq":6311,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:54.716046+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:54.716046+0000","last_peered":"2024-09-16T04:44:54.716046+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:54.716046+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6231,"log_dups_size":0,"ondisk_log_size":6231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50935052,"num_objects":6218,"num_object_clones":0,"num_object_copies":24872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6229,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6168","reported_seq":6232,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.230924+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:55.230924+0000","last_peered":"2024-09-16T04:44:55.230924+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:55.230924+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6168,"log_dups_size":0,"ondisk_log_size":6168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50287884,"num_objects":6139,"num_object_clones":0,"num_object_copies":24556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6166,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":823117504,"num_objects":100484,"num_object_clones":0,"num_object_copies":401936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100484,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101003,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2608,"num_bytes_recovered":21127168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1512390656,"data_stored":1512390656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101030,"ondisk_log_size":101030,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763268,"kb_used_data":402536,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93608572,"statfs":{"total":96636764160,"available":95855177728,"internally_reserved":0,"allocated":412196864,"data_stored":412016895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369356495},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,1,2,0,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758896,"kb_used_data":402840,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93612944,"statfs":{"total":96636764160,"available":95859654656,"internally_reserved":0,"allocated":412508160,"data_stored":412328191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,7,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515128,"kb_used_data":271016,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93856712,"statfs":{"total":96636764160,"available":96109273088,"internally_reserved":0,"allocated":277520384,"data_stored":277340415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249949903},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771836,"kb_used_data":402860,"kb_used_omap":4,"kb_used_meta":368955,"kb_avail":93600004,"statfs":{"total":96636764160,"available":95846404096,"internally_reserved":0,"allocated":412528640,"data_stored":412348671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":377810638},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,7,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":411533312,"data_stored":411533312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":277213184,"data_stored":277213184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":411971584,"data_stored":411971584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":411672576,"data_stored":411672576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:59.439 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:00.070 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:00.071 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:00.104 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-16T04:44:58.629775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":823117504,"num_objects":100484,"num_object_clones":0,"num_object_copies":401936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100484,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101003,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2608,"num_bytes_recovered":21127168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101030,"ondisk_log_size":101030,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2809128,"kb_used_data":1479252,"kb_used_omap":14,"kb_used_meta":1329777,"kb_avail":374678232,"statfs":{"total":386547056640,"available":383670509568,"internally_reserved":0,"allocated":1514754048,"data_stored":1514034172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1361692209},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,8,8,19,9,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18227224,"num_objects":2226,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-78,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2226,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2718,"num_write_kb":19769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003090"},"pg_stats":[{"pgid":"1.f","version":"33'6337","reported_seq":6381,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.028611+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:44:55.028611+0000","last_peered":"2024-09-16T04:44:55.028611+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:44:55.028611+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51748864,"num_objects":6317,"num_object_clones":0,"num_object_copies":25268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6337,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6350","reported_seq":6416,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:54.843542+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:44:54.843542+0000","last_peered":"2024-09-16T04:44:54.843542+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:44:54.843542+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6350,"log_dups_size":0,"ondisk_log_size":6350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51803404,"num_objects":6324,"num_object_clones":0,"num_object_copies":25296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6348,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6343","reported_seq":6420,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:54.635493+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:54.635493+0000","last_peered":"2024-09-16T04:44:54.635493+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:54.635493+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6343,"log_dups_size":0,"ondisk_log_size":6343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51829406,"num_objects":6328,"num_object_clones":0,"num_object_copies":25312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6342,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4101}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6358","reported_seq":6423,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.293775+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:44:55.293775+0000","last_peered":"2024-09-16T04:44:55.293775+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:44:55.293775+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6358,"log_dups_size":0,"ondisk_log_size":6358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51705100,"num_objects":6312,"num_object_clones":0,"num_object_copies":25248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6356,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6382","reported_seq":6439,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:54.898029+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:44:54.898029+0000","last_peered":"2024-09-16T04:44:54.898029+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:44:54.898029+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6382,"log_dups_size":0,"ondisk_log_size":6382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52090124,"num_objects":6359,"num_object_clones":0,"num_object_copies":25436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6380,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4076}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6176","reported_seq":6219,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.133618+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:44:55.133618+0000","last_peered":"2024-09-16T04:44:55.133618+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:44:55.133618+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6176,"log_dups_size":0,"ondisk_log_size":6176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50266112,"num_objects":6136,"num_object_clones":0,"num_object_copies":24544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6176,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3928}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6322","reported_seq":6388,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.216907+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:44:55.216907+0000","last_peered":"2024-09-16T04:44:55.216907+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:44:55.216907+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51426572,"num_objects":6278,"num_object_clones":0,"num_object_copies":25112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6320,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4052}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6450","reported_seq":11006,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:57.274412+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:57.274412+0000","last_peered":"2024-09-16T04:44:57.274412+0000","last_clean":"2024-09-16T04:44:57.274412+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:57.274412+0000","last_undegraded":"2024-09-16T04:44:57.274412+0000","last_fullsized":"2024-09-16T04:44:57.274412+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:02.042021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51893516,"num_objects":6335,"num_object_clones":0,"num_object_copies":25340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6448,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6404","reported_seq":7145,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.112162+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:44:55.112162+0000","last_peered":"2024-09-16T04:44:55.112162+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:44:55.112162+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6404,"log_dups_size":0,"ondisk_log_size":6404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52029976,"num_objects":6352,"num_object_clones":0,"num_object_copies":25408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6401,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":344,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4353}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6390","reported_seq":6469,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:54.745425+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:54.745425+0000","last_peered":"2024-09-16T04:44:54.745425+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:54.745425+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52163852,"num_objects":6368,"num_object_clones":0,"num_object_copies":25472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6388,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6256","reported_seq":6314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.053852+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:44:55.053852+0000","last_peered":"2024-09-16T04:44:55.053852+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:44:55.053852+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51072914,"num_objects":6235,"num_object_clones":0,"num_object_copies":24940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6253,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4046}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6183","reported_seq":6223,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.047725+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:44:55.047725+0000","last_peered":"2024-09-16T04:44:55.047725+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:44:55.047725+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6183,"log_dups_size":0,"ondisk_log_size":6183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50363014,"num_objects":6148,"num_object_clones":0,"num_object_copies":24592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6182,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4020}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6275","reported_seq":6350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:54.818557+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:54.818557+0000","last_peered":"2024-09-16T04:44:54.818557+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:54.818557+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6275,"log_dups_size":0,"ondisk_log_size":6275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51264134,"num_objects":6258,"num_object_clones":0,"num_object_copies":25032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6274,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4033}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6405","reported_seq":6459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.015011+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:55.015011+0000","last_peered":"2024-09-16T04:44:55.015011+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:55.015011+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6405,"log_dups_size":0,"ondisk_log_size":6405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52237580,"num_objects":6377,"num_object_clones":0,"num_object_copies":25508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6403,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4143}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6231","reported_seq":6311,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:54.716046+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:54.716046+0000","last_peered":"2024-09-16T04:44:54.716046+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:54.716046+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6231,"log_dups_size":0,"ondisk_log_size":6231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50935052,"num_objects":6218,"num_object_clones":0,"num_object_copies":24872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6229,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6168","reported_seq":6232,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:55.230924+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:44:55.230924+0000","last_peered":"2024-09-16T04:44:55.230924+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:44:55.230924+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6168,"log_dups_size":0,"ondisk_log_size":6168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50287884,"num_objects":6139,"num_object_clones":0,"num_object_copies":24556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6166,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":823117504,"num_objects":100484,"num_object_clones":0,"num_object_copies":401936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100484,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101003,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2608,"num_bytes_recovered":21127168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1512390656,"data_stored":1512390656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101030,"ondisk_log_size":101030,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763268,"kb_used_data":402536,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93608572,"statfs":{"total":96636764160,"available":95855177728,"internally_reserved":0,"allocated":412196864,"data_stored":412016895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369356495},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,1,2,0,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758896,"kb_used_data":402840,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93612944,"statfs":{"total":96636764160,"available":95859654656,"internally_reserved":0,"allocated":412508160,"data_stored":412328191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,7,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515128,"kb_used_data":271016,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93856712,"statfs":{"total":96636764160,"available":96109273088,"internally_reserved":0,"allocated":277520384,"data_stored":277340415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249949903},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771836,"kb_used_data":402860,"kb_used_omap":4,"kb_used_meta":368955,"kb_avail":93600004,"statfs":{"total":96636764160,"available":95846404096,"internally_reserved":0,"allocated":412528640,"data_stored":412348671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":377810638},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,7,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":411533312,"data_stored":411533312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":277213184,"data_stored":277213184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":411971584,"data_stored":411971584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":411672576,"data_stored":411672576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:00.105 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:45:00.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6337", 'reported_seq': 6381, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:55.028611+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:44:55.028611+0000', 'last_peered': '2024-09-16T04:44:55.028611+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:44:55.028611+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6337, 'log_dups_size': 0, 'ondisk_log_size': 6337, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51748864, 'num_objects': 6317, 'num_object_clones': 0, 'num_object_copies': 25268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6317, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6337, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4098}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:00.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:45:00.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6350", 'reported_seq': 6416, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:54.843542+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:44:54.843542+0000', 'last_peered': '2024-09-16T04:44:54.843542+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:44:54.843542+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6350, 'log_dups_size': 0, 'ondisk_log_size': 6350, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51803404, 'num_objects': 6324, 'num_object_clones': 0, 'num_object_copies': 25296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6348, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4113}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:00.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:45:00.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6343", 'reported_seq': 6420, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:54.635493+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:44:54.635493+0000', 'last_peered': '2024-09-16T04:44:54.635493+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:44:54.635493+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6343, 'log_dups_size': 0, 'ondisk_log_size': 6343, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51829406, 'num_objects': 6328, 'num_object_clones': 0, 'num_object_copies': 25312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6328, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6342, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4101}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:00.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:45:00.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6358", 'reported_seq': 6423, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:55.293775+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:44:55.293775+0000', 'last_peered': '2024-09-16T04:44:55.293775+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:44:55.293775+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6358, 'log_dups_size': 0, 'ondisk_log_size': 6358, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51705100, 'num_objects': 6312, 'num_object_clones': 0, 'num_object_copies': 25248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6312, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6356, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:00.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:45:00.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6382", 'reported_seq': 6439, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:54.898029+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:44:54.898029+0000', 'last_peered': '2024-09-16T04:44:54.898029+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:44:54.898029+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6382, 'log_dups_size': 0, 'ondisk_log_size': 6382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52090124, 'num_objects': 6359, 'num_object_clones': 0, 'num_object_copies': 25436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6380, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4076}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:00.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:45:00.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6176", 'reported_seq': 6219, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:55.133618+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:44:55.133618+0000', 'last_peered': '2024-09-16T04:44:55.133618+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:44:55.133618+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6176, 'log_dups_size': 0, 'ondisk_log_size': 6176, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50266112, 'num_objects': 6136, 'num_object_clones': 0, 'num_object_copies': 24544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6176, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3928}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:00.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:45:00.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6322", 'reported_seq': 6388, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:55.216907+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:44:55.216907+0000', 'last_peered': '2024-09-16T04:44:55.216907+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:44:55.216907+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6322, 'log_dups_size': 0, 'ondisk_log_size': 6322, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51426572, 'num_objects': 6278, 'num_object_clones': 0, 'num_object_copies': 25112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6320, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4052}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:00.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:45:00.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6404", 'reported_seq': 7145, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:55.112162+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:44:55.112162+0000', 'last_peered': '2024-09-16T04:44:55.112162+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:44:55.112162+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6404, 'log_dups_size': 0, 'ondisk_log_size': 6404, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52029976, 'num_objects': 6352, 'num_object_clones': 0, 'num_object_copies': 25408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1999, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6352, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6401, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 344, 'num_bytes_recovered': 2793472, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4353}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:00.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:45:00.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6390", 'reported_seq': 6469, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:54.745425+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:44:54.745425+0000', 'last_peered': '2024-09-16T04:44:54.745425+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:44:54.745425+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6390, 'log_dups_size': 0, 'ondisk_log_size': 6390, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52163852, 'num_objects': 6368, 'num_object_clones': 0, 'num_object_copies': 25472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6368, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6388, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4168}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:00.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:45:00.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6256", 'reported_seq': 6314, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:55.053852+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:44:55.053852+0000', 'last_peered': '2024-09-16T04:44:55.053852+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:44:55.053852+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6256, 'log_dups_size': 0, 'ondisk_log_size': 6256, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51072914, 'num_objects': 6235, 'num_object_clones': 0, 'num_object_copies': 24940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6235, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6253, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4046}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:00.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:45:00.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6183", 'reported_seq': 6223, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:55.047725+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:44:55.047725+0000', 'last_peered': '2024-09-16T04:44:55.047725+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:44:55.047725+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6183, 'log_dups_size': 0, 'ondisk_log_size': 6183, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50363014, 'num_objects': 6148, 'num_object_clones': 0, 'num_object_copies': 24592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6148, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6182, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4020}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:00.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:45:00.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6275", 'reported_seq': 6350, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:54.818557+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:44:54.818557+0000', 'last_peered': '2024-09-16T04:44:54.818557+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:44:54.818557+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6275, 'log_dups_size': 0, 'ondisk_log_size': 6275, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51264134, 'num_objects': 6258, 'num_object_clones': 0, 'num_object_copies': 25032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6274, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4033}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:00.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:45:00.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6405", 'reported_seq': 6459, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:55.015011+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:44:55.015011+0000', 'last_peered': '2024-09-16T04:44:55.015011+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:44:55.015011+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6405, 'log_dups_size': 0, 'ondisk_log_size': 6405, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52237580, 'num_objects': 6377, 'num_object_clones': 0, 'num_object_copies': 25508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6403, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4143}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:00.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:45:00.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6231", 'reported_seq': 6311, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:54.716046+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:44:54.716046+0000', 'last_peered': '2024-09-16T04:44:54.716046+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:44:54.716046+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6231, 'log_dups_size': 0, 'ondisk_log_size': 6231, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50935052, 'num_objects': 6218, 'num_object_clones': 0, 'num_object_copies': 24872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6218, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6229, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4086}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:00.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:45:00.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6168", 'reported_seq': 6232, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:55.230924+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:44:55.230924+0000', 'last_peered': '2024-09-16T04:44:55.230924+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:44:55.230924+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6168, 'log_dups_size': 0, 'ondisk_log_size': 6168, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50287884, 'num_objects': 6139, 'num_object_clones': 0, 'num_object_copies': 24556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6139, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6166, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3900}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:00.108 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:45:00.788 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:00.788 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32930/396940 objects degraded (8.296%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":313,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":99235,"data_bytes":812885696,"bytes_used":2855858176,"bytes_avail":383691198464,"bytes_total":386547056640,"degraded_objects":32930,"degraded_total":396940,"degraded_ratio":0.082959641255605371,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":79851,"recovering_keys_per_sec":0,"num_objects_recovered":119,"num_bytes_recovered":958464,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":1628518,"read_op_per_sec":0,"write_op_per_sec":323},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:45:00.824 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:45:00.825 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:01.385 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:01.386 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:01.410 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-09-16T04:45:00.630252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":812885696,"num_objects":99235,"num_object_clones":0,"num_object_copies":396940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99235,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102252,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2649,"num_bytes_recovered":21446656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102279,"ondisk_log_size":102279,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788924,"kb_used_data":1463192,"kb_used_omap":14,"kb_used_meta":1325617,"kb_avail":374698436,"statfs":{"total":386547056640,"available":383691198464,"internally_reserved":0,"allocated":1498308608,"data_stored":1497588732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1357432369},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,1,5,9,14,11,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7299096,"num_objects":892,"num_object_clones":0,"num_object_copies":3568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":892,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3882,"num_write_kb":19089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003017"},"pg_stats":[{"pgid":"1.f","version":"33'6416","reported_seq":6460,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.130060+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:00.130060+0000","last_peered":"2024-09-16T04:45:00.130060+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:00.130060+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6416,"log_dups_size":0,"ondisk_log_size":6416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51101696,"num_objects":6238,"num_object_clones":0,"num_object_copies":24952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6416,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6450","reported_seq":6516,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.241269+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:00.241269+0000","last_peered":"2024-09-16T04:45:00.241269+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:00.241269+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50984204,"num_objects":6224,"num_object_clones":0,"num_object_copies":24896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6448,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4013}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6433","reported_seq":6510,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:59.810856+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:59.810856+0000","last_peered":"2024-09-16T04:44:59.810856+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:59.810856+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6433,"log_dups_size":0,"ondisk_log_size":6433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51092126,"num_objects":6238,"num_object_clones":0,"num_object_copies":24952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6238,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6432,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4011}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6429","reported_seq":6498,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.057571+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:00.057571+0000","last_peered":"2024-09-16T04:45:00.057571+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:00.057571+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6429,"log_dups_size":0,"ondisk_log_size":6429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51123468,"num_objects":6241,"num_object_clones":0,"num_object_copies":24964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6427,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4013}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6469","reported_seq":6526,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.030414+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:00.030414+0000","last_peered":"2024-09-16T04:45:00.030414+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:00.030414+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6469,"log_dups_size":0,"ondisk_log_size":6469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51377420,"num_objects":6272,"num_object_clones":0,"num_object_copies":25088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6260","reported_seq":6304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.121374+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:00.121374+0000","last_peered":"2024-09-16T04:45:00.121374+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:00.121374+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49577984,"num_objects":6052,"num_object_clones":0,"num_object_copies":24208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3844}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6410","reported_seq":6476,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.202313+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:00.202313+0000","last_peered":"2024-09-16T04:45:00.202313+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:00.202313+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50705676,"num_objects":6190,"num_object_clones":0,"num_object_copies":24760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6408,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3964}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6450","reported_seq":11006,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:44:57.274412+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:44:57.274412+0000","last_peered":"2024-09-16T04:44:57.274412+0000","last_clean":"2024-09-16T04:44:57.274412+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:44:57.274412+0000","last_undegraded":"2024-09-16T04:44:57.274412+0000","last_fullsized":"2024-09-16T04:44:57.274412+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:02.042021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51893516,"num_objects":6335,"num_object_clones":0,"num_object_copies":25340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6448,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6506","reported_seq":7323,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.180380+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:00.180380+0000","last_peered":"2024-09-16T04:45:00.180380+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:00.180380+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6506,"log_dups_size":0,"ondisk_log_size":6506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51194392,"num_objects":6250,"num_object_clones":0,"num_object_copies":25000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6503,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":3112960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6472","reported_seq":6551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:59.656718+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:59.656718+0000","last_peered":"2024-09-16T04:44:59.656718+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:59.656718+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6472,"log_dups_size":0,"ondisk_log_size":6472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51492108,"num_objects":6286,"num_object_clones":0,"num_object_copies":25144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6470,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4086}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6332","reported_seq":6391,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.186862+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:00.186862+0000","last_peered":"2024-09-16T04:45:00.186862+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:00.186862+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6332,"log_dups_size":0,"ondisk_log_size":6332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50450322,"num_objects":6159,"num_object_clones":0,"num_object_copies":24636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6329,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6245","reported_seq":6285,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.043940+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:00.043940+0000","last_peered":"2024-09-16T04:45:00.043940+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:00.043940+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6245,"log_dups_size":0,"ondisk_log_size":6245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49855110,"num_objects":6086,"num_object_clones":0,"num_object_copies":24344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6244,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3958}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6354","reported_seq":6426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:59.759957+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:59.759957+0000","last_peered":"2024-09-16T04:44:59.759957+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:59.759957+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6354,"log_dups_size":0,"ondisk_log_size":6354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50616966,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6353,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6489","reported_seq":6543,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:59.936355+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:59.936355+0000","last_peered":"2024-09-16T04:44:59.936355+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:59.936355+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6489,"log_dups_size":0,"ondisk_log_size":6489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51549452,"num_objects":6293,"num_object_clones":0,"num_object_copies":25172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6487,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4059}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6316","reported_seq":6394,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:59.696788+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:59.696788+0000","last_peered":"2024-09-16T04:44:59.696788+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:59.696788+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50238732,"num_objects":6133,"num_object_clones":0,"num_object_copies":24532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6314,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4001}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6248","reported_seq":6314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.185045+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:00.185045+0000","last_peered":"2024-09-16T04:45:00.185045+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:00.185045+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6248,"log_dups_size":0,"ondisk_log_size":6248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49632524,"num_objects":6059,"num_object_clones":0,"num_object_copies":24236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6246,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3820}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":812885696,"num_objects":99235,"num_object_clones":0,"num_object_copies":396940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99235,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102252,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2649,"num_bytes_recovered":21446656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1495748608,"data_stored":1495748608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102279,"ondisk_log_size":102279,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762076,"kb_used_data":397188,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93609764,"statfs":{"total":96636764160,"available":95856398336,"internally_reserved":0,"allocated":406720512,"data_stored":406540543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373616335},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,0,3,8,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757752,"kb_used_data":397520,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93614088,"statfs":{"total":96636764160,"available":95860826112,"internally_reserved":0,"allocated":407060480,"data_stored":406880511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515128,"kb_used_data":271016,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93856712,"statfs":{"total":96636764160,"available":96109273088,"internally_reserved":0,"allocated":277520384,"data_stored":277340415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":249949903},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753968,"kb_used_data":397468,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93617872,"statfs":{"total":96636764160,"available":95864700928,"internally_reserved":0,"allocated":407007232,"data_stored":406827263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":365031118},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,7,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":406016000,"data_stored":406016000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":277213184,"data_stored":277213184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":406396928,"data_stored":406396928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":406122496,"data_stored":406122496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:04.412 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:05.009 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:05.009 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:05.041 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-16T04:45:04.632442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":812197568,"num_objects":99151,"num_object_clones":0,"num_object_copies":396604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99151,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102336,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2649,"num_bytes_recovered":21446656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102363,"ondisk_log_size":102363,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2792508,"kb_used_data":1462616,"kb_used_omap":14,"kb_used_meta":1329777,"kb_avail":374694852,"statfs":{"total":386547056640,"available":383687528448,"internally_reserved":0,"allocated":1497718784,"data_stored":1496998908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1361692209},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,2,8,16,14,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4153320,"num_objects":-506,"num_object_clones":0,"num_object_copies":-2024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-506,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2652,"num_write_kb":8577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004153"},"pg_stats":[{"pgid":"1.f","version":"33'6416","reported_seq":6460,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.130060+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:00.130060+0000","last_peered":"2024-09-16T04:45:00.130060+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:00.130060+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6416,"log_dups_size":0,"ondisk_log_size":6416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51101696,"num_objects":6238,"num_object_clones":0,"num_object_copies":24952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6416,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6450","reported_seq":6516,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.241269+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:00.241269+0000","last_peered":"2024-09-16T04:45:00.241269+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:00.241269+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50984204,"num_objects":6224,"num_object_clones":0,"num_object_copies":24896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6448,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4013}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6433","reported_seq":6510,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:59.810856+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:59.810856+0000","last_peered":"2024-09-16T04:44:59.810856+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:59.810856+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6433,"log_dups_size":0,"ondisk_log_size":6433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51092126,"num_objects":6238,"num_object_clones":0,"num_object_copies":24952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6238,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6432,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4011}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6429","reported_seq":6498,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.057571+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:00.057571+0000","last_peered":"2024-09-16T04:45:00.057571+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:00.057571+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6429,"log_dups_size":0,"ondisk_log_size":6429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51123468,"num_objects":6241,"num_object_clones":0,"num_object_copies":24964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6427,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4013}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6469","reported_seq":6526,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.030414+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:00.030414+0000","last_peered":"2024-09-16T04:45:00.030414+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:00.030414+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6469,"log_dups_size":0,"ondisk_log_size":6469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51377420,"num_objects":6272,"num_object_clones":0,"num_object_copies":25088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6260","reported_seq":6304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.121374+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:00.121374+0000","last_peered":"2024-09-16T04:45:00.121374+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:00.121374+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49577984,"num_objects":6052,"num_object_clones":0,"num_object_copies":24208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3844}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6410","reported_seq":6476,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.202313+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:00.202313+0000","last_peered":"2024-09-16T04:45:00.202313+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:00.202313+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50705676,"num_objects":6190,"num_object_clones":0,"num_object_copies":24760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6408,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3964}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6534","reported_seq":11089,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:02.261161+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:02.261161+0000","last_peered":"2024-09-16T04:45:02.261161+0000","last_clean":"2024-09-16T04:45:02.261161+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:02.261161+0000","last_undegraded":"2024-09-16T04:45:02.261161+0000","last_fullsized":"2024-09-16T04:45:02.261161+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6534,"log_dups_size":0,"ondisk_log_size":6534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:07.100481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51205388,"num_objects":6251,"num_object_clones":0,"num_object_copies":25004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6532,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6506","reported_seq":7323,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.180380+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:00.180380+0000","last_peered":"2024-09-16T04:45:00.180380+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:00.180380+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6506,"log_dups_size":0,"ondisk_log_size":6506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51194392,"num_objects":6250,"num_object_clones":0,"num_object_copies":25000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6503,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":3112960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6472","reported_seq":6551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:59.656718+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:59.656718+0000","last_peered":"2024-09-16T04:44:59.656718+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:59.656718+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6472,"log_dups_size":0,"ondisk_log_size":6472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51492108,"num_objects":6286,"num_object_clones":0,"num_object_copies":25144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6470,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4086}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6332","reported_seq":6391,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.186862+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:00.186862+0000","last_peered":"2024-09-16T04:45:00.186862+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:00.186862+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6332,"log_dups_size":0,"ondisk_log_size":6332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50450322,"num_objects":6159,"num_object_clones":0,"num_object_copies":24636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6329,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6245","reported_seq":6285,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.043940+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:00.043940+0000","last_peered":"2024-09-16T04:45:00.043940+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:00.043940+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6245,"log_dups_size":0,"ondisk_log_size":6245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49855110,"num_objects":6086,"num_object_clones":0,"num_object_copies":24344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6244,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3958}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6354","reported_seq":6426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:59.759957+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:59.759957+0000","last_peered":"2024-09-16T04:44:59.759957+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:59.759957+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6354,"log_dups_size":0,"ondisk_log_size":6354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50616966,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6353,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6489","reported_seq":6543,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:59.936355+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:59.936355+0000","last_peered":"2024-09-16T04:44:59.936355+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:59.936355+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6489,"log_dups_size":0,"ondisk_log_size":6489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51549452,"num_objects":6293,"num_object_clones":0,"num_object_copies":25172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6487,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4059}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6316","reported_seq":6394,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:59.696788+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:59.696788+0000","last_peered":"2024-09-16T04:44:59.696788+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:59.696788+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50238732,"num_objects":6133,"num_object_clones":0,"num_object_copies":24532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6314,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4001}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6248","reported_seq":6314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.185045+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:00.185045+0000","last_peered":"2024-09-16T04:45:00.185045+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:00.185045+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6248,"log_dups_size":0,"ondisk_log_size":6248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49632524,"num_objects":6059,"num_object_clones":0,"num_object_copies":24236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6246,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3820}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":812197568,"num_objects":99151,"num_object_clones":0,"num_object_copies":396604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99151,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102336,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2649,"num_bytes_recovered":21446656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1495175168,"data_stored":1495175168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102363,"ondisk_log_size":102363,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762076,"kb_used_data":397188,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93609764,"statfs":{"total":96636764160,"available":95856398336,"internally_reserved":0,"allocated":406720512,"data_stored":406540543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373616335},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,0,3,8,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757752,"kb_used_data":397520,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93614088,"statfs":{"total":96636764160,"available":95860826112,"internally_reserved":0,"allocated":407060480,"data_stored":406880511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051628,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518712,"kb_used_data":270440,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93853128,"statfs":{"total":96636764160,"available":96105603072,"internally_reserved":0,"allocated":276930560,"data_stored":276750591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254209743},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753968,"kb_used_data":397468,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93617872,"statfs":{"total":96636764160,"available":95864700928,"internally_reserved":0,"allocated":407007232,"data_stored":406827263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":365031118},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,7,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":406016000,"data_stored":406016000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276639744,"data_stored":276639744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":406396928,"data_stored":406396928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":406122496,"data_stored":406122496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:05.042 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:05.636 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:05.637 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:05.661 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-16T04:45:04.632442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":812197568,"num_objects":99151,"num_object_clones":0,"num_object_copies":396604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99151,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102336,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2649,"num_bytes_recovered":21446656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102363,"ondisk_log_size":102363,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2792508,"kb_used_data":1462616,"kb_used_omap":14,"kb_used_meta":1329777,"kb_avail":374694852,"statfs":{"total":386547056640,"available":383687528448,"internally_reserved":0,"allocated":1497718784,"data_stored":1496998908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1361692209},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,2,8,16,14,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4153320,"num_objects":-506,"num_object_clones":0,"num_object_copies":-2024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-506,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2652,"num_write_kb":8577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004153"},"pg_stats":[{"pgid":"1.f","version":"33'6416","reported_seq":6460,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.130060+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:00.130060+0000","last_peered":"2024-09-16T04:45:00.130060+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:00.130060+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6416,"log_dups_size":0,"ondisk_log_size":6416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51101696,"num_objects":6238,"num_object_clones":0,"num_object_copies":24952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6416,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6450","reported_seq":6516,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.241269+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:00.241269+0000","last_peered":"2024-09-16T04:45:00.241269+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:00.241269+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50984204,"num_objects":6224,"num_object_clones":0,"num_object_copies":24896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6448,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4013}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6433","reported_seq":6510,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:59.810856+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:44:59.810856+0000","last_peered":"2024-09-16T04:44:59.810856+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:44:59.810856+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6433,"log_dups_size":0,"ondisk_log_size":6433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51092126,"num_objects":6238,"num_object_clones":0,"num_object_copies":24952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6238,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6432,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":4011}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6429","reported_seq":6498,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.057571+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:00.057571+0000","last_peered":"2024-09-16T04:45:00.057571+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:00.057571+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6429,"log_dups_size":0,"ondisk_log_size":6429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51123468,"num_objects":6241,"num_object_clones":0,"num_object_copies":24964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6427,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":4013}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6469","reported_seq":6526,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.030414+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:00.030414+0000","last_peered":"2024-09-16T04:45:00.030414+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:00.030414+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6469,"log_dups_size":0,"ondisk_log_size":6469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51377420,"num_objects":6272,"num_object_clones":0,"num_object_copies":25088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6260","reported_seq":6304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.121374+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:00.121374+0000","last_peered":"2024-09-16T04:45:00.121374+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:00.121374+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49577984,"num_objects":6052,"num_object_clones":0,"num_object_copies":24208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3844}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6410","reported_seq":6476,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.202313+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:00.202313+0000","last_peered":"2024-09-16T04:45:00.202313+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:00.202313+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50705676,"num_objects":6190,"num_object_clones":0,"num_object_copies":24760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6408,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3964}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6534","reported_seq":11089,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:02.261161+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:02.261161+0000","last_peered":"2024-09-16T04:45:02.261161+0000","last_clean":"2024-09-16T04:45:02.261161+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:02.261161+0000","last_undegraded":"2024-09-16T04:45:02.261161+0000","last_fullsized":"2024-09-16T04:45:02.261161+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6534,"log_dups_size":0,"ondisk_log_size":6534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:07.100481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51205388,"num_objects":6251,"num_object_clones":0,"num_object_copies":25004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6532,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6506","reported_seq":7323,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.180380+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:00.180380+0000","last_peered":"2024-09-16T04:45:00.180380+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:00.180380+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6506,"log_dups_size":0,"ondisk_log_size":6506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51194392,"num_objects":6250,"num_object_clones":0,"num_object_copies":25000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6503,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":3112960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6472","reported_seq":6551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:59.656718+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:44:59.656718+0000","last_peered":"2024-09-16T04:44:59.656718+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:44:59.656718+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6472,"log_dups_size":0,"ondisk_log_size":6472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51492108,"num_objects":6286,"num_object_clones":0,"num_object_copies":25144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6470,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4086}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6332","reported_seq":6391,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.186862+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:00.186862+0000","last_peered":"2024-09-16T04:45:00.186862+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:00.186862+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6332,"log_dups_size":0,"ondisk_log_size":6332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50450322,"num_objects":6159,"num_object_clones":0,"num_object_copies":24636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6329,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6245","reported_seq":6285,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.043940+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:00.043940+0000","last_peered":"2024-09-16T04:45:00.043940+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:00.043940+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6245,"log_dups_size":0,"ondisk_log_size":6245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49855110,"num_objects":6086,"num_object_clones":0,"num_object_copies":24344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6244,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3958}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6354","reported_seq":6426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:59.759957+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:44:59.759957+0000","last_peered":"2024-09-16T04:44:59.759957+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:44:59.759957+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6354,"log_dups_size":0,"ondisk_log_size":6354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50616966,"num_objects":6179,"num_object_clones":0,"num_object_copies":24716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6353,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6489","reported_seq":6543,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:59.936355+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:44:59.936355+0000","last_peered":"2024-09-16T04:44:59.936355+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:44:59.936355+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6489,"log_dups_size":0,"ondisk_log_size":6489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51549452,"num_objects":6293,"num_object_clones":0,"num_object_copies":25172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6487,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4059}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6316","reported_seq":6394,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:44:59.696788+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:44:59.696788+0000","last_peered":"2024-09-16T04:44:59.696788+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:44:59.696788+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50238732,"num_objects":6133,"num_object_clones":0,"num_object_copies":24532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6314,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4001}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6248","reported_seq":6314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:00.185045+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:00.185045+0000","last_peered":"2024-09-16T04:45:00.185045+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:00.185045+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6248,"log_dups_size":0,"ondisk_log_size":6248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49632524,"num_objects":6059,"num_object_clones":0,"num_object_copies":24236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6246,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3820}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":812197568,"num_objects":99151,"num_object_clones":0,"num_object_copies":396604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99151,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102336,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2649,"num_bytes_recovered":21446656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1495175168,"data_stored":1495175168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102363,"ondisk_log_size":102363,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762076,"kb_used_data":397188,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93609764,"statfs":{"total":96636764160,"available":95856398336,"internally_reserved":0,"allocated":406720512,"data_stored":406540543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373616335},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,0,3,8,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757752,"kb_used_data":397520,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93614088,"statfs":{"total":96636764160,"available":95860826112,"internally_reserved":0,"allocated":407060480,"data_stored":406880511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051628,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518712,"kb_used_data":270440,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93853128,"statfs":{"total":96636764160,"available":96105603072,"internally_reserved":0,"allocated":276930560,"data_stored":276750591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254209743},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753968,"kb_used_data":397468,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93617872,"statfs":{"total":96636764160,"available":95864700928,"internally_reserved":0,"allocated":407007232,"data_stored":406827263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":365031118},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,7,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":406016000,"data_stored":406016000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276639744,"data_stored":276639744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":406396928,"data_stored":406396928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":406122496,"data_stored":406122496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:05.662 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:45:05.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6416", 'reported_seq': 6460, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:00.130060+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:45:00.130060+0000', 'last_peered': '2024-09-16T04:45:00.130060+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:45:00.130060+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6416, 'log_dups_size': 0, 'ondisk_log_size': 6416, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51101696, 'num_objects': 6238, 'num_object_clones': 0, 'num_object_copies': 24952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6416, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4019}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:05.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:45:05.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6450", 'reported_seq': 6516, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:00.241269+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:45:00.241269+0000', 'last_peered': '2024-09-16T04:45:00.241269+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:45:00.241269+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6450, 'log_dups_size': 0, 'ondisk_log_size': 6450, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50984204, 'num_objects': 6224, 'num_object_clones': 0, 'num_object_copies': 24896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2213, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6224, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6448, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4013}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:05.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:45:05.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6433", 'reported_seq': 6510, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:59.810856+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:44:59.810856+0000', 'last_peered': '2024-09-16T04:44:59.810856+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:44:59.810856+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6433, 'log_dups_size': 0, 'ondisk_log_size': 6433, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51092126, 'num_objects': 6238, 'num_object_clones': 0, 'num_object_copies': 24952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6238, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6432, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 4011}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:05.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:45:05.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6429", 'reported_seq': 6498, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:00.057571+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:45:00.057571+0000', 'last_peered': '2024-09-16T04:45:00.057571+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:45:00.057571+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6429, 'log_dups_size': 0, 'ondisk_log_size': 6429, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51123468, 'num_objects': 6241, 'num_object_clones': 0, 'num_object_copies': 24964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6427, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 4013}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:05.663 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:45:05.663 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6469", 'reported_seq': 6526, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:00.030414+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:45:00.030414+0000', 'last_peered': '2024-09-16T04:45:00.030414+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:45:00.030414+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6469, 'log_dups_size': 0, 'ondisk_log_size': 6469, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51377420, 'num_objects': 6272, 'num_object_clones': 0, 'num_object_copies': 25088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6467, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3989}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:05.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:45:05.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6260", 'reported_seq': 6304, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:00.121374+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:45:00.121374+0000', 'last_peered': '2024-09-16T04:45:00.121374+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:45:00.121374+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6260, 'log_dups_size': 0, 'ondisk_log_size': 6260, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49577984, 'num_objects': 6052, 'num_object_clones': 0, 'num_object_copies': 24208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6052, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6260, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3844}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:05.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:45:05.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6410", 'reported_seq': 6476, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:00.202313+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:45:00.202313+0000', 'last_peered': '2024-09-16T04:45:00.202313+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:45:00.202313+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6410, 'log_dups_size': 0, 'ondisk_log_size': 6410, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50705676, 'num_objects': 6190, 'num_object_clones': 0, 'num_object_copies': 24760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6190, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6408, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3964}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:05.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:45:05.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6506", 'reported_seq': 7323, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:00.180380+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:45:00.180380+0000', 'last_peered': '2024-09-16T04:45:00.180380+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:45:00.180380+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6506, 'log_dups_size': 0, 'ondisk_log_size': 6506, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51194392, 'num_objects': 6250, 'num_object_clones': 0, 'num_object_copies': 25000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1960, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6503, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 383, 'num_bytes_recovered': 3112960, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4290}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:05.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:45:05.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6472", 'reported_seq': 6551, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:59.656718+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:44:59.656718+0000', 'last_peered': '2024-09-16T04:44:59.656718+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:44:59.656718+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6472, 'log_dups_size': 0, 'ondisk_log_size': 6472, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51492108, 'num_objects': 6286, 'num_object_clones': 0, 'num_object_copies': 25144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6286, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6470, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 4086}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:05.664 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:45:05.664 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6332", 'reported_seq': 6391, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:00.186862+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:45:00.186862+0000', 'last_peered': '2024-09-16T04:45:00.186862+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:45:00.186862+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6332, 'log_dups_size': 0, 'ondisk_log_size': 6332, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50450322, 'num_objects': 6159, 'num_object_clones': 0, 'num_object_copies': 24636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6329, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3971}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:05.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:45:05.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6245", 'reported_seq': 6285, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:00.043940+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:45:00.043940+0000', 'last_peered': '2024-09-16T04:45:00.043940+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:45:00.043940+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6245, 'log_dups_size': 0, 'ondisk_log_size': 6245, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49855110, 'num_objects': 6086, 'num_object_clones': 0, 'num_object_copies': 24344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6244, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3958}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:05.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:45:05.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6354", 'reported_seq': 6426, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:59.759957+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:44:59.759957+0000', 'last_peered': '2024-09-16T04:44:59.759957+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:44:59.759957+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6354, 'log_dups_size': 0, 'ondisk_log_size': 6354, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50616966, 'num_objects': 6179, 'num_object_clones': 0, 'num_object_copies': 24716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6353, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:05.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:45:05.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6489", 'reported_seq': 6543, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:59.936355+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:44:59.936355+0000', 'last_peered': '2024-09-16T04:44:59.936355+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:44:59.936355+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6489, 'log_dups_size': 0, 'ondisk_log_size': 6489, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51549452, 'num_objects': 6293, 'num_object_clones': 0, 'num_object_copies': 25172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6487, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4059}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:05.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:45:05.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6316", 'reported_seq': 6394, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:44:59.696788+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:44:59.696788+0000', 'last_peered': '2024-09-16T04:44:59.696788+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:44:59.696788+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6316, 'log_dups_size': 0, 'ondisk_log_size': 6316, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50238732, 'num_objects': 6133, 'num_object_clones': 0, 'num_object_copies': 24532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6314, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4001}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:05.665 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:45:05.665 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6248", 'reported_seq': 6314, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:00.185045+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:45:00.185045+0000', 'last_peered': '2024-09-16T04:45:00.185045+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:45:00.185045+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6248, 'log_dups_size': 0, 'ondisk_log_size': 6248, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49632524, 'num_objects': 6059, 'num_object_clones': 0, 'num_object_copies': 24236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6246, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3820}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:05.666 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:45:06.350 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:06.350 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32930/396604 objects degraded (8.303%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":318,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":99151,"data_bytes":812197568,"bytes_used":2859528192,"bytes_avail":383687528448,"bytes_total":386547056640,"degraded_objects":32930,"degraded_total":396604,"degraded_ratio":0.083029924055228896,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":55959,"recovering_keys_per_sec":0,"num_objects_recovered":84,"num_bytes_recovered":671744,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":731650,"read_op_per_sec":0,"write_op_per_sec":220},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:45:06.377 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:45:06.377 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:06.995 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:06.995 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:07.030 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-09-16T04:45:06.633015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":802571968,"num_objects":97976,"num_object_clones":0,"num_object_copies":391904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97976,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103511,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2712,"num_bytes_recovered":21905408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103538,"ondisk_log_size":103538,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2777524,"kb_used_data":1447640,"kb_used_omap":14,"kb_used_meta":1329777,"kb_avail":374709836,"statfs":{"total":386547056640,"available":383702872064,"internally_reserved":0,"allocated":1482383360,"data_stored":1481663484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1361692209},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,9,2,9,8,7,0,0,0,0,0,9],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13778920,"num_objects":-1681,"num_object_clones":0,"num_object_copies":-6724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1681,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3827,"num_write_kb":8577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004242"},"pg_stats":[{"pgid":"1.f","version":"33'6500","reported_seq":6546,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.150248+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:05.150248+0000","last_peered":"2024-09-16T04:45:05.150248+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:05.150248+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6500,"log_dups_size":0,"ondisk_log_size":6500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50413568,"num_objects":6154,"num_object_clones":0,"num_object_copies":24616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6500,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6536","reported_seq":6604,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.277690+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:05.277690+0000","last_peered":"2024-09-16T04:45:05.277690+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:05.277690+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6536,"log_dups_size":0,"ondisk_log_size":6536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50279692,"num_objects":6138,"num_object_clones":0,"num_object_copies":24552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6534,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3928}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6504","reported_seq":6581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:04.914571+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:04.914571+0000","last_peered":"2024-09-16T04:45:04.914571+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:04.914571+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6504,"log_dups_size":0,"ondisk_log_size":6504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50510494,"num_objects":6167,"num_object_clones":0,"num_object_copies":24668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6167,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6503,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3940}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6509","reported_seq":6578,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.293782+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:05.293782+0000","last_peered":"2024-09-16T04:45:05.293782+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:05.293782+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6509,"log_dups_size":0,"ondisk_log_size":6509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50468108,"num_objects":6161,"num_object_clones":0,"num_object_copies":24644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6507,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3933}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6559","reported_seq":6616,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.203965+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:05.203965+0000","last_peered":"2024-09-16T04:45:05.203965+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:05.203965+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6559,"log_dups_size":0,"ondisk_log_size":6559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50640140,"num_objects":6182,"num_object_clones":0,"num_object_copies":24728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6557,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6337","reported_seq":6383,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.049814+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:05.049814+0000","last_peered":"2024-09-16T04:45:05.049814+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:05.049814+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48947200,"num_objects":5975,"num_object_clones":0,"num_object_copies":23900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6337,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3768}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6489","reported_seq":6555,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.236024+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:05.236024+0000","last_peered":"2024-09-16T04:45:05.236024+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:05.236024+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6489,"log_dups_size":0,"ondisk_log_size":6489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50058508,"num_objects":6111,"num_object_clones":0,"num_object_copies":24444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6487,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3885}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6534","reported_seq":11089,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:02.261161+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:02.261161+0000","last_peered":"2024-09-16T04:45:02.261161+0000","last_clean":"2024-09-16T04:45:02.261161+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:02.261161+0000","last_undegraded":"2024-09-16T04:45:02.261161+0000","last_fullsized":"2024-09-16T04:45:02.261161+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6534,"log_dups_size":0,"ondisk_log_size":6534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:07.100481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51205388,"num_objects":6251,"num_object_clones":0,"num_object_copies":25004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6532,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6587","reported_seq":7518,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.235809+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:05.235809+0000","last_peered":"2024-09-16T04:45:05.235809+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:05.235809+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6587,"log_dups_size":0,"ondisk_log_size":6587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50530840,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6584,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":439,"num_bytes_recovered":3571712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6551","reported_seq":6634,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:04.928811+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:04.928811+0000","last_peered":"2024-09-16T04:45:04.928811+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:04.928811+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6551,"log_dups_size":0,"ondisk_log_size":6551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50844940,"num_objects":6207,"num_object_clones":0,"num_object_copies":24828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6549,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4009}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6419","reported_seq":6480,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.237638+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:05.237638+0000","last_peered":"2024-09-16T04:45:05.237638+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:05.237638+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6419,"log_dups_size":0,"ondisk_log_size":6419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49737618,"num_objects":6072,"num_object_clones":0,"num_object_copies":24288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6416,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6310","reported_seq":6350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:04.951266+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:04.951266+0000","last_peered":"2024-09-16T04:45:04.951266+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:04.951266+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6310,"log_dups_size":0,"ondisk_log_size":6310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49322630,"num_objects":6021,"num_object_clones":0,"num_object_copies":24084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6309,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3893}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6430","reported_seq":6505,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:04.795600+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:04.795600+0000","last_peered":"2024-09-16T04:45:04.795600+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:04.795600+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6430,"log_dups_size":0,"ondisk_log_size":6430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49994374,"num_objects":6103,"num_object_clones":0,"num_object_copies":24412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6429,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3878}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6561","reported_seq":6615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.126369+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:05.126369+0000","last_peered":"2024-09-16T04:45:05.126369+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:05.126369+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6561,"log_dups_size":0,"ondisk_log_size":6561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50959628,"num_objects":6221,"num_object_clones":0,"num_object_copies":24884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6559,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3987}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6380","reported_seq":6464,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:04.875938+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:04.875938+0000","last_peered":"2024-09-16T04:45:04.875938+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:04.875938+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6380,"log_dups_size":0,"ondisk_log_size":6380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49714444,"num_objects":6069,"num_object_clones":0,"num_object_copies":24276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6378,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3939}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6332","reported_seq":6398,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.278210+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:05.278210+0000","last_peered":"2024-09-16T04:45:05.278210+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:05.278210+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6332,"log_dups_size":0,"ondisk_log_size":6332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48944396,"num_objects":5975,"num_object_clones":0,"num_object_copies":23900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6330,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3736}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":802571968,"num_objects":97976,"num_object_clones":0,"num_object_copies":391904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97976,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103511,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2712,"num_bytes_recovered":21905408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1479892992,"data_stored":1479892992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103538,"ondisk_log_size":103538,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748712,"kb_used_data":392148,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93623128,"statfs":{"total":96636764160,"available":95870083072,"internally_reserved":0,"allocated":401559552,"data_stored":401379583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365096655},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,0,4,0,0,0,0,0,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756888,"kb_used_data":392500,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93614952,"statfs":{"total":96636764160,"available":95861710848,"internally_reserved":0,"allocated":401920000,"data_stored":401740031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,3,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051628,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518712,"kb_used_data":270440,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93853128,"statfs":{"total":96636764160,"available":96105603072,"internally_reserved":0,"allocated":276930560,"data_stored":276750591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254209743},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753212,"kb_used_data":392552,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93618628,"statfs":{"total":96636764160,"available":95865475072,"internally_reserved":0,"allocated":401973248,"data_stored":401793279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":369290958},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,0,0,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":400953344,"data_stored":400953344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276639744,"data_stored":276639744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401297408,"data_stored":401297408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":401002496,"data_stored":401002496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:10.033 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:10.610 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:10.610 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:10.631 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-16T04:45:08.633551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":801949376,"num_objects":97900,"num_object_clones":0,"num_object_copies":391600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97900,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103587,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2712,"num_bytes_recovered":21905408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103614,"ondisk_log_size":103614,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799404,"kb_used_data":1447308,"kb_used_omap":14,"kb_used_meta":1351985,"kb_avail":374687956,"statfs":{"total":386547056640,"available":383680466944,"internally_reserved":0,"allocated":1482043392,"data_stored":1481323516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1384433201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,7,4,9,4,4,0,0,0,0,0,9],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21356544,"num_objects":-2607,"num_object_clones":0,"num_object_copies":-10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2671,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":778240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004297"},"pg_stats":[{"pgid":"1.f","version":"33'6500","reported_seq":6546,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.150248+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:05.150248+0000","last_peered":"2024-09-16T04:45:05.150248+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:05.150248+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6500,"log_dups_size":0,"ondisk_log_size":6500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50413568,"num_objects":6154,"num_object_clones":0,"num_object_copies":24616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6500,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3936}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6536","reported_seq":6604,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.277690+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:05.277690+0000","last_peered":"2024-09-16T04:45:05.277690+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:05.277690+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6536,"log_dups_size":0,"ondisk_log_size":6536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50279692,"num_objects":6138,"num_object_clones":0,"num_object_copies":24552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6534,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3928}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6504","reported_seq":6581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:04.914571+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:04.914571+0000","last_peered":"2024-09-16T04:45:04.914571+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:04.914571+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6504,"log_dups_size":0,"ondisk_log_size":6504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50510494,"num_objects":6167,"num_object_clones":0,"num_object_copies":24668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6167,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6503,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3940}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6509","reported_seq":6578,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.293782+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:05.293782+0000","last_peered":"2024-09-16T04:45:05.293782+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:05.293782+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6509,"log_dups_size":0,"ondisk_log_size":6509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50468108,"num_objects":6161,"num_object_clones":0,"num_object_copies":24644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6507,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3933}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6559","reported_seq":6616,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.203965+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:05.203965+0000","last_peered":"2024-09-16T04:45:05.203965+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:05.203965+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6559,"log_dups_size":0,"ondisk_log_size":6559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50640140,"num_objects":6182,"num_object_clones":0,"num_object_copies":24728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6557,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6337","reported_seq":6383,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.049814+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:05.049814+0000","last_peered":"2024-09-16T04:45:05.049814+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:05.049814+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48947200,"num_objects":5975,"num_object_clones":0,"num_object_copies":23900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6337,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3768}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6489","reported_seq":6555,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.236024+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:05.236024+0000","last_peered":"2024-09-16T04:45:05.236024+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:05.236024+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6489,"log_dups_size":0,"ondisk_log_size":6489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50058508,"num_objects":6111,"num_object_clones":0,"num_object_copies":24444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6487,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3885}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6610","reported_seq":11165,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:07.346963+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:07.346963+0000","last_peered":"2024-09-16T04:45:07.346963+0000","last_clean":"2024-09-16T04:45:07.346963+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:07.346963+0000","last_undegraded":"2024-09-16T04:45:07.346963+0000","last_fullsized":"2024-09-16T04:45:07.346963+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6610,"log_dups_size":0,"ondisk_log_size":6610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:12.114483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50582796,"num_objects":6175,"num_object_clones":0,"num_object_copies":24700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6608,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6587","reported_seq":7518,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.235809+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:05.235809+0000","last_peered":"2024-09-16T04:45:05.235809+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:05.235809+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6587,"log_dups_size":0,"ondisk_log_size":6587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50530840,"num_objects":6169,"num_object_clones":0,"num_object_copies":24676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6584,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":439,"num_bytes_recovered":3571712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6551","reported_seq":6634,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:04.928811+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:04.928811+0000","last_peered":"2024-09-16T04:45:04.928811+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:04.928811+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6551,"log_dups_size":0,"ondisk_log_size":6551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50844940,"num_objects":6207,"num_object_clones":0,"num_object_copies":24828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6549,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4009}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6419","reported_seq":6480,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.237638+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:05.237638+0000","last_peered":"2024-09-16T04:45:05.237638+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:05.237638+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6419,"log_dups_size":0,"ondisk_log_size":6419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49737618,"num_objects":6072,"num_object_clones":0,"num_object_copies":24288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6416,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6310","reported_seq":6350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:04.951266+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:04.951266+0000","last_peered":"2024-09-16T04:45:04.951266+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:04.951266+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6310,"log_dups_size":0,"ondisk_log_size":6310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49322630,"num_objects":6021,"num_object_clones":0,"num_object_copies":24084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6309,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3893}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6430","reported_seq":6505,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:04.795600+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:04.795600+0000","last_peered":"2024-09-16T04:45:04.795600+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:04.795600+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6430,"log_dups_size":0,"ondisk_log_size":6430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49994374,"num_objects":6103,"num_object_clones":0,"num_object_copies":24412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6429,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3878}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6561","reported_seq":6615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.126369+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:05.126369+0000","last_peered":"2024-09-16T04:45:05.126369+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:05.126369+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6561,"log_dups_size":0,"ondisk_log_size":6561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50959628,"num_objects":6221,"num_object_clones":0,"num_object_copies":24884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6559,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3987}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6380","reported_seq":6464,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:04.875938+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:04.875938+0000","last_peered":"2024-09-16T04:45:04.875938+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:04.875938+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6380,"log_dups_size":0,"ondisk_log_size":6380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49714444,"num_objects":6069,"num_object_clones":0,"num_object_copies":24276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6378,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3939}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6332","reported_seq":6398,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:05.278210+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:05.278210+0000","last_peered":"2024-09-16T04:45:05.278210+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:05.278210+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6332,"log_dups_size":0,"ondisk_log_size":6332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48944396,"num_objects":5975,"num_object_clones":0,"num_object_copies":23900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6330,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3736}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":801949376,"num_objects":97900,"num_object_clones":0,"num_object_copies":391600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97900,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103587,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2712,"num_bytes_recovered":21905408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1479544832,"data_stored":1479544832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103614,"ondisk_log_size":103614,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748712,"kb_used_data":392148,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93623128,"statfs":{"total":96636764160,"available":95870083072,"internally_reserved":0,"allocated":401559552,"data_stored":401379583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365096655},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,0,4,0,0,0,0,0,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756888,"kb_used_data":392500,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93614952,"statfs":{"total":96636764160,"available":95861710848,"internally_reserved":0,"allocated":401920000,"data_stored":401740031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,3,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051629,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540592,"kb_used_data":270108,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93831248,"statfs":{"total":96636764160,"available":96083197952,"internally_reserved":0,"allocated":276590592,"data_stored":276410623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276950735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753212,"kb_used_data":392552,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93618628,"statfs":{"total":96636764160,"available":95865475072,"internally_reserved":0,"allocated":401973248,"data_stored":401793279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":369290958},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,0,0,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":400953344,"data_stored":400953344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276291584,"data_stored":276291584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401297408,"data_stored":401297408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":401002496,"data_stored":401002496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:10.632 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:11.245 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:11.245 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:11.278 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-09-16T04:45:10.635306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":792135360,"num_objects":96702,"num_object_clones":0,"num_object_copies":386808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104785,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2783,"num_bytes_recovered":22355968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104812,"ondisk_log_size":104812,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2784504,"kb_used_data":1432424,"kb_used_omap":14,"kb_used_meta":1351985,"kb_avail":374702856,"statfs":{"total":386547056640,"available":383695724544,"internally_reserved":0,"allocated":1466802176,"data_stored":1466082300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1384433201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,8,4,12,4,3,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-30982144,"num_objects":-3782,"num_object_clones":0,"num_object_copies":-15128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3782,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":1228800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005531"},"pg_stats":[{"pgid":"1.f","version":"33'6592","reported_seq":6642,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.186016+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:10.186016+0000","last_peered":"2024-09-16T04:45:10.186016+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:10.186016+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6592,"log_dups_size":0,"ondisk_log_size":6592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49659904,"num_objects":6062,"num_object_clones":0,"num_object_copies":24248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6592,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6603","reported_seq":6671,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.308725+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:10.308725+0000","last_peered":"2024-09-16T04:45:10.308725+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:10.308725+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6603,"log_dups_size":0,"ondisk_log_size":6603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49730828,"num_objects":6071,"num_object_clones":0,"num_object_copies":24284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6601,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3861}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6582","reported_seq":6661,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:09.860780+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:09.860780+0000","last_peered":"2024-09-16T04:45:09.860780+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:09.860780+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6582,"log_dups_size":0,"ondisk_log_size":6582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49871518,"num_objects":6089,"num_object_clones":0,"num_object_copies":24356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6581,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3863}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6593","reported_seq":6662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.334817+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:10.334817+0000","last_peered":"2024-09-16T04:45:10.334817+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:10.334817+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6593,"log_dups_size":0,"ondisk_log_size":6593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49779980,"num_objects":6077,"num_object_clones":0,"num_object_copies":24308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6591,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3849}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6640","reported_seq":6703,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.177231+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:10.177231+0000","last_peered":"2024-09-16T04:45:10.177231+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:10.177231+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6640,"log_dups_size":0,"ondisk_log_size":6640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49976588,"num_objects":6101,"num_object_clones":0,"num_object_copies":24404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6638,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6424","reported_seq":6474,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.272295+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:10.272295+0000","last_peered":"2024-09-16T04:45:10.272295+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:10.272295+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6424,"log_dups_size":0,"ondisk_log_size":6424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48234496,"num_objects":5888,"num_object_clones":0,"num_object_copies":23552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6424,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6563","reported_seq":6629,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.325975+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:10.325975+0000","last_peered":"2024-09-16T04:45:10.325975+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:10.325975+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49452300,"num_objects":6037,"num_object_clones":0,"num_object_copies":24148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6561,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3811}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6610","reported_seq":11165,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:07.346963+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:07.346963+0000","last_peered":"2024-09-16T04:45:07.346963+0000","last_clean":"2024-09-16T04:45:07.346963+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:07.346963+0000","last_undegraded":"2024-09-16T04:45:07.346963+0000","last_fullsized":"2024-09-16T04:45:07.346963+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6610,"log_dups_size":0,"ondisk_log_size":6610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:12.114483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50582796,"num_objects":6175,"num_object_clones":0,"num_object_copies":24700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6608,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6660","reported_seq":7698,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.357845+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:10.357845+0000","last_peered":"2024-09-16T04:45:10.357845+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:10.357845+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6660,"log_dups_size":0,"ondisk_log_size":6660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49932824,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6657,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":494,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4247}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6624","reported_seq":6707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:09.948238+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:09.948238+0000","last_peered":"2024-09-16T04:45:09.948238+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:09.948238+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50246924,"num_objects":6134,"num_object_clones":0,"num_object_copies":24536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3936}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6501","reported_seq":6565,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.242150+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:10.242150+0000","last_peered":"2024-09-16T04:45:10.242150+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:10.242150+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6501,"log_dups_size":0,"ondisk_log_size":6501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49065874,"num_objects":5990,"num_object_clones":0,"num_object_copies":23960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6498,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3804}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6390","reported_seq":6433,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.368677+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:10.368677+0000","last_peered":"2024-09-16T04:45:10.368677+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:10.368677+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48667270,"num_objects":5941,"num_object_clones":0,"num_object_copies":23764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3814}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6512","reported_seq":6587,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:09.878764+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:09.878764+0000","last_peered":"2024-09-16T04:45:09.878764+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:09.878764+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6512,"log_dups_size":0,"ondisk_log_size":6512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49322630,"num_objects":6021,"num_object_clones":0,"num_object_copies":24084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6511,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3796}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6631","reported_seq":6696,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.261900+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:10.261900+0000","last_peered":"2024-09-16T04:45:10.261900+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:10.261900+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6631,"log_dups_size":0,"ondisk_log_size":6631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50386188,"num_objects":6151,"num_object_clones":0,"num_object_copies":24604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6629,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3922}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6470","reported_seq":6554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:09.987639+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:09.987639+0000","last_peered":"2024-09-16T04:45:09.987639+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:09.987639+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6470,"log_dups_size":0,"ondisk_log_size":6470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48977164,"num_objects":5979,"num_object_clones":0,"num_object_copies":23916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6468,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3849}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6417","reported_seq":6483,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.306870+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:10.306870+0000","last_peered":"2024-09-16T04:45:10.306870+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:10.306870+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6417,"log_dups_size":0,"ondisk_log_size":6417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48248076,"num_objects":5890,"num_object_clones":0,"num_object_copies":23560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6415,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":792135360,"num_objects":96702,"num_object_clones":0,"num_object_copies":386808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104785,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2783,"num_bytes_recovered":22355968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1463840768,"data_stored":1463840768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104812,"ondisk_log_size":104812,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747888,"kb_used_data":387168,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93623952,"statfs":{"total":96636764160,"available":95870926848,"internally_reserved":0,"allocated":396460032,"data_stored":396280063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369356495},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,0,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743624,"kb_used_data":387576,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93628216,"statfs":{"total":96636764160,"available":95875293184,"internally_reserved":0,"allocated":396877824,"data_stored":396697855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,1,1,3,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051629,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540592,"kb_used_data":270108,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93831248,"statfs":{"total":96636764160,"available":96083197952,"internally_reserved":0,"allocated":276590592,"data_stored":276410623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276950735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752400,"kb_used_data":387572,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93619440,"statfs":{"total":96636764160,"available":95866306560,"internally_reserved":0,"allocated":396873728,"data_stored":396693759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":373550798},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395718656,"data_stored":395718656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276291584,"data_stored":276291584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":396091392,"data_stored":396091392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":395739136,"data_stored":395739136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:11.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:45:11.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6592", 'reported_seq': 6642, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:10.186016+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:45:10.186016+0000', 'last_peered': '2024-09-16T04:45:10.186016+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:45:10.186016+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6592, 'log_dups_size': 0, 'ondisk_log_size': 6592, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49659904, 'num_objects': 6062, 'num_object_clones': 0, 'num_object_copies': 24248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2216, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6592, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3846}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:11.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:45:11.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6603", 'reported_seq': 6671, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:10.308725+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:45:10.308725+0000', 'last_peered': '2024-09-16T04:45:10.308725+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:45:10.308725+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6603, 'log_dups_size': 0, 'ondisk_log_size': 6603, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49730828, 'num_objects': 6071, 'num_object_clones': 0, 'num_object_copies': 24284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6071, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6601, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3861}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:11.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:45:11.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6582", 'reported_seq': 6661, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:09.860780+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:45:09.860780+0000', 'last_peered': '2024-09-16T04:45:09.860780+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:45:09.860780+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6582, 'log_dups_size': 0, 'ondisk_log_size': 6582, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49871518, 'num_objects': 6089, 'num_object_clones': 0, 'num_object_copies': 24356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6089, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6581, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3863}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:11.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:45:11.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6593", 'reported_seq': 6662, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:10.334817+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:45:10.334817+0000', 'last_peered': '2024-09-16T04:45:10.334817+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:45:10.334817+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6593, 'log_dups_size': 0, 'ondisk_log_size': 6593, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49779980, 'num_objects': 6077, 'num_object_clones': 0, 'num_object_copies': 24308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6591, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3849}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:11.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:45:11.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6640", 'reported_seq': 6703, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:10.177231+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:45:10.177231+0000', 'last_peered': '2024-09-16T04:45:10.177231+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:45:10.177231+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6640, 'log_dups_size': 0, 'ondisk_log_size': 6640, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49976588, 'num_objects': 6101, 'num_object_clones': 0, 'num_object_copies': 24404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2282, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6101, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6638, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3821}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:11.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:45:11.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6424", 'reported_seq': 6474, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:10.272295+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:45:10.272295+0000', 'last_peered': '2024-09-16T04:45:10.272295+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:45:10.272295+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6424, 'log_dups_size': 0, 'ondisk_log_size': 6424, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48234496, 'num_objects': 5888, 'num_object_clones': 0, 'num_object_copies': 23552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5888, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6424, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3683}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:11.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:45:11.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6563", 'reported_seq': 6629, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:10.325975+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:45:10.325975+0000', 'last_peered': '2024-09-16T04:45:10.325975+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:45:10.325975+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6563, 'log_dups_size': 0, 'ondisk_log_size': 6563, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49452300, 'num_objects': 6037, 'num_object_clones': 0, 'num_object_copies': 24148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6561, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3811}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:11.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:45:11.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6660", 'reported_seq': 7698, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:10.357845+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:45:10.357845+0000', 'last_peered': '2024-09-16T04:45:10.357845+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:45:10.357845+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6660, 'log_dups_size': 0, 'ondisk_log_size': 6660, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49932824, 'num_objects': 6096, 'num_object_clones': 0, 'num_object_copies': 24384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1849, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6657, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 494, 'num_bytes_recovered': 4022272, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4247}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:45:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6624", 'reported_seq': 6707, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:09.948238+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:45:09.948238+0000', 'last_peered': '2024-09-16T04:45:09.948238+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:45:09.948238+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6624, 'log_dups_size': 0, 'ondisk_log_size': 6624, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50246924, 'num_objects': 6134, 'num_object_clones': 0, 'num_object_copies': 24536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6622, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3936}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:45:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6501", 'reported_seq': 6565, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:10.242150+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:45:10.242150+0000', 'last_peered': '2024-09-16T04:45:10.242150+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:45:10.242150+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6501, 'log_dups_size': 0, 'ondisk_log_size': 6501, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49065874, 'num_objects': 5990, 'num_object_clones': 0, 'num_object_copies': 23960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2189, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5990, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6498, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3804}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:45:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6390", 'reported_seq': 6433, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:10.368677+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:45:10.368677+0000', 'last_peered': '2024-09-16T04:45:10.368677+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:45:10.368677+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6390, 'log_dups_size': 0, 'ondisk_log_size': 6390, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48667270, 'num_objects': 5941, 'num_object_clones': 0, 'num_object_copies': 23764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2128, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6389, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3814}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:45:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6512", 'reported_seq': 6587, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:09.878764+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:45:09.878764+0000', 'last_peered': '2024-09-16T04:45:09.878764+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:45:09.878764+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6512, 'log_dups_size': 0, 'ondisk_log_size': 6512, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49322630, 'num_objects': 6021, 'num_object_clones': 0, 'num_object_copies': 24084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6021, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6511, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3796}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:45:11.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6631", 'reported_seq': 6696, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:10.261900+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:45:10.261900+0000', 'last_peered': '2024-09-16T04:45:10.261900+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:45:10.261900+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6631, 'log_dups_size': 0, 'ondisk_log_size': 6631, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50386188, 'num_objects': 6151, 'num_object_clones': 0, 'num_object_copies': 24604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6629, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3922}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:45:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6470", 'reported_seq': 6554, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:09.987639+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:45:09.987639+0000', 'last_peered': '2024-09-16T04:45:09.987639+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:45:09.987639+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6470, 'log_dups_size': 0, 'ondisk_log_size': 6470, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48977164, 'num_objects': 5979, 'num_object_clones': 0, 'num_object_copies': 23916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6468, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3849}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:45:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6417", 'reported_seq': 6483, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:10.306870+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:45:10.306870+0000', 'last_peered': '2024-09-16T04:45:10.306870+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:45:10.306870+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6417, 'log_dups_size': 0, 'ondisk_log_size': 6417, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48248076, 'num_objects': 5890, 'num_object_clones': 0, 'num_object_copies': 23560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6415, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3651}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:11.282 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:45:11.961 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:11.961 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32796/386808 objects degraded (8.479%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":324,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":96702,"data_bytes":792135360,"bytes_used":2851332096,"bytes_avail":383695724544,"bytes_total":386547056640,"degraded_objects":32796,"degraded_total":386808,"degraded_ratio":0.084786250542905006,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":102352,"recovering_keys_per_sec":0,"num_objects_recovered":175,"num_bytes_recovered":1228800,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":315},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:45:11.987 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:45:11.987 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:12.620 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:12.620 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:12.652 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-09-16T04:45:10.635306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":792135360,"num_objects":96702,"num_object_clones":0,"num_object_copies":386808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104785,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2783,"num_bytes_recovered":22355968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104812,"ondisk_log_size":104812,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2784504,"kb_used_data":1432424,"kb_used_omap":14,"kb_used_meta":1351985,"kb_avail":374702856,"statfs":{"total":386547056640,"available":383695724544,"internally_reserved":0,"allocated":1466802176,"data_stored":1466082300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1384433201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,8,4,12,4,3,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-30982144,"num_objects":-3782,"num_object_clones":0,"num_object_copies":-15128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3782,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":1228800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005531"},"pg_stats":[{"pgid":"1.f","version":"33'6592","reported_seq":6642,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.186016+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:10.186016+0000","last_peered":"2024-09-16T04:45:10.186016+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:10.186016+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6592,"log_dups_size":0,"ondisk_log_size":6592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49659904,"num_objects":6062,"num_object_clones":0,"num_object_copies":24248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6592,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6603","reported_seq":6671,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.308725+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:10.308725+0000","last_peered":"2024-09-16T04:45:10.308725+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:10.308725+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6603,"log_dups_size":0,"ondisk_log_size":6603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49730828,"num_objects":6071,"num_object_clones":0,"num_object_copies":24284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6601,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3861}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6582","reported_seq":6661,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:09.860780+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:09.860780+0000","last_peered":"2024-09-16T04:45:09.860780+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:09.860780+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6582,"log_dups_size":0,"ondisk_log_size":6582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49871518,"num_objects":6089,"num_object_clones":0,"num_object_copies":24356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6581,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3863}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6593","reported_seq":6662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.334817+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:10.334817+0000","last_peered":"2024-09-16T04:45:10.334817+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:10.334817+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6593,"log_dups_size":0,"ondisk_log_size":6593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49779980,"num_objects":6077,"num_object_clones":0,"num_object_copies":24308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6591,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3849}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6640","reported_seq":6703,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.177231+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:10.177231+0000","last_peered":"2024-09-16T04:45:10.177231+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:10.177231+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6640,"log_dups_size":0,"ondisk_log_size":6640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49976588,"num_objects":6101,"num_object_clones":0,"num_object_copies":24404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6638,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6424","reported_seq":6474,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.272295+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:10.272295+0000","last_peered":"2024-09-16T04:45:10.272295+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:10.272295+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6424,"log_dups_size":0,"ondisk_log_size":6424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48234496,"num_objects":5888,"num_object_clones":0,"num_object_copies":23552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6424,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6563","reported_seq":6629,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.325975+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:10.325975+0000","last_peered":"2024-09-16T04:45:10.325975+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:10.325975+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49452300,"num_objects":6037,"num_object_clones":0,"num_object_copies":24148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6561,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3811}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6610","reported_seq":11165,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:07.346963+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:07.346963+0000","last_peered":"2024-09-16T04:45:07.346963+0000","last_clean":"2024-09-16T04:45:07.346963+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:07.346963+0000","last_undegraded":"2024-09-16T04:45:07.346963+0000","last_fullsized":"2024-09-16T04:45:07.346963+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6610,"log_dups_size":0,"ondisk_log_size":6610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:12.114483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50582796,"num_objects":6175,"num_object_clones":0,"num_object_copies":24700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6608,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6660","reported_seq":7698,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.357845+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:10.357845+0000","last_peered":"2024-09-16T04:45:10.357845+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:10.357845+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6660,"log_dups_size":0,"ondisk_log_size":6660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49932824,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6657,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":494,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4247}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6624","reported_seq":6707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:09.948238+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:09.948238+0000","last_peered":"2024-09-16T04:45:09.948238+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:09.948238+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50246924,"num_objects":6134,"num_object_clones":0,"num_object_copies":24536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3936}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6501","reported_seq":6565,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.242150+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:10.242150+0000","last_peered":"2024-09-16T04:45:10.242150+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:10.242150+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6501,"log_dups_size":0,"ondisk_log_size":6501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49065874,"num_objects":5990,"num_object_clones":0,"num_object_copies":23960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6498,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3804}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6390","reported_seq":6433,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.368677+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:10.368677+0000","last_peered":"2024-09-16T04:45:10.368677+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:10.368677+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48667270,"num_objects":5941,"num_object_clones":0,"num_object_copies":23764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3814}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6512","reported_seq":6587,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:09.878764+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:09.878764+0000","last_peered":"2024-09-16T04:45:09.878764+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:09.878764+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6512,"log_dups_size":0,"ondisk_log_size":6512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49322630,"num_objects":6021,"num_object_clones":0,"num_object_copies":24084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6511,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3796}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6631","reported_seq":6696,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.261900+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:10.261900+0000","last_peered":"2024-09-16T04:45:10.261900+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:10.261900+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6631,"log_dups_size":0,"ondisk_log_size":6631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50386188,"num_objects":6151,"num_object_clones":0,"num_object_copies":24604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6629,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3922}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6470","reported_seq":6554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:09.987639+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:09.987639+0000","last_peered":"2024-09-16T04:45:09.987639+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:09.987639+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6470,"log_dups_size":0,"ondisk_log_size":6470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48977164,"num_objects":5979,"num_object_clones":0,"num_object_copies":23916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6468,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3849}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6417","reported_seq":6483,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.306870+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:10.306870+0000","last_peered":"2024-09-16T04:45:10.306870+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:10.306870+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6417,"log_dups_size":0,"ondisk_log_size":6417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48248076,"num_objects":5890,"num_object_clones":0,"num_object_copies":23560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6415,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":792135360,"num_objects":96702,"num_object_clones":0,"num_object_copies":386808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104785,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2783,"num_bytes_recovered":22355968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1463840768,"data_stored":1463840768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104812,"ondisk_log_size":104812,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747888,"kb_used_data":387168,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93623952,"statfs":{"total":96636764160,"available":95870926848,"internally_reserved":0,"allocated":396460032,"data_stored":396280063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369356495},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,0,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743624,"kb_used_data":387576,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93628216,"statfs":{"total":96636764160,"available":95875293184,"internally_reserved":0,"allocated":396877824,"data_stored":396697855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,1,1,3,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051629,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540592,"kb_used_data":270108,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93831248,"statfs":{"total":96636764160,"available":96083197952,"internally_reserved":0,"allocated":276590592,"data_stored":276410623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276950735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752400,"kb_used_data":387572,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93619440,"statfs":{"total":96636764160,"available":95866306560,"internally_reserved":0,"allocated":396873728,"data_stored":396693759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":373550798},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395718656,"data_stored":395718656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276291584,"data_stored":276291584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":396091392,"data_stored":396091392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":395739136,"data_stored":395739136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:15.654 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:16.219 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:16.219 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:16.244 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-16T04:45:14.637786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":791488192,"num_objects":96623,"num_object_clones":0,"num_object_copies":386492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96623,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104864,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2783,"num_bytes_recovered":22355968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104891,"ondisk_log_size":104891,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2792416,"kb_used_data":1432020,"kb_used_omap":14,"kb_used_meta":1360305,"kb_avail":374694944,"statfs":{"total":386547056640,"available":383687622656,"internally_reserved":0,"allocated":1466388480,"data_stored":1465668604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1392952881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,6,7,15,4,3,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20709376,"num_objects":-2528,"num_object_clones":0,"num_object_copies":-10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2528,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":909312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005841"},"pg_stats":[{"pgid":"1.f","version":"33'6592","reported_seq":6642,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.186016+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:10.186016+0000","last_peered":"2024-09-16T04:45:10.186016+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:10.186016+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6592,"log_dups_size":0,"ondisk_log_size":6592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49659904,"num_objects":6062,"num_object_clones":0,"num_object_copies":24248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6592,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6603","reported_seq":6671,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.308725+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:10.308725+0000","last_peered":"2024-09-16T04:45:10.308725+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:10.308725+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6603,"log_dups_size":0,"ondisk_log_size":6603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49730828,"num_objects":6071,"num_object_clones":0,"num_object_copies":24284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6601,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3861}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6582","reported_seq":6661,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:09.860780+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:09.860780+0000","last_peered":"2024-09-16T04:45:09.860780+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:09.860780+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6582,"log_dups_size":0,"ondisk_log_size":6582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49871518,"num_objects":6089,"num_object_clones":0,"num_object_copies":24356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6581,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3863}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6593","reported_seq":6662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.334817+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:10.334817+0000","last_peered":"2024-09-16T04:45:10.334817+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:10.334817+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6593,"log_dups_size":0,"ondisk_log_size":6593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49779980,"num_objects":6077,"num_object_clones":0,"num_object_copies":24308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6591,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3849}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6640","reported_seq":6703,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.177231+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:10.177231+0000","last_peered":"2024-09-16T04:45:10.177231+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:10.177231+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6640,"log_dups_size":0,"ondisk_log_size":6640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49976588,"num_objects":6101,"num_object_clones":0,"num_object_copies":24404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6638,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6424","reported_seq":6474,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.272295+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:10.272295+0000","last_peered":"2024-09-16T04:45:10.272295+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:10.272295+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6424,"log_dups_size":0,"ondisk_log_size":6424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48234496,"num_objects":5888,"num_object_clones":0,"num_object_copies":23552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6424,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6563","reported_seq":6629,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.325975+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:10.325975+0000","last_peered":"2024-09-16T04:45:10.325975+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:10.325975+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49452300,"num_objects":6037,"num_object_clones":0,"num_object_copies":24148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6561,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3811}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6689","reported_seq":11245,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:12.287222+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:12.287222+0000","last_peered":"2024-09-16T04:45:12.287222+0000","last_clean":"2024-09-16T04:45:12.287222+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:12.287222+0000","last_undegraded":"2024-09-16T04:45:12.287222+0000","last_fullsized":"2024-09-16T04:45:12.287222+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49935628,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6660","reported_seq":7698,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.357845+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:10.357845+0000","last_peered":"2024-09-16T04:45:10.357845+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:10.357845+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6660,"log_dups_size":0,"ondisk_log_size":6660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49932824,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6657,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":494,"num_bytes_recovered":4022272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4247}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6624","reported_seq":6707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:09.948238+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:09.948238+0000","last_peered":"2024-09-16T04:45:09.948238+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:09.948238+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50246924,"num_objects":6134,"num_object_clones":0,"num_object_copies":24536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3936}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6501","reported_seq":6565,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.242150+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:10.242150+0000","last_peered":"2024-09-16T04:45:10.242150+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:10.242150+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6501,"log_dups_size":0,"ondisk_log_size":6501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49065874,"num_objects":5990,"num_object_clones":0,"num_object_copies":23960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6498,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3804}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6390","reported_seq":6433,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.368677+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:10.368677+0000","last_peered":"2024-09-16T04:45:10.368677+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:10.368677+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48667270,"num_objects":5941,"num_object_clones":0,"num_object_copies":23764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3814}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6512","reported_seq":6587,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:09.878764+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:09.878764+0000","last_peered":"2024-09-16T04:45:09.878764+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:09.878764+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6512,"log_dups_size":0,"ondisk_log_size":6512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49322630,"num_objects":6021,"num_object_clones":0,"num_object_copies":24084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6511,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3796}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6631","reported_seq":6696,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.261900+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:10.261900+0000","last_peered":"2024-09-16T04:45:10.261900+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:10.261900+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6631,"log_dups_size":0,"ondisk_log_size":6631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50386188,"num_objects":6151,"num_object_clones":0,"num_object_copies":24604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6629,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3922}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6470","reported_seq":6554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:09.987639+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:09.987639+0000","last_peered":"2024-09-16T04:45:09.987639+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:09.987639+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6470,"log_dups_size":0,"ondisk_log_size":6470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48977164,"num_objects":5979,"num_object_clones":0,"num_object_copies":23916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6468,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3849}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6417","reported_seq":6483,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:10.306870+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:10.306870+0000","last_peered":"2024-09-16T04:45:10.306870+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:10.306870+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6417,"log_dups_size":0,"ondisk_log_size":6417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48248076,"num_objects":5890,"num_object_clones":0,"num_object_copies":23560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6415,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":791488192,"num_objects":96623,"num_object_clones":0,"num_object_copies":386492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96623,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104864,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2783,"num_bytes_recovered":22355968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1463414784,"data_stored":1463414784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104891,"ondisk_log_size":104891,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747888,"kb_used_data":387168,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93623952,"statfs":{"total":96636764160,"available":95870926848,"internally_reserved":0,"allocated":396460032,"data_stored":396280063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369356495},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,0,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743624,"kb_used_data":387576,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93628216,"statfs":{"total":96636764160,"available":95875293184,"internally_reserved":0,"allocated":396877824,"data_stored":396697855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,1,1,3,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051630,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548504,"kb_used_data":269704,"kb_used_omap":4,"kb_used_meta":278779,"kb_avail":93823336,"statfs":{"total":96636764160,"available":96075096064,"internally_reserved":0,"allocated":276176896,"data_stored":275996927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":285470415},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752400,"kb_used_data":387572,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93619440,"statfs":{"total":96636764160,"available":95866306560,"internally_reserved":0,"allocated":396873728,"data_stored":396693759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":373550798},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395718656,"data_stored":395718656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275865600,"data_stored":275865600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":396091392,"data_stored":396091392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":395739136,"data_stored":395739136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:16.246 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:16.839 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:16.841 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:16.866 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-16T04:45:16.638265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":782001856,"num_objects":95465,"num_object_clones":0,"num_object_copies":381860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95465,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106022,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2844,"num_bytes_recovered":22749184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106049,"ondisk_log_size":106049,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773636,"kb_used_data":1417412,"kb_used_omap":14,"kb_used_meta":1356145,"kb_avail":374713724,"statfs":{"total":386547056640,"available":383706853376,"internally_reserved":0,"allocated":1451429888,"data_stored":1450710012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1388693041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2,6,12,9,0,0,0,0,0,0,2,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-30195712,"num_objects":-3686,"num_object_clones":0,"num_object_copies":-14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3686,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":1302528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005822"},"pg_stats":[{"pgid":"1.f","version":"33'6659","reported_seq":6712,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.363179+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:15.363179+0000","last_peered":"2024-09-16T04:45:15.363179+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:15.363179+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6659,"log_dups_size":0,"ondisk_log_size":6659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49111040,"num_objects":5995,"num_object_clones":0,"num_object_copies":23980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6659,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3780}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6678","reported_seq":6747,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.214706+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:15.214706+0000","last_peered":"2024-09-16T04:45:15.214706+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:15.214706+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6678,"log_dups_size":0,"ondisk_log_size":6678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49116428,"num_objects":5996,"num_object_clones":0,"num_object_copies":23984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6676,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6664","reported_seq":6743,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:14.982984+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:14.982984+0000","last_peered":"2024-09-16T04:45:14.982984+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:14.982984+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6664,"log_dups_size":0,"ondisk_log_size":6664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49199774,"num_objects":6007,"num_object_clones":0,"num_object_copies":24028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6663,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3781}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6668","reported_seq":6737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.167333+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:15.167333+0000","last_peered":"2024-09-16T04:45:15.167333+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:15.167333+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6668,"log_dups_size":0,"ondisk_log_size":6668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49165580,"num_objects":6002,"num_object_clones":0,"num_object_copies":24008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3774}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6726","reported_seq":6793,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.290992+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:15.290992+0000","last_peered":"2024-09-16T04:45:15.290992+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:15.290992+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49272076,"num_objects":6015,"num_object_clones":0,"num_object_copies":24060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3737}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6509","reported_seq":6562,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.279112+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:15.279112+0000","last_peered":"2024-09-16T04:45:15.279112+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:15.279112+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6509,"log_dups_size":0,"ondisk_log_size":6509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47538176,"num_objects":5803,"num_object_clones":0,"num_object_copies":23212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6509,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3600}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6645","reported_seq":6711,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.272978+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:15.272978+0000","last_peered":"2024-09-16T04:45:15.272978+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:15.272978+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6645,"log_dups_size":0,"ondisk_log_size":6645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48780556,"num_objects":5955,"num_object_clones":0,"num_object_copies":23820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3729}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6689","reported_seq":11245,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:12.287222+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:12.287222+0000","last_peered":"2024-09-16T04:45:12.287222+0000","last_clean":"2024-09-16T04:45:12.287222+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:12.287222+0000","last_undegraded":"2024-09-16T04:45:12.287222+0000","last_fullsized":"2024-09-16T04:45:12.287222+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49935628,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6726","reported_seq":7862,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.380171+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:15.380171+0000","last_peered":"2024-09-16T04:45:15.380171+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:15.380171+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49392152,"num_objects":6030,"num_object_clones":0,"num_object_copies":24120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1801,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6723,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":542,"num_bytes_recovered":4415488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4229}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6696","reported_seq":6779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:14.961286+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:14.961286+0000","last_peered":"2024-09-16T04:45:14.961286+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:14.961286+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6696,"log_dups_size":0,"ondisk_log_size":6696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49657100,"num_objects":6062,"num_object_clones":0,"num_object_copies":24248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6694,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3864}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6569","reported_seq":6635,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.319447+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:15.319447+0000","last_peered":"2024-09-16T04:45:15.319447+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:15.319447+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6569,"log_dups_size":0,"ondisk_log_size":6569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48508818,"num_objects":5922,"num_object_clones":0,"num_object_copies":23688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6566,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3737}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6473","reported_seq":6522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.358134+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:15.358134+0000","last_peered":"2024-09-16T04:45:15.358134+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:15.358134+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6473,"log_dups_size":0,"ondisk_log_size":6473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47987334,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6472,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3734}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6595","reported_seq":6670,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:14.916583+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:14.916583+0000","last_peered":"2024-09-16T04:45:14.916583+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:14.916583+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6595,"log_dups_size":0,"ondisk_log_size":6595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48642694,"num_objects":5938,"num_object_clones":0,"num_object_copies":23752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6594,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6710","reported_seq":6783,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.336420+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:15.336420+0000","last_peered":"2024-09-16T04:45:15.336420+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:15.336420+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49739020,"num_objects":6072,"num_object_clones":0,"num_object_copies":24288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6708,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3847}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6552","reported_seq":6636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:14.939034+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:14.939034+0000","last_peered":"2024-09-16T04:45:14.939034+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:14.939034+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6552,"log_dups_size":0,"ondisk_log_size":6552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48305420,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6550,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6490","reported_seq":6557,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.298923+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:15.298923+0000","last_peered":"2024-09-16T04:45:15.298923+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:15.298923+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6490,"log_dups_size":0,"ondisk_log_size":6490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47650060,"num_objects":5817,"num_object_clones":0,"num_object_copies":23268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6488,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":782001856,"num_objects":95465,"num_object_clones":0,"num_object_copies":381860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95465,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106022,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2844,"num_bytes_recovered":22749184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1448206336,"data_stored":1448206336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106049,"ondisk_log_size":106049,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747172,"kb_used_data":382292,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93624668,"statfs":{"total":96636764160,"available":95871660032,"internally_reserved":0,"allocated":391467008,"data_stored":391287039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373616335},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,0,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742920,"kb_used_data":382712,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93628920,"statfs":{"total":96636764160,"available":95876014080,"internally_reserved":0,"allocated":391897088,"data_stored":391717119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,3,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051630,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548504,"kb_used_data":269704,"kb_used_omap":4,"kb_used_meta":278779,"kb_avail":93823336,"statfs":{"total":96636764160,"available":96075096064,"internally_reserved":0,"allocated":276176896,"data_stored":275996927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":285470415},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735040,"kb_used_data":382704,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93636800,"statfs":{"total":96636764160,"available":95884083200,"internally_reserved":0,"allocated":391888896,"data_stored":391708927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":360771278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,0,3,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":390656000,"data_stored":390656000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275865600,"data_stored":275865600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391057408,"data_stored":391057408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":390627328,"data_stored":390627328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:16.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:45:16.867 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6659", 'reported_seq': 6712, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:15.363179+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:45:15.363179+0000', 'last_peered': '2024-09-16T04:45:15.363179+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:45:15.363179+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6659, 'log_dups_size': 0, 'ondisk_log_size': 6659, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49111040, 'num_objects': 5995, 'num_object_clones': 0, 'num_object_copies': 23980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6659, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3780}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:16.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:45:16.867 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6678", 'reported_seq': 6747, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:15.214706+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:45:15.214706+0000', 'last_peered': '2024-09-16T04:45:15.214706+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:45:15.214706+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6678, 'log_dups_size': 0, 'ondisk_log_size': 6678, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49116428, 'num_objects': 5996, 'num_object_clones': 0, 'num_object_copies': 23984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5996, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6676, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3786}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:16.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:45:16.867 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6664", 'reported_seq': 6743, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:14.982984+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:45:14.982984+0000', 'last_peered': '2024-09-16T04:45:14.982984+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:45:14.982984+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6664, 'log_dups_size': 0, 'ondisk_log_size': 6664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49199774, 'num_objects': 6007, 'num_object_clones': 0, 'num_object_copies': 24028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6007, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6663, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3781}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:16.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:45:16.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6668", 'reported_seq': 6737, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:15.167333+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:45:15.167333+0000', 'last_peered': '2024-09-16T04:45:15.167333+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:45:15.167333+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6668, 'log_dups_size': 0, 'ondisk_log_size': 6668, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49165580, 'num_objects': 6002, 'num_object_clones': 0, 'num_object_copies': 24008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6666, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3774}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:16.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:45:16.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6726", 'reported_seq': 6793, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:15.290992+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:45:15.290992+0000', 'last_peered': '2024-09-16T04:45:15.290992+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:45:15.290992+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6726, 'log_dups_size': 0, 'ondisk_log_size': 6726, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49272076, 'num_objects': 6015, 'num_object_clones': 0, 'num_object_copies': 24060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6015, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6724, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3737}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:16.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:45:16.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6509", 'reported_seq': 6562, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:15.279112+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:45:15.279112+0000', 'last_peered': '2024-09-16T04:45:15.279112+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:45:15.279112+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6509, 'log_dups_size': 0, 'ondisk_log_size': 6509, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47538176, 'num_objects': 5803, 'num_object_clones': 0, 'num_object_copies': 23212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2203, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6509, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3600}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:16.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:45:16.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6645", 'reported_seq': 6711, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:15.272978+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:45:15.272978+0000', 'last_peered': '2024-09-16T04:45:15.272978+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:45:15.272978+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6645, 'log_dups_size': 0, 'ondisk_log_size': 6645, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48780556, 'num_objects': 5955, 'num_object_clones': 0, 'num_object_copies': 23820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5955, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6643, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3729}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:16.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:45:16.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6726", 'reported_seq': 7862, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:15.380171+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:45:15.380171+0000', 'last_peered': '2024-09-16T04:45:15.380171+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:45:15.380171+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6726, 'log_dups_size': 0, 'ondisk_log_size': 6726, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49392152, 'num_objects': 6030, 'num_object_clones': 0, 'num_object_copies': 24120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1801, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6030, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6723, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 542, 'num_bytes_recovered': 4415488, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4229}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:16.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:45:16.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6696", 'reported_seq': 6779, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:14.961286+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:45:14.961286+0000', 'last_peered': '2024-09-16T04:45:14.961286+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:45:14.961286+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6696, 'log_dups_size': 0, 'ondisk_log_size': 6696, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49657100, 'num_objects': 6062, 'num_object_clones': 0, 'num_object_copies': 24248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6694, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3864}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:16.869 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:45:16.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6569", 'reported_seq': 6635, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:15.319447+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:45:15.319447+0000', 'last_peered': '2024-09-16T04:45:15.319447+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:45:15.319447+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6569, 'log_dups_size': 0, 'ondisk_log_size': 6569, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48508818, 'num_objects': 5922, 'num_object_clones': 0, 'num_object_copies': 23688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2188, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6566, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3737}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:16.869 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:45:16.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6473", 'reported_seq': 6522, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:15.358134+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:45:15.358134+0000', 'last_peered': '2024-09-16T04:45:15.358134+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:45:15.358134+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6473, 'log_dups_size': 0, 'ondisk_log_size': 6473, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47987334, 'num_objects': 5858, 'num_object_clones': 0, 'num_object_copies': 23432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6472, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3734}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:16.869 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:45:16.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6595", 'reported_seq': 6670, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:14.916583+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:45:14.916583+0000', 'last_peered': '2024-09-16T04:45:14.916583+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:45:14.916583+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6595, 'log_dups_size': 0, 'ondisk_log_size': 6595, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48642694, 'num_objects': 5938, 'num_object_clones': 0, 'num_object_copies': 23752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6594, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3713}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:16.869 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:45:16.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6710", 'reported_seq': 6783, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:15.336420+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:45:15.336420+0000', 'last_peered': '2024-09-16T04:45:15.336420+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:45:15.336420+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6710, 'log_dups_size': 0, 'ondisk_log_size': 6710, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49739020, 'num_objects': 6072, 'num_object_clones': 0, 'num_object_copies': 24288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6072, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6708, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3847}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:16.869 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:45:16.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6552", 'reported_seq': 6636, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:14.939034+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:45:14.939034+0000', 'last_peered': '2024-09-16T04:45:14.939034+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:45:14.939034+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6552, 'log_dups_size': 0, 'ondisk_log_size': 6552, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48305420, 'num_objects': 5897, 'num_object_clones': 0, 'num_object_copies': 23588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5897, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6550, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3767}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:16.870 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:45:16.870 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6490", 'reported_seq': 6557, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:15.298923+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:45:15.298923+0000', 'last_peered': '2024-09-16T04:45:15.298923+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:45:15.298923+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6490, 'log_dups_size': 0, 'ondisk_log_size': 6490, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47650060, 'num_objects': 5817, 'num_object_clones': 0, 'num_object_copies': 23268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6488, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3578}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:16.870 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:45:17.563 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:17.564 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32735/381860 objects degraded (8.573%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":329,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":95465,"data_bytes":782001856,"bytes_used":2840203264,"bytes_avail":383706853376,"bytes_total":386547056640,"degraded_objects":32735,"degraded_total":381860,"degraded_ratio":0.085725134866181329,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":108491,"recovering_keys_per_sec":0,"num_objects_recovered":195,"num_bytes_recovered":1302528,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":307},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:45:17.589 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:45:17.590 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:18.160 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:18.160 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:18.187 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-16T04:45:16.638265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":782001856,"num_objects":95465,"num_object_clones":0,"num_object_copies":381860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95465,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106022,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2844,"num_bytes_recovered":22749184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106049,"ondisk_log_size":106049,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773636,"kb_used_data":1417412,"kb_used_omap":14,"kb_used_meta":1356145,"kb_avail":374713724,"statfs":{"total":386547056640,"available":383706853376,"internally_reserved":0,"allocated":1451429888,"data_stored":1450710012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1388693041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2,6,12,9,0,0,0,0,0,0,2,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-30195712,"num_objects":-3686,"num_object_clones":0,"num_object_copies":-14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3686,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":1302528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005822"},"pg_stats":[{"pgid":"1.f","version":"33'6659","reported_seq":6712,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.363179+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:15.363179+0000","last_peered":"2024-09-16T04:45:15.363179+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:15.363179+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6659,"log_dups_size":0,"ondisk_log_size":6659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49111040,"num_objects":5995,"num_object_clones":0,"num_object_copies":23980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6659,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3780}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6678","reported_seq":6747,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.214706+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:15.214706+0000","last_peered":"2024-09-16T04:45:15.214706+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:15.214706+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6678,"log_dups_size":0,"ondisk_log_size":6678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49116428,"num_objects":5996,"num_object_clones":0,"num_object_copies":23984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6676,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6664","reported_seq":6743,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:14.982984+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:14.982984+0000","last_peered":"2024-09-16T04:45:14.982984+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:14.982984+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6664,"log_dups_size":0,"ondisk_log_size":6664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49199774,"num_objects":6007,"num_object_clones":0,"num_object_copies":24028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6663,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3781}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6668","reported_seq":6737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.167333+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:15.167333+0000","last_peered":"2024-09-16T04:45:15.167333+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:15.167333+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6668,"log_dups_size":0,"ondisk_log_size":6668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49165580,"num_objects":6002,"num_object_clones":0,"num_object_copies":24008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3774}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6726","reported_seq":6793,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.290992+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:15.290992+0000","last_peered":"2024-09-16T04:45:15.290992+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:15.290992+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49272076,"num_objects":6015,"num_object_clones":0,"num_object_copies":24060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3737}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6509","reported_seq":6562,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.279112+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:15.279112+0000","last_peered":"2024-09-16T04:45:15.279112+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:15.279112+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6509,"log_dups_size":0,"ondisk_log_size":6509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47538176,"num_objects":5803,"num_object_clones":0,"num_object_copies":23212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6509,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3600}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6645","reported_seq":6711,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.272978+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:15.272978+0000","last_peered":"2024-09-16T04:45:15.272978+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:15.272978+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6645,"log_dups_size":0,"ondisk_log_size":6645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48780556,"num_objects":5955,"num_object_clones":0,"num_object_copies":23820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3729}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6689","reported_seq":11245,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:12.287222+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:12.287222+0000","last_peered":"2024-09-16T04:45:12.287222+0000","last_clean":"2024-09-16T04:45:12.287222+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:12.287222+0000","last_undegraded":"2024-09-16T04:45:12.287222+0000","last_fullsized":"2024-09-16T04:45:12.287222+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49935628,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6726","reported_seq":7862,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.380171+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:15.380171+0000","last_peered":"2024-09-16T04:45:15.380171+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:15.380171+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49392152,"num_objects":6030,"num_object_clones":0,"num_object_copies":24120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1801,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6723,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":542,"num_bytes_recovered":4415488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4229}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6696","reported_seq":6779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:14.961286+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:14.961286+0000","last_peered":"2024-09-16T04:45:14.961286+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:14.961286+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6696,"log_dups_size":0,"ondisk_log_size":6696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49657100,"num_objects":6062,"num_object_clones":0,"num_object_copies":24248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6694,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3864}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6569","reported_seq":6635,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.319447+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:15.319447+0000","last_peered":"2024-09-16T04:45:15.319447+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:15.319447+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6569,"log_dups_size":0,"ondisk_log_size":6569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48508818,"num_objects":5922,"num_object_clones":0,"num_object_copies":23688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6566,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3737}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6473","reported_seq":6522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.358134+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:15.358134+0000","last_peered":"2024-09-16T04:45:15.358134+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:15.358134+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6473,"log_dups_size":0,"ondisk_log_size":6473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47987334,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6472,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3734}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6595","reported_seq":6670,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:14.916583+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:14.916583+0000","last_peered":"2024-09-16T04:45:14.916583+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:14.916583+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6595,"log_dups_size":0,"ondisk_log_size":6595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48642694,"num_objects":5938,"num_object_clones":0,"num_object_copies":23752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6594,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6710","reported_seq":6783,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.336420+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:15.336420+0000","last_peered":"2024-09-16T04:45:15.336420+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:15.336420+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49739020,"num_objects":6072,"num_object_clones":0,"num_object_copies":24288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6708,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3847}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6552","reported_seq":6636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:14.939034+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:14.939034+0000","last_peered":"2024-09-16T04:45:14.939034+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:14.939034+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6552,"log_dups_size":0,"ondisk_log_size":6552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48305420,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6550,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6490","reported_seq":6557,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:15.298923+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:15.298923+0000","last_peered":"2024-09-16T04:45:15.298923+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:15.298923+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6490,"log_dups_size":0,"ondisk_log_size":6490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47650060,"num_objects":5817,"num_object_clones":0,"num_object_copies":23268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6488,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":782001856,"num_objects":95465,"num_object_clones":0,"num_object_copies":381860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95465,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106022,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2844,"num_bytes_recovered":22749184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1448206336,"data_stored":1448206336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106049,"ondisk_log_size":106049,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747172,"kb_used_data":382292,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93624668,"statfs":{"total":96636764160,"available":95871660032,"internally_reserved":0,"allocated":391467008,"data_stored":391287039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373616335},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,0,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742920,"kb_used_data":382712,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93628920,"statfs":{"total":96636764160,"available":95876014080,"internally_reserved":0,"allocated":391897088,"data_stored":391717119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,3,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051630,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548504,"kb_used_data":269704,"kb_used_omap":4,"kb_used_meta":278779,"kb_avail":93823336,"statfs":{"total":96636764160,"available":96075096064,"internally_reserved":0,"allocated":276176896,"data_stored":275996927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":285470415},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735040,"kb_used_data":382704,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93636800,"statfs":{"total":96636764160,"available":95884083200,"internally_reserved":0,"allocated":391888896,"data_stored":391708927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":360771278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,0,3,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":390656000,"data_stored":390656000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275865600,"data_stored":275865600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391057408,"data_stored":391057408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":390627328,"data_stored":390627328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:21.189 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:21.799 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:21.800 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:21.819 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-16T04:45:20.639258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":772490944,"num_objects":94304,"num_object_clones":0,"num_object_copies":377216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94304,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107183,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2919,"num_bytes_recovered":23248896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107210,"ondisk_log_size":107210,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2813488,"kb_used_data":1403100,"kb_used_omap":14,"kb_used_meta":1410289,"kb_avail":374673872,"statfs":{"total":386547056640,"available":383666044928,"internally_reserved":0,"allocated":1436774400,"data_stored":1436054524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1444136497},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,4,9,2,6,4,0,0,0,3,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29458432,"num_objects":-3596,"num_object_clones":0,"num_object_copies":-14384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3596,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005707"},"pg_stats":[{"pgid":"1.f","version":"33'6724","reported_seq":6779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.216229+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:20.216229+0000","last_peered":"2024-09-16T04:45:20.216229+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:20.216229+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6724,"log_dups_size":0,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48578560,"num_objects":5930,"num_object_clones":0,"num_object_copies":23720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3716}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6752","reported_seq":6822,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.404154+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:20.404154+0000","last_peered":"2024-09-16T04:45:20.404154+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:20.404154+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6752,"log_dups_size":0,"ondisk_log_size":6752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48510220,"num_objects":5922,"num_object_clones":0,"num_object_copies":23688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6750,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6737","reported_seq":6816,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:19.942698+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:19.942698+0000","last_peered":"2024-09-16T04:45:19.942698+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:19.942698+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6737,"log_dups_size":0,"ondisk_log_size":6737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48601758,"num_objects":5934,"num_object_clones":0,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6736,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3708}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6754","reported_seq":6825,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.288887+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:20.288887+0000","last_peered":"2024-09-16T04:45:20.288887+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:20.288887+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6754,"log_dups_size":0,"ondisk_log_size":6754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48461068,"num_objects":5916,"num_object_clones":0,"num_object_copies":23664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6752,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6798","reported_seq":6867,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.326335+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:20.326335+0000","last_peered":"2024-09-16T04:45:20.326335+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:20.326335+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6798,"log_dups_size":0,"ondisk_log_size":6798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48682252,"num_objects":5943,"num_object_clones":0,"num_object_copies":23772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6796,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3666}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6571","reported_seq":6632,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.327349+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:20.327349+0000","last_peered":"2024-09-16T04:45:20.327349+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:20.327349+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6571,"log_dups_size":0,"ondisk_log_size":6571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47030272,"num_objects":5741,"num_object_clones":0,"num_object_copies":22964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6571,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6710","reported_seq":6778,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.360943+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:20.360943+0000","last_peered":"2024-09-16T04:45:20.360943+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:20.360943+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48248076,"num_objects":5890,"num_object_clones":0,"num_object_copies":23560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6708,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3665}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6778","reported_seq":11334,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:17.386235+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:17.386235+0000","last_peered":"2024-09-16T04:45:17.386235+0000","last_clean":"2024-09-16T04:45:17.386235+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:17.386235+0000","last_undegraded":"2024-09-16T04:45:17.386235+0000","last_fullsized":"2024-09-16T04:45:17.386235+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6778,"log_dups_size":0,"ondisk_log_size":6778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:18.093581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49206540,"num_objects":6007,"num_object_clones":0,"num_object_copies":24028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6776,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6804","reported_seq":8063,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.395228+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:20.395228+0000","last_peered":"2024-09-16T04:45:20.395228+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:20.395228+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6804,"log_dups_size":0,"ondisk_log_size":6804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48753176,"num_objects":5952,"num_object_clones":0,"num_object_copies":23808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6801,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":4915200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4212}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6778","reported_seq":6861,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:19.986419+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:19.986419+0000","last_peered":"2024-09-16T04:45:19.986419+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:19.986419+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6778,"log_dups_size":0,"ondisk_log_size":6778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48985356,"num_objects":5980,"num_object_clones":0,"num_object_copies":23920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6776,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3782}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6641","reported_seq":6712,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.382734+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:20.382734+0000","last_peered":"2024-09-16T04:45:20.382734+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:20.382734+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6641,"log_dups_size":0,"ondisk_log_size":6641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47918994,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6638,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6539","reported_seq":6590,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.326299+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:20.326299+0000","last_peered":"2024-09-16T04:45:20.326299+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:20.326299+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6539,"log_dups_size":0,"ondisk_log_size":6539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47446662,"num_objects":5792,"num_object_clones":0,"num_object_copies":23168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6538,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3669}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6664","reported_seq":6739,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:19.968716+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:19.968716+0000","last_peered":"2024-09-16T04:45:19.968716+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:19.968716+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6664,"log_dups_size":0,"ondisk_log_size":6664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48077446,"num_objects":5869,"num_object_clones":0,"num_object_copies":23476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6663,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3644}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6781","reported_seq":6860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.333476+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:20.333476+0000","last_peered":"2024-09-16T04:45:20.333476+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:20.333476+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6781,"log_dups_size":0,"ondisk_log_size":6781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49157388,"num_objects":6001,"num_object_clones":0,"num_object_copies":24004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6779,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3779}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6618","reported_seq":6702,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:19.775281+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:19.775281+0000","last_peered":"2024-09-16T04:45:19.775281+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:19.775281+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6618,"log_dups_size":0,"ondisk_log_size":6618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47764748,"num_objects":5831,"num_object_clones":0,"num_object_copies":23324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6616,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3701}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6561","reported_seq":6628,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.177280+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:20.177280+0000","last_peered":"2024-09-16T04:45:20.177280+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:20.177280+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6561,"log_dups_size":0,"ondisk_log_size":6561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47068428,"num_objects":5746,"num_object_clones":0,"num_object_copies":22984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6559,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3507}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":772490944,"num_objects":94304,"num_object_clones":0,"num_object_copies":377216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94304,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107183,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2919,"num_bytes_recovered":23248896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1433559040,"data_stored":1433559040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107210,"ondisk_log_size":107210,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764896,"kb_used_data":377796,"kb_used_omap":4,"kb_used_meta":387067,"kb_avail":93606944,"statfs":{"total":96636764160,"available":95853510656,"internally_reserved":0,"allocated":386863104,"data_stored":386683135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":396357327},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760428,"kb_used_data":378012,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93611412,"statfs":{"total":96636764160,"available":95858085888,"internally_reserved":0,"allocated":387084288,"data_stored":386904319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,0,0,0,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051631,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535592,"kb_used_data":269272,"kb_used_omap":4,"kb_used_meta":266299,"kb_avail":93836248,"statfs":{"total":96636764160,"available":96088317952,"internally_reserved":0,"allocated":275734528,"data_stored":275554559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272690895},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,8,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752572,"kb_used_data":378020,"kb_used_omap":4,"kb_used_meta":374523,"kb_avail":93619268,"statfs":{"total":96636764160,"available":95866130432,"internally_reserved":0,"allocated":387092480,"data_stored":386912511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":383512270},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,0,0,3,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385921024,"data_stored":385921024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275423232,"data_stored":275423232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386297856,"data_stored":386297856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":385916928,"data_stored":385916928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:21.820 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:22.381 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:22.382 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:22.404 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-16T04:45:20.639258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":772490944,"num_objects":94304,"num_object_clones":0,"num_object_copies":377216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94304,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107183,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2919,"num_bytes_recovered":23248896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107210,"ondisk_log_size":107210,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2813488,"kb_used_data":1403100,"kb_used_omap":14,"kb_used_meta":1410289,"kb_avail":374673872,"statfs":{"total":386547056640,"available":383666044928,"internally_reserved":0,"allocated":1436774400,"data_stored":1436054524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1444136497},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,4,9,2,6,4,0,0,0,3,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29458432,"num_objects":-3596,"num_object_clones":0,"num_object_copies":-14384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3596,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005707"},"pg_stats":[{"pgid":"1.f","version":"33'6724","reported_seq":6779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.216229+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:20.216229+0000","last_peered":"2024-09-16T04:45:20.216229+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:20.216229+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6724,"log_dups_size":0,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48578560,"num_objects":5930,"num_object_clones":0,"num_object_copies":23720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3716}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6752","reported_seq":6822,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.404154+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:20.404154+0000","last_peered":"2024-09-16T04:45:20.404154+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:20.404154+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6752,"log_dups_size":0,"ondisk_log_size":6752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48510220,"num_objects":5922,"num_object_clones":0,"num_object_copies":23688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6750,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6737","reported_seq":6816,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:19.942698+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:19.942698+0000","last_peered":"2024-09-16T04:45:19.942698+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:19.942698+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6737,"log_dups_size":0,"ondisk_log_size":6737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48601758,"num_objects":5934,"num_object_clones":0,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6736,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3708}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6754","reported_seq":6825,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.288887+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:20.288887+0000","last_peered":"2024-09-16T04:45:20.288887+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:20.288887+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6754,"log_dups_size":0,"ondisk_log_size":6754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48461068,"num_objects":5916,"num_object_clones":0,"num_object_copies":23664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6752,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6798","reported_seq":6867,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.326335+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:20.326335+0000","last_peered":"2024-09-16T04:45:20.326335+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:20.326335+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6798,"log_dups_size":0,"ondisk_log_size":6798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48682252,"num_objects":5943,"num_object_clones":0,"num_object_copies":23772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6796,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3666}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6571","reported_seq":6632,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.327349+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:20.327349+0000","last_peered":"2024-09-16T04:45:20.327349+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:20.327349+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6571,"log_dups_size":0,"ondisk_log_size":6571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47030272,"num_objects":5741,"num_object_clones":0,"num_object_copies":22964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6571,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6710","reported_seq":6778,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.360943+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:20.360943+0000","last_peered":"2024-09-16T04:45:20.360943+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:20.360943+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48248076,"num_objects":5890,"num_object_clones":0,"num_object_copies":23560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6708,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3665}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6778","reported_seq":11334,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:17.386235+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:17.386235+0000","last_peered":"2024-09-16T04:45:17.386235+0000","last_clean":"2024-09-16T04:45:17.386235+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:17.386235+0000","last_undegraded":"2024-09-16T04:45:17.386235+0000","last_fullsized":"2024-09-16T04:45:17.386235+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6778,"log_dups_size":0,"ondisk_log_size":6778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:18.093581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49206540,"num_objects":6007,"num_object_clones":0,"num_object_copies":24028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6776,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6804","reported_seq":8063,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.395228+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:20.395228+0000","last_peered":"2024-09-16T04:45:20.395228+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:20.395228+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6804,"log_dups_size":0,"ondisk_log_size":6804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48753176,"num_objects":5952,"num_object_clones":0,"num_object_copies":23808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6801,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":4915200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4212}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6778","reported_seq":6861,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:19.986419+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:19.986419+0000","last_peered":"2024-09-16T04:45:19.986419+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:19.986419+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6778,"log_dups_size":0,"ondisk_log_size":6778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48985356,"num_objects":5980,"num_object_clones":0,"num_object_copies":23920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6776,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3782}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6641","reported_seq":6712,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.382734+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:20.382734+0000","last_peered":"2024-09-16T04:45:20.382734+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:20.382734+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6641,"log_dups_size":0,"ondisk_log_size":6641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47918994,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6638,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6539","reported_seq":6590,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.326299+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:20.326299+0000","last_peered":"2024-09-16T04:45:20.326299+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:20.326299+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6539,"log_dups_size":0,"ondisk_log_size":6539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47446662,"num_objects":5792,"num_object_clones":0,"num_object_copies":23168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6538,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3669}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6664","reported_seq":6739,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:19.968716+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:19.968716+0000","last_peered":"2024-09-16T04:45:19.968716+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:19.968716+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6664,"log_dups_size":0,"ondisk_log_size":6664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48077446,"num_objects":5869,"num_object_clones":0,"num_object_copies":23476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6663,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3644}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6781","reported_seq":6860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.333476+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:20.333476+0000","last_peered":"2024-09-16T04:45:20.333476+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:20.333476+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6781,"log_dups_size":0,"ondisk_log_size":6781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49157388,"num_objects":6001,"num_object_clones":0,"num_object_copies":24004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6779,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3779}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6618","reported_seq":6702,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:19.775281+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:19.775281+0000","last_peered":"2024-09-16T04:45:19.775281+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:19.775281+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6618,"log_dups_size":0,"ondisk_log_size":6618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47764748,"num_objects":5831,"num_object_clones":0,"num_object_copies":23324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6616,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3701}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6561","reported_seq":6628,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.177280+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:20.177280+0000","last_peered":"2024-09-16T04:45:20.177280+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:20.177280+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6561,"log_dups_size":0,"ondisk_log_size":6561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47068428,"num_objects":5746,"num_object_clones":0,"num_object_copies":22984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6559,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3507}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":772490944,"num_objects":94304,"num_object_clones":0,"num_object_copies":377216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94304,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107183,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2919,"num_bytes_recovered":23248896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1433559040,"data_stored":1433559040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107210,"ondisk_log_size":107210,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764896,"kb_used_data":377796,"kb_used_omap":4,"kb_used_meta":387067,"kb_avail":93606944,"statfs":{"total":96636764160,"available":95853510656,"internally_reserved":0,"allocated":386863104,"data_stored":386683135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":396357327},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760428,"kb_used_data":378012,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93611412,"statfs":{"total":96636764160,"available":95858085888,"internally_reserved":0,"allocated":387084288,"data_stored":386904319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,0,0,0,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051631,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535592,"kb_used_data":269272,"kb_used_omap":4,"kb_used_meta":266299,"kb_avail":93836248,"statfs":{"total":96636764160,"available":96088317952,"internally_reserved":0,"allocated":275734528,"data_stored":275554559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272690895},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,8,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752572,"kb_used_data":378020,"kb_used_omap":4,"kb_used_meta":374523,"kb_avail":93619268,"statfs":{"total":96636764160,"available":95866130432,"internally_reserved":0,"allocated":387092480,"data_stored":386912511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":383512270},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,0,0,3,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385921024,"data_stored":385921024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275423232,"data_stored":275423232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386297856,"data_stored":386297856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":385916928,"data_stored":385916928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:22.405 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:45:22.405 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6724", 'reported_seq': 6779, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:20.216229+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:45:20.216229+0000', 'last_peered': '2024-09-16T04:45:20.216229+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:45:20.216229+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6724, 'log_dups_size': 0, 'ondisk_log_size': 6724, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48578560, 'num_objects': 5930, 'num_object_clones': 0, 'num_object_copies': 23720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5930, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6724, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3716}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:22.405 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:45:22.405 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6752", 'reported_seq': 6822, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:20.404154+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:45:20.404154+0000', 'last_peered': '2024-09-16T04:45:20.404154+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:45:20.404154+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6752, 'log_dups_size': 0, 'ondisk_log_size': 6752, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48510220, 'num_objects': 5922, 'num_object_clones': 0, 'num_object_copies': 23688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2211, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6750, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3713}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:22.406 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:45:22.406 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6737", 'reported_seq': 6816, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:19.942698+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:45:19.942698+0000', 'last_peered': '2024-09-16T04:45:19.942698+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:45:19.942698+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6737, 'log_dups_size': 0, 'ondisk_log_size': 6737, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48601758, 'num_objects': 5934, 'num_object_clones': 0, 'num_object_copies': 23736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5934, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6736, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3708}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:22.406 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:45:22.406 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6754", 'reported_seq': 6825, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:20.288887+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:45:20.288887+0000', 'last_peered': '2024-09-16T04:45:20.288887+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:45:20.288887+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6754, 'log_dups_size': 0, 'ondisk_log_size': 6754, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48461068, 'num_objects': 5916, 'num_object_clones': 0, 'num_object_copies': 23664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6752, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3689}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:22.406 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:45:22.406 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6798", 'reported_seq': 6867, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:20.326335+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:45:20.326335+0000', 'last_peered': '2024-09-16T04:45:20.326335+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:45:20.326335+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6798, 'log_dups_size': 0, 'ondisk_log_size': 6798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48682252, 'num_objects': 5943, 'num_object_clones': 0, 'num_object_copies': 23772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6796, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3666}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:22.406 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:45:22.406 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6571", 'reported_seq': 6632, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:20.327349+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:45:20.327349+0000', 'last_peered': '2024-09-16T04:45:20.327349+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:45:20.327349+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6571, 'log_dups_size': 0, 'ondisk_log_size': 6571, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47030272, 'num_objects': 5741, 'num_object_clones': 0, 'num_object_copies': 22964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6571, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3541}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:22.406 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:45:22.406 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6710", 'reported_seq': 6778, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:20.360943+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:45:20.360943+0000', 'last_peered': '2024-09-16T04:45:20.360943+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:45:20.360943+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6710, 'log_dups_size': 0, 'ondisk_log_size': 6710, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48248076, 'num_objects': 5890, 'num_object_clones': 0, 'num_object_copies': 23560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6708, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3665}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:22.406 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:45:22.407 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6804", 'reported_seq': 8063, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:20.395228+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:45:20.395228+0000', 'last_peered': '2024-09-16T04:45:20.395228+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:45:20.395228+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6804, 'log_dups_size': 0, 'ondisk_log_size': 6804, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48753176, 'num_objects': 5952, 'num_object_clones': 0, 'num_object_copies': 23808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1740, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5952, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6801, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 603, 'num_bytes_recovered': 4915200, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4212}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:22.407 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:45:22.407 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6778", 'reported_seq': 6861, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:19.986419+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:45:19.986419+0000', 'last_peered': '2024-09-16T04:45:19.986419+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:45:19.986419+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6778, 'log_dups_size': 0, 'ondisk_log_size': 6778, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48985356, 'num_objects': 5980, 'num_object_clones': 0, 'num_object_copies': 23920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6776, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3782}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:22.407 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:45:22.407 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6641", 'reported_seq': 6712, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:20.382734+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:45:20.382734+0000', 'last_peered': '2024-09-16T04:45:20.382734+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:45:20.382734+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6641, 'log_dups_size': 0, 'ondisk_log_size': 6641, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47918994, 'num_objects': 5850, 'num_object_clones': 0, 'num_object_copies': 23400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6638, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3667}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:22.407 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:45:22.407 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6539", 'reported_seq': 6590, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:20.326299+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:45:20.326299+0000', 'last_peered': '2024-09-16T04:45:20.326299+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:45:20.326299+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6539, 'log_dups_size': 0, 'ondisk_log_size': 6539, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47446662, 'num_objects': 5792, 'num_object_clones': 0, 'num_object_copies': 23168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2124, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5792, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6538, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3669}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:22.407 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:45:22.407 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6664", 'reported_seq': 6739, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:19.968716+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:45:19.968716+0000', 'last_peered': '2024-09-16T04:45:19.968716+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:45:19.968716+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6664, 'log_dups_size': 0, 'ondisk_log_size': 6664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48077446, 'num_objects': 5869, 'num_object_clones': 0, 'num_object_copies': 23476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5869, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6663, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3644}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:22.407 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:45:22.408 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6781", 'reported_seq': 6860, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:20.333476+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:45:20.333476+0000', 'last_peered': '2024-09-16T04:45:20.333476+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:45:20.333476+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6781, 'log_dups_size': 0, 'ondisk_log_size': 6781, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49157388, 'num_objects': 6001, 'num_object_clones': 0, 'num_object_copies': 24004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6779, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3779}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:22.408 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:45:22.408 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6618", 'reported_seq': 6702, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:19.775281+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:45:19.775281+0000', 'last_peered': '2024-09-16T04:45:19.775281+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:45:19.775281+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6618, 'log_dups_size': 0, 'ondisk_log_size': 6618, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47764748, 'num_objects': 5831, 'num_object_clones': 0, 'num_object_copies': 23324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6616, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3701}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:22.408 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:45:22.408 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6561", 'reported_seq': 6628, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:20.177280+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:45:20.177280+0000', 'last_peered': '2024-09-16T04:45:20.177280+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:45:20.177280+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6561, 'log_dups_size': 0, 'ondisk_log_size': 6561, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47068428, 'num_objects': 5746, 'num_object_clones': 0, 'num_object_copies': 22984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5746, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6559, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3507}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:22.408 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:45:23.112 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:23.113 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32660/376952 objects degraded (8.664%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":335,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":94238,"data_bytes":771950272,"bytes_used":2884915200,"bytes_avail":383662141440,"bytes_total":386547056640,"degraded_objects":32660,"degraded_total":376952,"degraded_ratio":0.086642331119081456,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":74382,"recovering_keys_per_sec":0,"num_objects_recovered":136,"num_bytes_recovered":892928,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":205},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:45:23.142 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:45:23.142 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:23.766 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:23.767 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:23.794 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-16T04:45:22.639815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":771950272,"num_objects":94238,"num_object_clones":0,"num_object_copies":376952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94238,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107249,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2919,"num_bytes_recovered":23248896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107276,"ondisk_log_size":107276,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2817300,"kb_used_data":1402752,"kb_used_omap":14,"kb_used_meta":1414449,"kb_avail":374670060,"statfs":{"total":386547056640,"available":383662141440,"internally_reserved":0,"allocated":1436418048,"data_stored":1435698172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1448396337},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,1,2,1,5,4,0,0,0,3,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20185088,"num_objects":-2464,"num_object_clones":0,"num_object_copies":-9856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2464,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":892928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004508"},"pg_stats":[{"pgid":"1.f","version":"33'6724","reported_seq":6779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.216229+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:20.216229+0000","last_peered":"2024-09-16T04:45:20.216229+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:20.216229+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6724,"log_dups_size":0,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48578560,"num_objects":5930,"num_object_clones":0,"num_object_copies":23720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3716}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6752","reported_seq":6822,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.404154+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:20.404154+0000","last_peered":"2024-09-16T04:45:20.404154+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:20.404154+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6752,"log_dups_size":0,"ondisk_log_size":6752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48510220,"num_objects":5922,"num_object_clones":0,"num_object_copies":23688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6750,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6737","reported_seq":6816,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:19.942698+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:19.942698+0000","last_peered":"2024-09-16T04:45:19.942698+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:19.942698+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6737,"log_dups_size":0,"ondisk_log_size":6737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48601758,"num_objects":5934,"num_object_clones":0,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6736,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3708}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6754","reported_seq":6825,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.288887+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:20.288887+0000","last_peered":"2024-09-16T04:45:20.288887+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:20.288887+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6754,"log_dups_size":0,"ondisk_log_size":6754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48461068,"num_objects":5916,"num_object_clones":0,"num_object_copies":23664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6752,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6798","reported_seq":6867,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.326335+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:20.326335+0000","last_peered":"2024-09-16T04:45:20.326335+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:20.326335+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6798,"log_dups_size":0,"ondisk_log_size":6798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48682252,"num_objects":5943,"num_object_clones":0,"num_object_copies":23772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6796,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3666}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6571","reported_seq":6632,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.327349+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:20.327349+0000","last_peered":"2024-09-16T04:45:20.327349+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:20.327349+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6571,"log_dups_size":0,"ondisk_log_size":6571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47030272,"num_objects":5741,"num_object_clones":0,"num_object_copies":22964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6571,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6710","reported_seq":6778,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.360943+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:20.360943+0000","last_peered":"2024-09-16T04:45:20.360943+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:20.360943+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48248076,"num_objects":5890,"num_object_clones":0,"num_object_copies":23560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6708,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3665}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6844","reported_seq":11400,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:22.160898+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:22.160898+0000","last_peered":"2024-09-16T04:45:22.160898+0000","last_clean":"2024-09-16T04:45:22.160898+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:22.160898+0000","last_undegraded":"2024-09-16T04:45:22.160898+0000","last_fullsized":"2024-09-16T04:45:22.160898+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:24.060166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48665868,"num_objects":5941,"num_object_clones":0,"num_object_copies":23764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6842,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6804","reported_seq":8063,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.395228+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:20.395228+0000","last_peered":"2024-09-16T04:45:20.395228+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:20.395228+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6804,"log_dups_size":0,"ondisk_log_size":6804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48753176,"num_objects":5952,"num_object_clones":0,"num_object_copies":23808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6801,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":4915200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4212}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6778","reported_seq":6861,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:19.986419+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:19.986419+0000","last_peered":"2024-09-16T04:45:19.986419+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:19.986419+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6778,"log_dups_size":0,"ondisk_log_size":6778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48985356,"num_objects":5980,"num_object_clones":0,"num_object_copies":23920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6776,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3782}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6641","reported_seq":6712,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.382734+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:20.382734+0000","last_peered":"2024-09-16T04:45:20.382734+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:20.382734+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6641,"log_dups_size":0,"ondisk_log_size":6641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47918994,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6638,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6539","reported_seq":6590,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.326299+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:20.326299+0000","last_peered":"2024-09-16T04:45:20.326299+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:20.326299+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6539,"log_dups_size":0,"ondisk_log_size":6539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47446662,"num_objects":5792,"num_object_clones":0,"num_object_copies":23168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6538,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3669}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6664","reported_seq":6739,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:19.968716+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:19.968716+0000","last_peered":"2024-09-16T04:45:19.968716+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:19.968716+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6664,"log_dups_size":0,"ondisk_log_size":6664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48077446,"num_objects":5869,"num_object_clones":0,"num_object_copies":23476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6663,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3644}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6781","reported_seq":6860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.333476+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:20.333476+0000","last_peered":"2024-09-16T04:45:20.333476+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:20.333476+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6781,"log_dups_size":0,"ondisk_log_size":6781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49157388,"num_objects":6001,"num_object_clones":0,"num_object_copies":24004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6779,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3779}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6618","reported_seq":6702,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:19.775281+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:19.775281+0000","last_peered":"2024-09-16T04:45:19.775281+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:19.775281+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6618,"log_dups_size":0,"ondisk_log_size":6618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47764748,"num_objects":5831,"num_object_clones":0,"num_object_copies":23324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6616,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3701}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6561","reported_seq":6628,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:20.177280+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:20.177280+0000","last_peered":"2024-09-16T04:45:20.177280+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:20.177280+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6561,"log_dups_size":0,"ondisk_log_size":6561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47068428,"num_objects":5746,"num_object_clones":0,"num_object_copies":22984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6559,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3507}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":771950272,"num_objects":94238,"num_object_clones":0,"num_object_copies":376952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94238,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107249,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2919,"num_bytes_recovered":23248896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1433182208,"data_stored":1433182208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107276,"ondisk_log_size":107276,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764896,"kb_used_data":377796,"kb_used_omap":4,"kb_used_meta":387067,"kb_avail":93606944,"statfs":{"total":96636764160,"available":95853510656,"internally_reserved":0,"allocated":386863104,"data_stored":386683135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":396357327},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760428,"kb_used_data":378012,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93611412,"statfs":{"total":96636764160,"available":95858085888,"internally_reserved":0,"allocated":387084288,"data_stored":386904319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,0,0,0,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539404,"kb_used_data":268924,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93832436,"statfs":{"total":96636764160,"available":96084414464,"internally_reserved":0,"allocated":275378176,"data_stored":275198207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276950735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752572,"kb_used_data":378020,"kb_used_omap":4,"kb_used_meta":374523,"kb_avail":93619268,"statfs":{"total":96636764160,"available":95866130432,"internally_reserved":0,"allocated":387092480,"data_stored":386912511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":383512270},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,0,0,3,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385921024,"data_stored":385921024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275046400,"data_stored":275046400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386297856,"data_stored":386297856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":385916928,"data_stored":385916928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:26.795 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:27.362 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:27.362 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:27.381 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-16T04:45:26.640758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":762889920,"num_objects":93132,"num_object_clones":0,"num_object_copies":372528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108355,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2997,"num_bytes_recovered":23789568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108382,"ondisk_log_size":108382,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2782432,"kb_used_data":1388696,"kb_used_omap":14,"kb_used_meta":1393649,"kb_avail":374704928,"statfs":{"total":386547056640,"available":383697846272,"internally_reserved":0,"allocated":1422024704,"data_stored":1421304828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1427097137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,5,6,7,8,1,4,0,0,0,0,3,5,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28598272,"num_objects":-3491,"num_object_clones":0,"num_object_copies":-13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3491,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002971"},"pg_stats":[{"pgid":"1.f","version":"33'6795","reported_seq":6854,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.407696+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:25.407696+0000","last_peered":"2024-09-16T04:45:25.407696+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:25.407696+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6795,"log_dups_size":0,"ondisk_log_size":6795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47996928,"num_objects":5859,"num_object_clones":0,"num_object_copies":23436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6795,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6833","reported_seq":6904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.361778+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:25.361778+0000","last_peered":"2024-09-16T04:45:25.361778+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:25.361778+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6833,"log_dups_size":0,"ondisk_log_size":6833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47846668,"num_objects":5841,"num_object_clones":0,"num_object_copies":23364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6831,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6815","reported_seq":6894,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:24.981202+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:24.981202+0000","last_peered":"2024-09-16T04:45:24.981202+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:24.981202+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6815,"log_dups_size":0,"ondisk_log_size":6815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47962782,"num_objects":5856,"num_object_clones":0,"num_object_copies":23424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6814,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3630}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6828","reported_seq":6899,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.373638+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:25.373638+0000","last_peered":"2024-09-16T04:45:25.373638+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:25.373638+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6828,"log_dups_size":0,"ondisk_log_size":6828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47854860,"num_objects":5842,"num_object_clones":0,"num_object_copies":23368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6826,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6870","reported_seq":6946,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.476802+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:25.476802+0000","last_peered":"2024-09-16T04:45:25.476802+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:25.476802+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6870,"log_dups_size":0,"ondisk_log_size":6870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48092428,"num_objects":5871,"num_object_clones":0,"num_object_copies":23484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6868,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6642","reported_seq":6703,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.401282+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:25.401282+0000","last_peered":"2024-09-16T04:45:25.401282+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:25.401282+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6642,"log_dups_size":0,"ondisk_log_size":6642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46448640,"num_objects":5670,"num_object_clones":0,"num_object_copies":22680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6642,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3470}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6790","reported_seq":6858,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.328648+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:25.328648+0000","last_peered":"2024-09-16T04:45:25.328648+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:25.328648+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6790,"log_dups_size":0,"ondisk_log_size":6790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47592716,"num_objects":5810,"num_object_clones":0,"num_object_copies":23240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3585}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6844","reported_seq":11400,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:22.160898+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:22.160898+0000","last_peered":"2024-09-16T04:45:22.160898+0000","last_clean":"2024-09-16T04:45:22.160898+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:22.160898+0000","last_undegraded":"2024-09-16T04:45:22.160898+0000","last_fullsized":"2024-09-16T04:45:22.160898+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:24.060166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48665868,"num_objects":5941,"num_object_clones":0,"num_object_copies":23764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6842,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6882","reported_seq":8273,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.474135+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:25.474135+0000","last_peered":"2024-09-16T04:45:25.474135+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:25.474135+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6882,"log_dups_size":0,"ondisk_log_size":6882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48114200,"num_objects":5874,"num_object_clones":0,"num_object_copies":23496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6879,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":669,"num_bytes_recovered":5455872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4200}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6861","reported_seq":6944,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.037570+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:25.037570+0000","last_peered":"2024-09-16T04:45:25.037570+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:25.037570+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6861,"log_dups_size":0,"ondisk_log_size":6861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48305420,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6859,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3699}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6713","reported_seq":6788,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.408037+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:25.408037+0000","last_peered":"2024-09-16T04:45:25.408037+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:25.408037+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6713,"log_dups_size":0,"ondisk_log_size":6713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47329170,"num_objects":5778,"num_object_clones":0,"num_object_copies":23112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6710,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6604","reported_seq":6657,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.348211+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:25.348211+0000","last_peered":"2024-09-16T04:45:25.348211+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:25.348211+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6604,"log_dups_size":0,"ondisk_log_size":6604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46914182,"num_objects":5727,"num_object_clones":0,"num_object_copies":22908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6603,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3605}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6729","reported_seq":6804,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.048090+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:25.048090+0000","last_peered":"2024-09-16T04:45:25.048090+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:25.048090+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6729,"log_dups_size":0,"ondisk_log_size":6729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47544966,"num_objects":5804,"num_object_clones":0,"num_object_copies":23216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6728,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3579}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6855","reported_seq":6937,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.381018+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:25.381018+0000","last_peered":"2024-09-16T04:45:25.381018+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:25.381018+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48551180,"num_objects":5927,"num_object_clones":0,"num_object_copies":23708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3707}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6677","reported_seq":6761,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:24.832300+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:24.832300+0000","last_peered":"2024-09-16T04:45:24.832300+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:24.832300+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6677,"log_dups_size":0,"ondisk_log_size":6677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47281420,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6675,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3642}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6644","reported_seq":6711,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.329194+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:25.329194+0000","last_peered":"2024-09-16T04:45:25.329194+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:25.329194+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6644,"log_dups_size":0,"ondisk_log_size":6644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46388492,"num_objects":5663,"num_object_clones":0,"num_object_copies":22652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6642,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":762889920,"num_objects":93132,"num_object_clones":0,"num_object_copies":372528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108355,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2997,"num_bytes_recovered":23789568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1418842112,"data_stored":1418842112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108382,"ondisk_log_size":108382,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747668,"kb_used_data":373052,"kb_used_omap":4,"kb_used_meta":374587,"kb_avail":93624172,"statfs":{"total":96636764160,"available":95871152128,"internally_reserved":0,"allocated":382005248,"data_stored":381825279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":383577807},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,3,5,1,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743324,"kb_used_data":373388,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93628516,"statfs":{"total":96636764160,"available":95875600384,"internally_reserved":0,"allocated":382349312,"data_stored":382169343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,0,0,2,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539404,"kb_used_data":268924,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93832436,"statfs":{"total":96636764160,"available":96084414464,"internally_reserved":0,"allocated":275378176,"data_stored":275198207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276950735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752036,"kb_used_data":373332,"kb_used_omap":4,"kb_used_meta":378683,"kb_avail":93619804,"statfs":{"total":96636764160,"available":95866679296,"internally_reserved":0,"allocated":382291968,"data_stored":382111999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":387772110},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2,3,0,2,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":381165568,"data_stored":381165568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275046400,"data_stored":275046400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381513728,"data_stored":381513728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":381116416,"data_stored":381116416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:27.382 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:27.991 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:27.991 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:28.011 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-16T04:45:26.640758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":762889920,"num_objects":93132,"num_object_clones":0,"num_object_copies":372528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108355,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2997,"num_bytes_recovered":23789568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108382,"ondisk_log_size":108382,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2782432,"kb_used_data":1388696,"kb_used_omap":14,"kb_used_meta":1393649,"kb_avail":374704928,"statfs":{"total":386547056640,"available":383697846272,"internally_reserved":0,"allocated":1422024704,"data_stored":1421304828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1427097137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,5,6,7,8,1,4,0,0,0,0,3,5,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28598272,"num_objects":-3491,"num_object_clones":0,"num_object_copies":-13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3491,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002971"},"pg_stats":[{"pgid":"1.f","version":"33'6795","reported_seq":6854,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.407696+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:25.407696+0000","last_peered":"2024-09-16T04:45:25.407696+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:25.407696+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6795,"log_dups_size":0,"ondisk_log_size":6795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47996928,"num_objects":5859,"num_object_clones":0,"num_object_copies":23436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6795,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6833","reported_seq":6904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.361778+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:25.361778+0000","last_peered":"2024-09-16T04:45:25.361778+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:25.361778+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6833,"log_dups_size":0,"ondisk_log_size":6833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47846668,"num_objects":5841,"num_object_clones":0,"num_object_copies":23364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6831,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6815","reported_seq":6894,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:24.981202+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:24.981202+0000","last_peered":"2024-09-16T04:45:24.981202+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:24.981202+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6815,"log_dups_size":0,"ondisk_log_size":6815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47962782,"num_objects":5856,"num_object_clones":0,"num_object_copies":23424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6814,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3630}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6828","reported_seq":6899,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.373638+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:25.373638+0000","last_peered":"2024-09-16T04:45:25.373638+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:25.373638+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6828,"log_dups_size":0,"ondisk_log_size":6828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47854860,"num_objects":5842,"num_object_clones":0,"num_object_copies":23368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6826,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6870","reported_seq":6946,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.476802+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:25.476802+0000","last_peered":"2024-09-16T04:45:25.476802+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:25.476802+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6870,"log_dups_size":0,"ondisk_log_size":6870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48092428,"num_objects":5871,"num_object_clones":0,"num_object_copies":23484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6868,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6642","reported_seq":6703,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.401282+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:25.401282+0000","last_peered":"2024-09-16T04:45:25.401282+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:25.401282+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6642,"log_dups_size":0,"ondisk_log_size":6642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46448640,"num_objects":5670,"num_object_clones":0,"num_object_copies":22680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6642,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3470}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6790","reported_seq":6858,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.328648+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:25.328648+0000","last_peered":"2024-09-16T04:45:25.328648+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:25.328648+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6790,"log_dups_size":0,"ondisk_log_size":6790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47592716,"num_objects":5810,"num_object_clones":0,"num_object_copies":23240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3585}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6844","reported_seq":11400,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:22.160898+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:22.160898+0000","last_peered":"2024-09-16T04:45:22.160898+0000","last_clean":"2024-09-16T04:45:22.160898+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:22.160898+0000","last_undegraded":"2024-09-16T04:45:22.160898+0000","last_fullsized":"2024-09-16T04:45:22.160898+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:24.060166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48665868,"num_objects":5941,"num_object_clones":0,"num_object_copies":23764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6842,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6882","reported_seq":8273,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.474135+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:25.474135+0000","last_peered":"2024-09-16T04:45:25.474135+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:25.474135+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6882,"log_dups_size":0,"ondisk_log_size":6882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48114200,"num_objects":5874,"num_object_clones":0,"num_object_copies":23496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6879,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":669,"num_bytes_recovered":5455872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4200}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6861","reported_seq":6944,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.037570+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:25.037570+0000","last_peered":"2024-09-16T04:45:25.037570+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:25.037570+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6861,"log_dups_size":0,"ondisk_log_size":6861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48305420,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6859,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3699}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6713","reported_seq":6788,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.408037+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:25.408037+0000","last_peered":"2024-09-16T04:45:25.408037+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:25.408037+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6713,"log_dups_size":0,"ondisk_log_size":6713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47329170,"num_objects":5778,"num_object_clones":0,"num_object_copies":23112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6710,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6604","reported_seq":6657,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.348211+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:25.348211+0000","last_peered":"2024-09-16T04:45:25.348211+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:25.348211+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6604,"log_dups_size":0,"ondisk_log_size":6604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46914182,"num_objects":5727,"num_object_clones":0,"num_object_copies":22908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6603,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3605}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6729","reported_seq":6804,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.048090+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:25.048090+0000","last_peered":"2024-09-16T04:45:25.048090+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:25.048090+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6729,"log_dups_size":0,"ondisk_log_size":6729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47544966,"num_objects":5804,"num_object_clones":0,"num_object_copies":23216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6728,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3579}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6855","reported_seq":6937,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.381018+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:25.381018+0000","last_peered":"2024-09-16T04:45:25.381018+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:25.381018+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48551180,"num_objects":5927,"num_object_clones":0,"num_object_copies":23708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3707}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6677","reported_seq":6761,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:24.832300+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:24.832300+0000","last_peered":"2024-09-16T04:45:24.832300+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:24.832300+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6677,"log_dups_size":0,"ondisk_log_size":6677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47281420,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6675,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3642}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6644","reported_seq":6711,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.329194+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:25.329194+0000","last_peered":"2024-09-16T04:45:25.329194+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:25.329194+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6644,"log_dups_size":0,"ondisk_log_size":6644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46388492,"num_objects":5663,"num_object_clones":0,"num_object_copies":22652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6642,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":762889920,"num_objects":93132,"num_object_clones":0,"num_object_copies":372528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108355,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2997,"num_bytes_recovered":23789568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1418842112,"data_stored":1418842112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108382,"ondisk_log_size":108382,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747668,"kb_used_data":373052,"kb_used_omap":4,"kb_used_meta":374587,"kb_avail":93624172,"statfs":{"total":96636764160,"available":95871152128,"internally_reserved":0,"allocated":382005248,"data_stored":381825279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":383577807},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,3,5,1,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743324,"kb_used_data":373388,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93628516,"statfs":{"total":96636764160,"available":95875600384,"internally_reserved":0,"allocated":382349312,"data_stored":382169343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,0,0,2,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539404,"kb_used_data":268924,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93832436,"statfs":{"total":96636764160,"available":96084414464,"internally_reserved":0,"allocated":275378176,"data_stored":275198207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276950735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752036,"kb_used_data":373332,"kb_used_omap":4,"kb_used_meta":378683,"kb_avail":93619804,"statfs":{"total":96636764160,"available":95866679296,"internally_reserved":0,"allocated":382291968,"data_stored":382111999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":387772110},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2,3,0,2,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":381165568,"data_stored":381165568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275046400,"data_stored":275046400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381513728,"data_stored":381513728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":381116416,"data_stored":381116416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:28.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:45:28.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6795", 'reported_seq': 6854, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:25.407696+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:45:25.407696+0000', 'last_peered': '2024-09-16T04:45:25.407696+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:45:25.407696+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6795, 'log_dups_size': 0, 'ondisk_log_size': 6795, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47996928, 'num_objects': 5859, 'num_object_clones': 0, 'num_object_copies': 23436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6795, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3647}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:28.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:45:28.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6833", 'reported_seq': 6904, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:25.361778+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:45:25.361778+0000', 'last_peered': '2024-09-16T04:45:25.361778+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:45:25.361778+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6833, 'log_dups_size': 0, 'ondisk_log_size': 6833, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47846668, 'num_objects': 5841, 'num_object_clones': 0, 'num_object_copies': 23364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2211, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5841, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6831, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3632}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:28.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:45:28.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6815", 'reported_seq': 6894, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:24.981202+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:45:24.981202+0000', 'last_peered': '2024-09-16T04:45:24.981202+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:45:24.981202+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6815, 'log_dups_size': 0, 'ondisk_log_size': 6815, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47962782, 'num_objects': 5856, 'num_object_clones': 0, 'num_object_copies': 23424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5856, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6814, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3630}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:28.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:45:28.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6828", 'reported_seq': 6899, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:25.373638+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:45:25.373638+0000', 'last_peered': '2024-09-16T04:45:25.373638+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:45:25.373638+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6828, 'log_dups_size': 0, 'ondisk_log_size': 6828, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47854860, 'num_objects': 5842, 'num_object_clones': 0, 'num_object_copies': 23368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6826, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3615}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:28.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:45:28.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6870", 'reported_seq': 6946, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:25.476802+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:45:25.476802+0000', 'last_peered': '2024-09-16T04:45:25.476802+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:45:25.476802+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6870, 'log_dups_size': 0, 'ondisk_log_size': 6870, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48092428, 'num_objects': 5871, 'num_object_clones': 0, 'num_object_copies': 23484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2274, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6868, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:28.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:45:28.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6642", 'reported_seq': 6703, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:25.401282+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:45:25.401282+0000', 'last_peered': '2024-09-16T04:45:25.401282+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:45:25.401282+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6642, 'log_dups_size': 0, 'ondisk_log_size': 6642, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46448640, 'num_objects': 5670, 'num_object_clones': 0, 'num_object_copies': 22680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5670, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6642, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3470}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:28.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:45:28.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6790", 'reported_seq': 6858, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:25.328648+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:45:25.328648+0000', 'last_peered': '2024-09-16T04:45:25.328648+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:45:25.328648+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6790, 'log_dups_size': 0, 'ondisk_log_size': 6790, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47592716, 'num_objects': 5810, 'num_object_clones': 0, 'num_object_copies': 23240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6788, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3585}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:28.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:45:28.013 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6882", 'reported_seq': 8273, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:25.474135+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:45:25.474135+0000', 'last_peered': '2024-09-16T04:45:25.474135+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:45:25.474135+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6882, 'log_dups_size': 0, 'ondisk_log_size': 6882, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48114200, 'num_objects': 5874, 'num_object_clones': 0, 'num_object_copies': 23496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1674, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5874, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6879, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 669, 'num_bytes_recovered': 5455872, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4200}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:28.013 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:45:28.014 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6861", 'reported_seq': 6944, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:25.037570+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:45:25.037570+0000', 'last_peered': '2024-09-16T04:45:25.037570+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:45:25.037570+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6861, 'log_dups_size': 0, 'ondisk_log_size': 6861, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48305420, 'num_objects': 5897, 'num_object_clones': 0, 'num_object_copies': 23588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5897, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6859, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3699}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:28.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:45:28.014 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6713", 'reported_seq': 6788, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:25.408037+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:45:25.408037+0000', 'last_peered': '2024-09-16T04:45:25.408037+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:45:25.408037+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6713, 'log_dups_size': 0, 'ondisk_log_size': 6713, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47329170, 'num_objects': 5778, 'num_object_clones': 0, 'num_object_copies': 23112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2184, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5778, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6710, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3597}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:28.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:45:28.014 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6604", 'reported_seq': 6657, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:25.348211+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:45:25.348211+0000', 'last_peered': '2024-09-16T04:45:25.348211+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:45:25.348211+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6604, 'log_dups_size': 0, 'ondisk_log_size': 6604, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46914182, 'num_objects': 5727, 'num_object_clones': 0, 'num_object_copies': 22908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2123, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5727, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6603, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3605}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:28.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:45:28.014 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6729", 'reported_seq': 6804, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:25.048090+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:45:25.048090+0000', 'last_peered': '2024-09-16T04:45:25.048090+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:45:25.048090+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6729, 'log_dups_size': 0, 'ondisk_log_size': 6729, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47544966, 'num_objects': 5804, 'num_object_clones': 0, 'num_object_copies': 23216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6728, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3579}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:28.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:45:28.014 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6855", 'reported_seq': 6937, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:25.381018+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:45:25.381018+0000', 'last_peered': '2024-09-16T04:45:25.381018+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:45:25.381018+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6855, 'log_dups_size': 0, 'ondisk_log_size': 6855, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48551180, 'num_objects': 5927, 'num_object_clones': 0, 'num_object_copies': 23708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5927, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6853, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3707}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:28.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:45:28.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6677", 'reported_seq': 6761, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:24.832300+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:45:24.832300+0000', 'last_peered': '2024-09-16T04:45:24.832300+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:45:24.832300+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6677, 'log_dups_size': 0, 'ondisk_log_size': 6677, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47281420, 'num_objects': 5772, 'num_object_clones': 0, 'num_object_copies': 23088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6675, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3642}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:28.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:45:28.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6644", 'reported_seq': 6711, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:25.329194+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:45:25.329194+0000', 'last_peered': '2024-09-16T04:45:25.329194+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:45:25.329194+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6644, 'log_dups_size': 0, 'ondisk_log_size': 6644, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46388492, 'num_objects': 5663, 'num_object_clones': 0, 'num_object_copies': 22652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6642, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3424}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:28.015 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:45:28.714 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:28.714 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32582/372528 objects degraded (8.746%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":341,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":93132,"data_bytes":762889920,"bytes_used":2849210368,"bytes_avail":383697846272,"bytes_total":386547056640,"degraded_objects":32582,"degraded_total":372528,"degraded_ratio":0.087461882059872001,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":119437,"recovering_keys_per_sec":0,"num_objects_recovered":214,"num_bytes_recovered":1433600,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":290},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:45:28.741 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:45:28.741 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:29.339 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:29.339 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:29.362 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-16T04:45:28.641293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":762267328,"num_objects":93056,"num_object_clones":0,"num_object_copies":372224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93056,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108431,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2997,"num_bytes_recovered":23789568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108458,"ondisk_log_size":108458,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2786220,"kb_used_data":1388324,"kb_used_omap":14,"kb_used_meta":1397809,"kb_avail":374701140,"statfs":{"total":386547056640,"available":383693967360,"internally_reserved":0,"allocated":1421643776,"data_stored":1420923900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1431356977},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,8,9,13,2,4,0,0,0,0,3,5,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19734528,"num_objects":-2409,"num_object_clones":0,"num_object_copies":-9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2409,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003028"},"pg_stats":[{"pgid":"1.f","version":"33'6795","reported_seq":6854,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.407696+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:25.407696+0000","last_peered":"2024-09-16T04:45:25.407696+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:25.407696+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6795,"log_dups_size":0,"ondisk_log_size":6795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47996928,"num_objects":5859,"num_object_clones":0,"num_object_copies":23436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6795,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6833","reported_seq":6904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.361778+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:25.361778+0000","last_peered":"2024-09-16T04:45:25.361778+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:25.361778+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6833,"log_dups_size":0,"ondisk_log_size":6833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47846668,"num_objects":5841,"num_object_clones":0,"num_object_copies":23364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6831,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6815","reported_seq":6894,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:24.981202+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:24.981202+0000","last_peered":"2024-09-16T04:45:24.981202+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:24.981202+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6815,"log_dups_size":0,"ondisk_log_size":6815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47962782,"num_objects":5856,"num_object_clones":0,"num_object_copies":23424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6814,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3630}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6828","reported_seq":6899,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.373638+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:25.373638+0000","last_peered":"2024-09-16T04:45:25.373638+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:25.373638+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6828,"log_dups_size":0,"ondisk_log_size":6828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47854860,"num_objects":5842,"num_object_clones":0,"num_object_copies":23368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6826,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6870","reported_seq":6946,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.476802+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:25.476802+0000","last_peered":"2024-09-16T04:45:25.476802+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:25.476802+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6870,"log_dups_size":0,"ondisk_log_size":6870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48092428,"num_objects":5871,"num_object_clones":0,"num_object_copies":23484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6868,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6642","reported_seq":6703,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.401282+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:25.401282+0000","last_peered":"2024-09-16T04:45:25.401282+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:25.401282+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6642,"log_dups_size":0,"ondisk_log_size":6642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46448640,"num_objects":5670,"num_object_clones":0,"num_object_copies":22680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6642,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3470}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6790","reported_seq":6858,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.328648+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:25.328648+0000","last_peered":"2024-09-16T04:45:25.328648+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:25.328648+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6790,"log_dups_size":0,"ondisk_log_size":6790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47592716,"num_objects":5810,"num_object_clones":0,"num_object_copies":23240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3585}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6920","reported_seq":11476,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:27.271502+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:27.271502+0000","last_peered":"2024-09-16T04:45:27.271502+0000","last_clean":"2024-09-16T04:45:27.271502+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:27.271502+0000","last_undegraded":"2024-09-16T04:45:27.271502+0000","last_fullsized":"2024-09-16T04:45:27.271502+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6920,"log_dups_size":0,"ondisk_log_size":6920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:29.141999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48043276,"num_objects":5865,"num_object_clones":0,"num_object_copies":23460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6918,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6882","reported_seq":8273,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.474135+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:25.474135+0000","last_peered":"2024-09-16T04:45:25.474135+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:25.474135+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6882,"log_dups_size":0,"ondisk_log_size":6882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48114200,"num_objects":5874,"num_object_clones":0,"num_object_copies":23496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6879,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":669,"num_bytes_recovered":5455872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4200}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6861","reported_seq":6944,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.037570+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:25.037570+0000","last_peered":"2024-09-16T04:45:25.037570+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:25.037570+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6861,"log_dups_size":0,"ondisk_log_size":6861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48305420,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6859,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3699}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6713","reported_seq":6788,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.408037+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:25.408037+0000","last_peered":"2024-09-16T04:45:25.408037+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:25.408037+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6713,"log_dups_size":0,"ondisk_log_size":6713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47329170,"num_objects":5778,"num_object_clones":0,"num_object_copies":23112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6710,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6604","reported_seq":6657,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.348211+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:25.348211+0000","last_peered":"2024-09-16T04:45:25.348211+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:25.348211+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6604,"log_dups_size":0,"ondisk_log_size":6604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46914182,"num_objects":5727,"num_object_clones":0,"num_object_copies":22908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6603,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3605}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6729","reported_seq":6804,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.048090+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:25.048090+0000","last_peered":"2024-09-16T04:45:25.048090+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:25.048090+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6729,"log_dups_size":0,"ondisk_log_size":6729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47544966,"num_objects":5804,"num_object_clones":0,"num_object_copies":23216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6728,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3579}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6855","reported_seq":6937,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.381018+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:25.381018+0000","last_peered":"2024-09-16T04:45:25.381018+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:25.381018+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48551180,"num_objects":5927,"num_object_clones":0,"num_object_copies":23708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3707}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6677","reported_seq":6761,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:24.832300+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:24.832300+0000","last_peered":"2024-09-16T04:45:24.832300+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:24.832300+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6677,"log_dups_size":0,"ondisk_log_size":6677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47281420,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6675,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3642}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6644","reported_seq":6711,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:25.329194+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:25.329194+0000","last_peered":"2024-09-16T04:45:25.329194+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:25.329194+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6644,"log_dups_size":0,"ondisk_log_size":6644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46388492,"num_objects":5663,"num_object_clones":0,"num_object_copies":22652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6642,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":762267328,"num_objects":93056,"num_object_clones":0,"num_object_copies":372224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93056,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108431,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2997,"num_bytes_recovered":23789568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1418498048,"data_stored":1418498048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108458,"ondisk_log_size":108458,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747668,"kb_used_data":373052,"kb_used_omap":4,"kb_used_meta":374587,"kb_avail":93624172,"statfs":{"total":96636764160,"available":95871152128,"internally_reserved":0,"allocated":382005248,"data_stored":381825279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":383577807},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,3,5,1,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743324,"kb_used_data":373388,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93628516,"statfs":{"total":96636764160,"available":95875600384,"internally_reserved":0,"allocated":382349312,"data_stored":382169343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,0,0,2,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051633,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543192,"kb_used_data":268552,"kb_used_omap":4,"kb_used_meta":274619,"kb_avail":93828648,"statfs":{"total":96636764160,"available":96080535552,"internally_reserved":0,"allocated":274997248,"data_stored":274817279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281210575},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752036,"kb_used_data":373332,"kb_used_omap":4,"kb_used_meta":378683,"kb_avail":93619804,"statfs":{"total":96636764160,"available":95866679296,"internally_reserved":0,"allocated":382291968,"data_stored":382111999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":387772110},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2,3,0,2,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":381165568,"data_stored":381165568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274702336,"data_stored":274702336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381513728,"data_stored":381513728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":381116416,"data_stored":381116416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:32.365 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:32.990 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:32.991 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:33.015 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-16T04:45:32.642331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":752387776,"num_objects":91850,"num_object_clones":0,"num_object_copies":367400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109637,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3051,"num_bytes_recovered":24207360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109664,"ondisk_log_size":109664,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2825640,"kb_used_data":1373608,"kb_used_omap":14,"kb_used_meta":1451953,"kb_avail":374661720,"statfs":{"total":386547056640,"available":383653601280,"internally_reserved":0,"allocated":1406574592,"data_stored":1405854716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1486800433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,8,12,5,7,5,0,0,0,0,2,0,3,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20103168,"num_objects":-2454,"num_object_clones":0,"num_object_copies":-9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003072"},"pg_stats":[{"pgid":"1.f","version":"33'6878","reported_seq":6939,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.416599+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:30.416599+0000","last_peered":"2024-09-16T04:45:30.416599+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:30.416599+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6878,"log_dups_size":0,"ondisk_log_size":6878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47316992,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6878,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3565}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6908","reported_seq":6978,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.416490+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:30.416490+0000","last_peered":"2024-09-16T04:45:30.416490+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:30.416490+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6908,"log_dups_size":0,"ondisk_log_size":6908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47232268,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6906,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6889","reported_seq":6967,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.084123+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:30.084123+0000","last_peered":"2024-09-16T04:45:30.084123+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:30.084123+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6889,"log_dups_size":0,"ondisk_log_size":6889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47356574,"num_objects":5782,"num_object_clones":0,"num_object_copies":23128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6888,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3556}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6899","reported_seq":6970,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.482807+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:30.482807+0000","last_peered":"2024-09-16T04:45:30.482807+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:30.482807+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47273228,"num_objects":5771,"num_object_clones":0,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6897,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6955","reported_seq":7034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.481569+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:30.481569+0000","last_peered":"2024-09-16T04:45:30.481569+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:30.481569+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6955,"log_dups_size":0,"ondisk_log_size":6955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47396108,"num_objects":5786,"num_object_clones":0,"num_object_copies":23144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6953,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6714","reported_seq":6777,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.420903+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:30.420903+0000","last_peered":"2024-09-16T04:45:30.420903+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:30.420903+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6714,"log_dups_size":0,"ondisk_log_size":6714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45858816,"num_objects":5598,"num_object_clones":0,"num_object_copies":22392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6714,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3399}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6871","reported_seq":6939,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.307104+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:30.307104+0000","last_peered":"2024-09-16T04:45:30.307104+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:30.307104+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6871,"log_dups_size":0,"ondisk_log_size":6871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46929164,"num_objects":5729,"num_object_clones":0,"num_object_copies":22916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6869,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3504}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7002","reported_seq":11558,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:32.468420+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:32.468420+0000","last_peered":"2024-09-16T04:45:32.468420+0000","last_clean":"2024-09-16T04:45:32.468420+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:32.468420+0000","last_undegraded":"2024-09-16T04:45:32.468420+0000","last_fullsized":"2024-09-16T04:45:32.468420+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7002,"log_dups_size":0,"ondisk_log_size":7002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:34.224583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47371532,"num_objects":5783,"num_object_clones":0,"num_object_copies":23132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7000,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6966","reported_seq":8459,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.446853+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:30.446853+0000","last_peered":"2024-09-16T04:45:30.446853+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:30.446853+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6966,"log_dups_size":0,"ondisk_log_size":6966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47426072,"num_objects":5790,"num_object_clones":0,"num_object_copies":23160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6963,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":720,"num_bytes_recovered":5873664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6932","reported_seq":7015,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.063344+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:30.063344+0000","last_peered":"2024-09-16T04:45:30.063344+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:30.063344+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6932,"log_dups_size":0,"ondisk_log_size":6932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47723788,"num_objects":5826,"num_object_clones":0,"num_object_copies":23304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6930,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3628}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6780","reported_seq":6857,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.406358+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:30.406358+0000","last_peered":"2024-09-16T04:45:30.406358+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:30.406358+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6780,"log_dups_size":0,"ondisk_log_size":6780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46780306,"num_objects":5711,"num_object_clones":0,"num_object_copies":22844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6777,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3531}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6676","reported_seq":6729,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.452243+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:30.452243+0000","last_peered":"2024-09-16T04:45:30.452243+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:30.452243+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6676,"log_dups_size":0,"ondisk_log_size":6676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46324358,"num_objects":5655,"num_object_clones":0,"num_object_copies":22620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6675,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3533}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6800","reported_seq":6875,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.023482+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:30.023482+0000","last_peered":"2024-09-16T04:45:30.023482+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:30.023482+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6800,"log_dups_size":0,"ondisk_log_size":6800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46963334,"num_objects":5733,"num_object_clones":0,"num_object_copies":22932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6799,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3508}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6910","reported_seq":6993,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.395198+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:30.395198+0000","last_peered":"2024-09-16T04:45:30.395198+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:30.395198+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6910,"log_dups_size":0,"ondisk_log_size":6910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48100620,"num_objects":5872,"num_object_clones":0,"num_object_copies":23488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6908,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3652}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6760","reported_seq":6843,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.031124+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:30.031124+0000","last_peered":"2024-09-16T04:45:30.031124+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:30.031124+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6760,"log_dups_size":0,"ondisk_log_size":6760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46601484,"num_objects":5689,"num_object_clones":0,"num_object_copies":22756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6758,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6724","reported_seq":6791,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.330701+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:30.330701+0000","last_peered":"2024-09-16T04:45:30.330701+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:30.330701+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6724,"log_dups_size":0,"ondisk_log_size":6724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45733132,"num_objects":5583,"num_object_clones":0,"num_object_copies":22332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6722,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":752387776,"num_objects":91850,"num_object_clones":0,"num_object_copies":367400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109637,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3051,"num_bytes_recovered":24207360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1403228160,"data_stored":1403228160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109664,"ondisk_log_size":109664,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765104,"kb_used_data":368288,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93606736,"statfs":{"total":96636764160,"available":95853297664,"internally_reserved":0,"allocated":377126912,"data_stored":376946943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406318799},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,0,6,3,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760768,"kb_used_data":368624,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93611072,"statfs":{"total":96636764160,"available":95857737728,"internally_reserved":0,"allocated":377470976,"data_stored":377291007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,0,1,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051634,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530304,"kb_used_data":268144,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93841536,"statfs":{"total":96636764160,"available":96093732864,"internally_reserved":0,"allocated":274579456,"data_stored":274399487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268431055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769464,"kb_used_data":368552,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93602376,"statfs":{"total":96636764160,"available":95848833024,"internally_reserved":0,"allocated":377397248,"data_stored":377217279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":410513102},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,2,2,1,1,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":376188928,"data_stored":376188928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274255872,"data_stored":274255872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376594432,"data_stored":376594432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":376188928,"data_stored":376188928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:33.016 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:33.594 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:33.595 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:33.619 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-16T04:45:32.642331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":752387776,"num_objects":91850,"num_object_clones":0,"num_object_copies":367400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109637,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3051,"num_bytes_recovered":24207360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109664,"ondisk_log_size":109664,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2825640,"kb_used_data":1373608,"kb_used_omap":14,"kb_used_meta":1451953,"kb_avail":374661720,"statfs":{"total":386547056640,"available":383653601280,"internally_reserved":0,"allocated":1406574592,"data_stored":1405854716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1486800433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,8,12,5,7,5,0,0,0,0,2,0,3,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20103168,"num_objects":-2454,"num_object_clones":0,"num_object_copies":-9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003072"},"pg_stats":[{"pgid":"1.f","version":"33'6878","reported_seq":6939,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.416599+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:30.416599+0000","last_peered":"2024-09-16T04:45:30.416599+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:30.416599+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6878,"log_dups_size":0,"ondisk_log_size":6878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47316992,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6878,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3565}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6908","reported_seq":6978,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.416490+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:30.416490+0000","last_peered":"2024-09-16T04:45:30.416490+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:30.416490+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6908,"log_dups_size":0,"ondisk_log_size":6908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47232268,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6906,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6889","reported_seq":6967,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.084123+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:30.084123+0000","last_peered":"2024-09-16T04:45:30.084123+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:30.084123+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6889,"log_dups_size":0,"ondisk_log_size":6889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47356574,"num_objects":5782,"num_object_clones":0,"num_object_copies":23128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6888,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3556}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6899","reported_seq":6970,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.482807+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:30.482807+0000","last_peered":"2024-09-16T04:45:30.482807+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:30.482807+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47273228,"num_objects":5771,"num_object_clones":0,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6897,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6955","reported_seq":7034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.481569+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:30.481569+0000","last_peered":"2024-09-16T04:45:30.481569+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:30.481569+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6955,"log_dups_size":0,"ondisk_log_size":6955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47396108,"num_objects":5786,"num_object_clones":0,"num_object_copies":23144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6953,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6714","reported_seq":6777,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.420903+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:30.420903+0000","last_peered":"2024-09-16T04:45:30.420903+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:30.420903+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6714,"log_dups_size":0,"ondisk_log_size":6714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45858816,"num_objects":5598,"num_object_clones":0,"num_object_copies":22392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6714,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3399}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6871","reported_seq":6939,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.307104+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:30.307104+0000","last_peered":"2024-09-16T04:45:30.307104+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:30.307104+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6871,"log_dups_size":0,"ondisk_log_size":6871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46929164,"num_objects":5729,"num_object_clones":0,"num_object_copies":22916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6869,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3504}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7002","reported_seq":11558,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:32.468420+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:32.468420+0000","last_peered":"2024-09-16T04:45:32.468420+0000","last_clean":"2024-09-16T04:45:32.468420+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:32.468420+0000","last_undegraded":"2024-09-16T04:45:32.468420+0000","last_fullsized":"2024-09-16T04:45:32.468420+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7002,"log_dups_size":0,"ondisk_log_size":7002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:34.224583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47371532,"num_objects":5783,"num_object_clones":0,"num_object_copies":23132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7000,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6966","reported_seq":8459,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.446853+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:30.446853+0000","last_peered":"2024-09-16T04:45:30.446853+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:30.446853+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6966,"log_dups_size":0,"ondisk_log_size":6966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47426072,"num_objects":5790,"num_object_clones":0,"num_object_copies":23160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6963,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":720,"num_bytes_recovered":5873664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6932","reported_seq":7015,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.063344+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:30.063344+0000","last_peered":"2024-09-16T04:45:30.063344+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:30.063344+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6932,"log_dups_size":0,"ondisk_log_size":6932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47723788,"num_objects":5826,"num_object_clones":0,"num_object_copies":23304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6930,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3628}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6780","reported_seq":6857,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.406358+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:30.406358+0000","last_peered":"2024-09-16T04:45:30.406358+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:30.406358+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6780,"log_dups_size":0,"ondisk_log_size":6780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46780306,"num_objects":5711,"num_object_clones":0,"num_object_copies":22844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6777,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3531}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6676","reported_seq":6729,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.452243+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:30.452243+0000","last_peered":"2024-09-16T04:45:30.452243+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:30.452243+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6676,"log_dups_size":0,"ondisk_log_size":6676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46324358,"num_objects":5655,"num_object_clones":0,"num_object_copies":22620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6675,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3533}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6800","reported_seq":6875,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.023482+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:30.023482+0000","last_peered":"2024-09-16T04:45:30.023482+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:30.023482+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6800,"log_dups_size":0,"ondisk_log_size":6800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46963334,"num_objects":5733,"num_object_clones":0,"num_object_copies":22932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6799,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3508}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6910","reported_seq":6993,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.395198+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:30.395198+0000","last_peered":"2024-09-16T04:45:30.395198+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:30.395198+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6910,"log_dups_size":0,"ondisk_log_size":6910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48100620,"num_objects":5872,"num_object_clones":0,"num_object_copies":23488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6908,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3652}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6760","reported_seq":6843,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.031124+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:30.031124+0000","last_peered":"2024-09-16T04:45:30.031124+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:30.031124+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6760,"log_dups_size":0,"ondisk_log_size":6760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46601484,"num_objects":5689,"num_object_clones":0,"num_object_copies":22756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6758,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6724","reported_seq":6791,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.330701+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:30.330701+0000","last_peered":"2024-09-16T04:45:30.330701+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:30.330701+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6724,"log_dups_size":0,"ondisk_log_size":6724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45733132,"num_objects":5583,"num_object_clones":0,"num_object_copies":22332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6722,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":752387776,"num_objects":91850,"num_object_clones":0,"num_object_copies":367400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109637,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3051,"num_bytes_recovered":24207360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1403228160,"data_stored":1403228160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109664,"ondisk_log_size":109664,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765104,"kb_used_data":368288,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93606736,"statfs":{"total":96636764160,"available":95853297664,"internally_reserved":0,"allocated":377126912,"data_stored":376946943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406318799},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,0,6,3,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760768,"kb_used_data":368624,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93611072,"statfs":{"total":96636764160,"available":95857737728,"internally_reserved":0,"allocated":377470976,"data_stored":377291007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,0,1,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051634,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530304,"kb_used_data":268144,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93841536,"statfs":{"total":96636764160,"available":96093732864,"internally_reserved":0,"allocated":274579456,"data_stored":274399487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268431055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769464,"kb_used_data":368552,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93602376,"statfs":{"total":96636764160,"available":95848833024,"internally_reserved":0,"allocated":377397248,"data_stored":377217279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":410513102},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,2,2,1,1,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":376188928,"data_stored":376188928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274255872,"data_stored":274255872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376594432,"data_stored":376594432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":376188928,"data_stored":376188928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:33.620 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:45:33.621 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6878", 'reported_seq': 6939, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:30.416599+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:45:30.416599+0000', 'last_peered': '2024-09-16T04:45:30.416599+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:45:30.416599+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6878, 'log_dups_size': 0, 'ondisk_log_size': 6878, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47316992, 'num_objects': 5776, 'num_object_clones': 0, 'num_object_copies': 23104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2211, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6878, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3565}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:33.621 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:45:33.621 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6908", 'reported_seq': 6978, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:30.416490+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:45:30.416490+0000', 'last_peered': '2024-09-16T04:45:30.416490+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:45:30.416490+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6908, 'log_dups_size': 0, 'ondisk_log_size': 6908, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47232268, 'num_objects': 5766, 'num_object_clones': 0, 'num_object_copies': 23064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2211, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6906, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:33.621 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:45:33.621 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6889", 'reported_seq': 6967, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:30.084123+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:45:30.084123+0000', 'last_peered': '2024-09-16T04:45:30.084123+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:45:30.084123+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6889, 'log_dups_size': 0, 'ondisk_log_size': 6889, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47356574, 'num_objects': 5782, 'num_object_clones': 0, 'num_object_copies': 23128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5782, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6888, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3556}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:33.621 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:45:33.621 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6899", 'reported_seq': 6970, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:30.482807+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:45:30.482807+0000', 'last_peered': '2024-09-16T04:45:30.482807+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:45:30.482807+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6899, 'log_dups_size': 0, 'ondisk_log_size': 6899, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47273228, 'num_objects': 5771, 'num_object_clones': 0, 'num_object_copies': 23084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5771, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6897, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3544}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:33.621 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:45:33.622 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6955", 'reported_seq': 7034, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:30.481569+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:45:30.481569+0000', 'last_peered': '2024-09-16T04:45:30.481569+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:45:30.481569+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6955, 'log_dups_size': 0, 'ondisk_log_size': 6955, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47396108, 'num_objects': 5786, 'num_object_clones': 0, 'num_object_copies': 23144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2274, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5786, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6953, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3514}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:33.622 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:45:33.622 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6714", 'reported_seq': 6777, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:30.420903+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:45:30.420903+0000', 'last_peered': '2024-09-16T04:45:30.420903+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:45:30.420903+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6714, 'log_dups_size': 0, 'ondisk_log_size': 6714, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45858816, 'num_objects': 5598, 'num_object_clones': 0, 'num_object_copies': 22392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2199, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6714, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3399}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:33.622 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:45:33.622 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6871", 'reported_seq': 6939, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:30.307104+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:45:30.307104+0000', 'last_peered': '2024-09-16T04:45:30.307104+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:45:30.307104+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6871, 'log_dups_size': 0, 'ondisk_log_size': 6871, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46929164, 'num_objects': 5729, 'num_object_clones': 0, 'num_object_copies': 22916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5729, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6869, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3504}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:33.622 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:45:33.622 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6966", 'reported_seq': 8459, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:30.446853+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:45:30.446853+0000', 'last_peered': '2024-09-16T04:45:30.446853+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:45:30.446853+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6966, 'log_dups_size': 0, 'ondisk_log_size': 6966, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47426072, 'num_objects': 5790, 'num_object_clones': 0, 'num_object_copies': 23160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1623, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5790, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6963, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 720, 'num_bytes_recovered': 5873664, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4167}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:33.622 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:45:33.622 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6932", 'reported_seq': 7015, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:30.063344+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:45:30.063344+0000', 'last_peered': '2024-09-16T04:45:30.063344+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:45:30.063344+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6932, 'log_dups_size': 0, 'ondisk_log_size': 6932, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47723788, 'num_objects': 5826, 'num_object_clones': 0, 'num_object_copies': 23304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6930, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3628}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:33.623 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:45:33.623 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6780", 'reported_seq': 6857, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:30.406358+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:45:30.406358+0000', 'last_peered': '2024-09-16T04:45:30.406358+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:45:30.406358+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6780, 'log_dups_size': 0, 'ondisk_log_size': 6780, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46780306, 'num_objects': 5711, 'num_object_clones': 0, 'num_object_copies': 22844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2183, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5711, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6777, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3531}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:33.623 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:45:33.623 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6676", 'reported_seq': 6729, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:30.452243+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:45:30.452243+0000', 'last_peered': '2024-09-16T04:45:30.452243+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:45:30.452243+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6676, 'log_dups_size': 0, 'ondisk_log_size': 6676, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46324358, 'num_objects': 5655, 'num_object_clones': 0, 'num_object_copies': 22620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2123, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5655, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6675, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3533}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:33.623 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:45:33.623 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6800", 'reported_seq': 6875, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:30.023482+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:45:30.023482+0000', 'last_peered': '2024-09-16T04:45:30.023482+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:45:30.023482+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6800, 'log_dups_size': 0, 'ondisk_log_size': 6800, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46963334, 'num_objects': 5733, 'num_object_clones': 0, 'num_object_copies': 22932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6799, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3508}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:33.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:45:33.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6910", 'reported_seq': 6993, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:30.395198+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:45:30.395198+0000', 'last_peered': '2024-09-16T04:45:30.395198+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:45:30.395198+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6910, 'log_dups_size': 0, 'ondisk_log_size': 6910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48100620, 'num_objects': 5872, 'num_object_clones': 0, 'num_object_copies': 23488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5872, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6908, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3652}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:33.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:45:33.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6760", 'reported_seq': 6843, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:30.031124+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:45:30.031124+0000', 'last_peered': '2024-09-16T04:45:30.031124+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:45:30.031124+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6760, 'log_dups_size': 0, 'ondisk_log_size': 6760, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46601484, 'num_objects': 5689, 'num_object_clones': 0, 'num_object_copies': 22756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5689, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6758, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:33.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:45:33.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6724", 'reported_seq': 6791, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:30.330701+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:45:30.330701+0000', 'last_peered': '2024-09-16T04:45:30.330701+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:45:30.330701+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6724, 'log_dups_size': 0, 'ondisk_log_size': 6724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45733132, 'num_objects': 5583, 'num_object_clones': 0, 'num_object_copies': 22332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6722, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3344}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:33.624 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:45:34.355 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:34.355 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32528/367400 objects degraded (8.854%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":346,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":91850,"data_bytes":752387776,"bytes_used":2893455360,"bytes_avail":383653601280,"bytes_total":386547056640,"degraded_objects":32528,"degraded_total":367400,"degraded_ratio":0.088535655960805662,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":79851,"recovering_keys_per_sec":0,"num_objects_recovered":132,"num_bytes_recovered":958464,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":204},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:45:34.380 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:45:34.381 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:34.988 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:34.988 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:35.014 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-16T04:45:34.642778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":752387776,"num_objects":91850,"num_object_clones":0,"num_object_copies":367400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109637,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3051,"num_bytes_recovered":24207360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109664,"ondisk_log_size":109664,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2825640,"kb_used_data":1373608,"kb_used_omap":14,"kb_used_meta":1451953,"kb_avail":374661720,"statfs":{"total":386547056640,"available":383653601280,"internally_reserved":0,"allocated":1406574592,"data_stored":1405854716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1486800433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,8,12,5,7,5,0,0,0,0,2,0,3,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19562496,"num_objects":-2388,"num_object_clones":0,"num_object_copies":-9552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2388,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002962"},"pg_stats":[{"pgid":"1.f","version":"33'6878","reported_seq":6939,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.416599+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:30.416599+0000","last_peered":"2024-09-16T04:45:30.416599+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:30.416599+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6878,"log_dups_size":0,"ondisk_log_size":6878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47316992,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6878,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3565}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6908","reported_seq":6978,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.416490+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:30.416490+0000","last_peered":"2024-09-16T04:45:30.416490+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:30.416490+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6908,"log_dups_size":0,"ondisk_log_size":6908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47232268,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6906,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6889","reported_seq":6967,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.084123+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:30.084123+0000","last_peered":"2024-09-16T04:45:30.084123+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:30.084123+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6889,"log_dups_size":0,"ondisk_log_size":6889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47356574,"num_objects":5782,"num_object_clones":0,"num_object_copies":23128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6888,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3556}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6899","reported_seq":6970,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.482807+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:30.482807+0000","last_peered":"2024-09-16T04:45:30.482807+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:30.482807+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47273228,"num_objects":5771,"num_object_clones":0,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6897,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6955","reported_seq":7034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.481569+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:30.481569+0000","last_peered":"2024-09-16T04:45:30.481569+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:30.481569+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6955,"log_dups_size":0,"ondisk_log_size":6955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47396108,"num_objects":5786,"num_object_clones":0,"num_object_copies":23144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6953,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6714","reported_seq":6777,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.420903+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:30.420903+0000","last_peered":"2024-09-16T04:45:30.420903+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:30.420903+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6714,"log_dups_size":0,"ondisk_log_size":6714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45858816,"num_objects":5598,"num_object_clones":0,"num_object_copies":22392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6714,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3399}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6871","reported_seq":6939,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.307104+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:30.307104+0000","last_peered":"2024-09-16T04:45:30.307104+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:30.307104+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6871,"log_dups_size":0,"ondisk_log_size":6871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46929164,"num_objects":5729,"num_object_clones":0,"num_object_copies":22916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6869,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3504}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7002","reported_seq":11558,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:32.468420+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:32.468420+0000","last_peered":"2024-09-16T04:45:32.468420+0000","last_clean":"2024-09-16T04:45:32.468420+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:32.468420+0000","last_undegraded":"2024-09-16T04:45:32.468420+0000","last_fullsized":"2024-09-16T04:45:32.468420+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7002,"log_dups_size":0,"ondisk_log_size":7002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:34.224583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47371532,"num_objects":5783,"num_object_clones":0,"num_object_copies":23132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7000,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'6966","reported_seq":8459,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.446853+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:30.446853+0000","last_peered":"2024-09-16T04:45:30.446853+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:30.446853+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6966,"log_dups_size":0,"ondisk_log_size":6966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47426072,"num_objects":5790,"num_object_clones":0,"num_object_copies":23160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6963,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":720,"num_bytes_recovered":5873664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6932","reported_seq":7015,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.063344+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:30.063344+0000","last_peered":"2024-09-16T04:45:30.063344+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:30.063344+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6932,"log_dups_size":0,"ondisk_log_size":6932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47723788,"num_objects":5826,"num_object_clones":0,"num_object_copies":23304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6930,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3628}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6780","reported_seq":6857,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.406358+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:30.406358+0000","last_peered":"2024-09-16T04:45:30.406358+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:30.406358+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6780,"log_dups_size":0,"ondisk_log_size":6780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46780306,"num_objects":5711,"num_object_clones":0,"num_object_copies":22844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6777,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3531}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6676","reported_seq":6729,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.452243+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:30.452243+0000","last_peered":"2024-09-16T04:45:30.452243+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:30.452243+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6676,"log_dups_size":0,"ondisk_log_size":6676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46324358,"num_objects":5655,"num_object_clones":0,"num_object_copies":22620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6675,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3533}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6800","reported_seq":6875,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.023482+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:30.023482+0000","last_peered":"2024-09-16T04:45:30.023482+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:30.023482+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6800,"log_dups_size":0,"ondisk_log_size":6800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46963334,"num_objects":5733,"num_object_clones":0,"num_object_copies":22932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6799,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3508}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6910","reported_seq":6993,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.395198+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:30.395198+0000","last_peered":"2024-09-16T04:45:30.395198+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:30.395198+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6910,"log_dups_size":0,"ondisk_log_size":6910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48100620,"num_objects":5872,"num_object_clones":0,"num_object_copies":23488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6908,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3652}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6760","reported_seq":6843,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.031124+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:30.031124+0000","last_peered":"2024-09-16T04:45:30.031124+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:30.031124+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6760,"log_dups_size":0,"ondisk_log_size":6760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46601484,"num_objects":5689,"num_object_clones":0,"num_object_copies":22756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6758,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6724","reported_seq":6791,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:30.330701+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:30.330701+0000","last_peered":"2024-09-16T04:45:30.330701+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:30.330701+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6724,"log_dups_size":0,"ondisk_log_size":6724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45733132,"num_objects":5583,"num_object_clones":0,"num_object_copies":22332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6722,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":752387776,"num_objects":91850,"num_object_clones":0,"num_object_copies":367400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109637,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3051,"num_bytes_recovered":24207360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1403228160,"data_stored":1403228160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109664,"ondisk_log_size":109664,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765104,"kb_used_data":368288,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93606736,"statfs":{"total":96636764160,"available":95853297664,"internally_reserved":0,"allocated":377126912,"data_stored":376946943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406318799},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,0,6,3,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760768,"kb_used_data":368624,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93611072,"statfs":{"total":96636764160,"available":95857737728,"internally_reserved":0,"allocated":377470976,"data_stored":377291007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,0,1,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051634,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530304,"kb_used_data":268144,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93841536,"statfs":{"total":96636764160,"available":96093732864,"internally_reserved":0,"allocated":274579456,"data_stored":274399487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268431055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769464,"kb_used_data":368552,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93602376,"statfs":{"total":96636764160,"available":95848833024,"internally_reserved":0,"allocated":377397248,"data_stored":377217279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":410513102},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,2,2,1,1,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":376188928,"data_stored":376188928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274255872,"data_stored":274255872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376594432,"data_stored":376594432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":376188928,"data_stored":376188928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:38.016 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:38.646 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:38.646 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:38.672 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-16T04:45:36.643272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":743294656,"num_objects":90740,"num_object_clones":0,"num_object_copies":362960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90740,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110747,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3114,"num_bytes_recovered":24666112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110774,"ondisk_log_size":110774,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2810360,"kb_used_data":1358304,"kb_used_omap":14,"kb_used_meta":1451953,"kb_avail":374677000,"statfs":{"total":386547056640,"available":383669248000,"internally_reserved":0,"allocated":1390903296,"data_stored":1390183420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1486800433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,3,15,9,3,3,0,2,0,0,0,0,0,5],"upper_bound":65536},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28655616,"num_objects":-3498,"num_object_clones":0,"num_object_copies":-13992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3498,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002992"},"pg_stats":[{"pgid":"1.f","version":"33'6955","reported_seq":7018,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.574793+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:35.574793+0000","last_peered":"2024-09-16T04:45:35.574793+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:35.574793+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6955,"log_dups_size":0,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46686208,"num_objects":5699,"num_object_clones":0,"num_object_copies":22796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6980","reported_seq":7053,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.462838+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:35.462838+0000","last_peered":"2024-09-16T04:45:35.462838+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:35.462838+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46642444,"num_objects":5694,"num_object_clones":0,"num_object_copies":22776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6978,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6959","reported_seq":7038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.039283+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:35.039283+0000","last_peered":"2024-09-16T04:45:35.039283+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:35.039283+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6959,"log_dups_size":0,"ondisk_log_size":6959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46783134,"num_objects":5712,"num_object_clones":0,"num_object_copies":22848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5712,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6958,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3486}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6990","reported_seq":7061,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.430403+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:35.430403+0000","last_peered":"2024-09-16T04:45:35.430403+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:35.430403+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6990,"log_dups_size":0,"ondisk_log_size":6990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46527756,"num_objects":5680,"num_object_clones":0,"num_object_copies":22720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6988,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7024","reported_seq":7103,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.451632+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:35.451632+0000","last_peered":"2024-09-16T04:45:35.451632+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:35.451632+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7024,"log_dups_size":0,"ondisk_log_size":7024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46830860,"num_objects":5717,"num_object_clones":0,"num_object_copies":22868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7022,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3445}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6797","reported_seq":6860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.545582+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:35.545582+0000","last_peered":"2024-09-16T04:45:35.545582+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:35.545582+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6797,"log_dups_size":0,"ondisk_log_size":6797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45178880,"num_objects":5515,"num_object_clones":0,"num_object_copies":22060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6938","reported_seq":7008,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.471604+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:35.471604+0000","last_peered":"2024-09-16T04:45:35.471604+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:35.471604+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6938,"log_dups_size":0,"ondisk_log_size":6938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46380300,"num_objects":5662,"num_object_clones":0,"num_object_copies":22648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6936,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3438}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7002","reported_seq":11558,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:32.468420+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:32.468420+0000","last_peered":"2024-09-16T04:45:32.468420+0000","last_clean":"2024-09-16T04:45:32.468420+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:32.468420+0000","last_undegraded":"2024-09-16T04:45:32.468420+0000","last_fullsized":"2024-09-16T04:45:32.468420+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7002,"log_dups_size":0,"ondisk_log_size":7002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:34.224583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47371532,"num_objects":5783,"num_object_clones":0,"num_object_copies":23132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7000,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7036","reported_seq":8640,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.542067+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:35.542067+0000","last_peered":"2024-09-16T04:45:35.542067+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:35.542067+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7036,"log_dups_size":0,"ondisk_log_size":7036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46852632,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7033,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":776,"num_bytes_recovered":6332416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7016","reported_seq":7099,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:34.955848+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:34.955848+0000","last_peered":"2024-09-16T04:45:34.955848+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:34.955848+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7016,"log_dups_size":0,"ondisk_log_size":7016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47035660,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7014,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6857","reported_seq":6932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.455356+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:35.455356+0000","last_peered":"2024-09-16T04:45:35.455356+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:35.455356+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6857,"log_dups_size":0,"ondisk_log_size":6857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46149522,"num_objects":5634,"num_object_clones":0,"num_object_copies":22536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6854,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6740","reported_seq":6793,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.471500+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:35.471500+0000","last_peered":"2024-09-16T04:45:35.471500+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:35.471500+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6740,"log_dups_size":0,"ondisk_log_size":6740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45800070,"num_objects":5591,"num_object_clones":0,"num_object_copies":22364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6739,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3469}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6878","reported_seq":6953,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:34.785595+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:34.785595+0000","last_peered":"2024-09-16T04:45:34.785595+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:34.785595+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6878,"log_dups_size":0,"ondisk_log_size":6878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46324358,"num_objects":5655,"num_object_clones":0,"num_object_copies":22620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6877,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3430}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6979","reported_seq":7064,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.448834+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:35.448834+0000","last_peered":"2024-09-16T04:45:35.448834+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:35.448834+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6979,"log_dups_size":0,"ondisk_log_size":6979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47535372,"num_objects":5803,"num_object_clones":0,"num_object_copies":23212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6977,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3584}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6829","reported_seq":6913,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.064074+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:35.064074+0000","last_peered":"2024-09-16T04:45:35.064074+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:35.064074+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6829,"log_dups_size":0,"ondisk_log_size":6829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46036236,"num_objects":5620,"num_object_clones":0,"num_object_copies":22480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3490}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6794","reported_seq":6867,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.500049+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:35.500049+0000","last_peered":"2024-09-16T04:45:35.500049+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:35.500049+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6794,"log_dups_size":0,"ondisk_log_size":6794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45159692,"num_objects":5513,"num_object_clones":0,"num_object_copies":22052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6792,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3277}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":743294656,"num_objects":90740,"num_object_clones":0,"num_object_copies":362960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90740,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110747,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3114,"num_bytes_recovered":24666112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1388470272,"data_stored":1388470272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110774,"ondisk_log_size":110774,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764496,"kb_used_data":363516,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93607344,"statfs":{"total":96636764160,"available":95853920256,"internally_reserved":0,"allocated":372240384,"data_stored":372060415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410578639},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,2,3,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760088,"kb_used_data":363776,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93611752,"statfs":{"total":96636764160,"available":95858434048,"internally_reserved":0,"allocated":372506624,"data_stored":372326655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2,4,1,0,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051634,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530304,"kb_used_data":268144,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93841536,"statfs":{"total":96636764160,"available":96093732864,"internally_reserved":0,"allocated":274579456,"data_stored":274399487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268431055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755472,"kb_used_data":362868,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93616368,"statfs":{"total":96636764160,"available":95863160832,"internally_reserved":0,"allocated":371576832,"data_stored":371396863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":401993422},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,6,2,0,0,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":371261440,"data_stored":371261440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274255872,"data_stored":274255872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":371716096,"data_stored":371716096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":371236864,"data_stored":371236864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:38.674 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:39.230 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:39.230 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:39.253 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-16T04:45:38.643944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":742614720,"num_objects":90657,"num_object_clones":0,"num_object_copies":362628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90657,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110830,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3114,"num_bytes_recovered":24666112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110857,"ondisk_log_size":110857,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2818212,"kb_used_data":1357836,"kb_used_omap":14,"kb_used_meta":1460273,"kb_avail":374669148,"statfs":{"total":386547056640,"available":383661207552,"internally_reserved":0,"allocated":1390424064,"data_stored":1389704188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1495320113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,2,15,11,4,3,0,2,0,0,0,0,0,5],"upper_bound":65536},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20275200,"num_objects":-2475,"num_object_clones":0,"num_object_copies":-9900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2475,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003186"},"pg_stats":[{"pgid":"1.f","version":"33'6955","reported_seq":7018,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.574793+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:35.574793+0000","last_peered":"2024-09-16T04:45:35.574793+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:35.574793+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6955,"log_dups_size":0,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46686208,"num_objects":5699,"num_object_clones":0,"num_object_copies":22796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6980","reported_seq":7053,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.462838+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:35.462838+0000","last_peered":"2024-09-16T04:45:35.462838+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:35.462838+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46642444,"num_objects":5694,"num_object_clones":0,"num_object_copies":22776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6978,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6959","reported_seq":7038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.039283+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:35.039283+0000","last_peered":"2024-09-16T04:45:35.039283+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:35.039283+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6959,"log_dups_size":0,"ondisk_log_size":6959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46783134,"num_objects":5712,"num_object_clones":0,"num_object_copies":22848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5712,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6958,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3486}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6990","reported_seq":7061,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.430403+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:35.430403+0000","last_peered":"2024-09-16T04:45:35.430403+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:35.430403+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6990,"log_dups_size":0,"ondisk_log_size":6990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46527756,"num_objects":5680,"num_object_clones":0,"num_object_copies":22720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6988,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7024","reported_seq":7103,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.451632+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:35.451632+0000","last_peered":"2024-09-16T04:45:35.451632+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:35.451632+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7024,"log_dups_size":0,"ondisk_log_size":7024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46830860,"num_objects":5717,"num_object_clones":0,"num_object_copies":22868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7022,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3445}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6797","reported_seq":6860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.545582+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:35.545582+0000","last_peered":"2024-09-16T04:45:35.545582+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:35.545582+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6797,"log_dups_size":0,"ondisk_log_size":6797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45178880,"num_objects":5515,"num_object_clones":0,"num_object_copies":22060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6938","reported_seq":7008,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.471604+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:35.471604+0000","last_peered":"2024-09-16T04:45:35.471604+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:35.471604+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6938,"log_dups_size":0,"ondisk_log_size":6938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46380300,"num_objects":5662,"num_object_clones":0,"num_object_copies":22648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6936,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3438}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7085","reported_seq":11641,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:37.438913+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:37.438913+0000","last_peered":"2024-09-16T04:45:37.438913+0000","last_clean":"2024-09-16T04:45:37.438913+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:37.438913+0000","last_undegraded":"2024-09-16T04:45:37.438913+0000","last_fullsized":"2024-09-16T04:45:37.438913+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7085,"log_dups_size":0,"ondisk_log_size":7085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:39.258821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46691596,"num_objects":5700,"num_object_clones":0,"num_object_copies":22800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7083,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7036","reported_seq":8640,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.542067+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:35.542067+0000","last_peered":"2024-09-16T04:45:35.542067+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:35.542067+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7036,"log_dups_size":0,"ondisk_log_size":7036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46852632,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7033,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":776,"num_bytes_recovered":6332416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7016","reported_seq":7099,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:34.955848+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:34.955848+0000","last_peered":"2024-09-16T04:45:34.955848+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:34.955848+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7016,"log_dups_size":0,"ondisk_log_size":7016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47035660,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7014,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6857","reported_seq":6932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.455356+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:35.455356+0000","last_peered":"2024-09-16T04:45:35.455356+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:35.455356+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6857,"log_dups_size":0,"ondisk_log_size":6857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46149522,"num_objects":5634,"num_object_clones":0,"num_object_copies":22536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6854,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6740","reported_seq":6793,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.471500+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:35.471500+0000","last_peered":"2024-09-16T04:45:35.471500+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:35.471500+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6740,"log_dups_size":0,"ondisk_log_size":6740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45800070,"num_objects":5591,"num_object_clones":0,"num_object_copies":22364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6739,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3469}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6878","reported_seq":6953,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:34.785595+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:34.785595+0000","last_peered":"2024-09-16T04:45:34.785595+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:34.785595+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6878,"log_dups_size":0,"ondisk_log_size":6878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46324358,"num_objects":5655,"num_object_clones":0,"num_object_copies":22620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6877,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3430}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6979","reported_seq":7064,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.448834+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:35.448834+0000","last_peered":"2024-09-16T04:45:35.448834+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:35.448834+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6979,"log_dups_size":0,"ondisk_log_size":6979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47535372,"num_objects":5803,"num_object_clones":0,"num_object_copies":23212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6977,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3584}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6829","reported_seq":6913,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.064074+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:35.064074+0000","last_peered":"2024-09-16T04:45:35.064074+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:35.064074+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6829,"log_dups_size":0,"ondisk_log_size":6829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46036236,"num_objects":5620,"num_object_clones":0,"num_object_copies":22480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3490}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6794","reported_seq":6867,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.500049+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:35.500049+0000","last_peered":"2024-09-16T04:45:35.500049+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:35.500049+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6794,"log_dups_size":0,"ondisk_log_size":6794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45159692,"num_objects":5513,"num_object_clones":0,"num_object_copies":22052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6792,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3277}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":742614720,"num_objects":90657,"num_object_clones":0,"num_object_copies":362628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90657,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110830,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3114,"num_bytes_recovered":24666112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1388015616,"data_stored":1388015616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110857,"ondisk_log_size":110857,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764496,"kb_used_data":363516,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93607344,"statfs":{"total":96636764160,"available":95853920256,"internally_reserved":0,"allocated":372240384,"data_stored":372060415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410578639},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,2,3,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760088,"kb_used_data":363776,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93611752,"statfs":{"total":96636764160,"available":95858434048,"internally_reserved":0,"allocated":372506624,"data_stored":372326655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2,4,1,0,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051635,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538156,"kb_used_data":267676,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93833684,"statfs":{"total":96636764160,"available":96085692416,"internally_reserved":0,"allocated":274100224,"data_stored":273920255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276950735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755472,"kb_used_data":362868,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93616368,"statfs":{"total":96636764160,"available":95863160832,"internally_reserved":0,"allocated":371576832,"data_stored":371396863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":401993422},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,6,2,0,0,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":371261440,"data_stored":371261440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273801216,"data_stored":273801216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":371716096,"data_stored":371716096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":371236864,"data_stored":371236864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:39.254 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:45:39.254 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6955", 'reported_seq': 7018, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:35.574793+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:45:35.574793+0000', 'last_peered': '2024-09-16T04:45:35.574793+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:45:35.574793+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6955, 'log_dups_size': 0, 'ondisk_log_size': 6955, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46686208, 'num_objects': 5699, 'num_object_clones': 0, 'num_object_copies': 22796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2210, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6955, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3489}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:39.254 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:45:39.254 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6980", 'reported_seq': 7053, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:35.462838+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:45:35.462838+0000', 'last_peered': '2024-09-16T04:45:35.462838+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:45:35.462838+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6980, 'log_dups_size': 0, 'ondisk_log_size': 6980, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46642444, 'num_objects': 5694, 'num_object_clones': 0, 'num_object_copies': 22776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2210, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5694, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6978, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3486}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:39.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:45:39.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6959", 'reported_seq': 7038, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:35.039283+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:45:35.039283+0000', 'last_peered': '2024-09-16T04:45:35.039283+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:45:35.039283+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6959, 'log_dups_size': 0, 'ondisk_log_size': 6959, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46783134, 'num_objects': 5712, 'num_object_clones': 0, 'num_object_copies': 22848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5712, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6958, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3486}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:39.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:45:39.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6990", 'reported_seq': 7061, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:35.430403+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:45:35.430403+0000', 'last_peered': '2024-09-16T04:45:35.430403+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:45:35.430403+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6990, 'log_dups_size': 0, 'ondisk_log_size': 6990, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46527756, 'num_objects': 5680, 'num_object_clones': 0, 'num_object_copies': 22720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5680, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6988, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3453}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:39.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:45:39.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7024", 'reported_seq': 7103, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:35.451632+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:45:35.451632+0000', 'last_peered': '2024-09-16T04:45:35.451632+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:45:35.451632+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7024, 'log_dups_size': 0, 'ondisk_log_size': 7024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46830860, 'num_objects': 5717, 'num_object_clones': 0, 'num_object_copies': 22868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2274, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5717, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7022, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3445}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:39.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:45:39.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6797", 'reported_seq': 6860, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:35.545582+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:45:35.545582+0000', 'last_peered': '2024-09-16T04:45:35.545582+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:45:35.545582+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6797, 'log_dups_size': 0, 'ondisk_log_size': 6797, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45178880, 'num_objects': 5515, 'num_object_clones': 0, 'num_object_copies': 22060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2199, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6797, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:39.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:45:39.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6938", 'reported_seq': 7008, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:35.471604+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:45:35.471604+0000', 'last_peered': '2024-09-16T04:45:35.471604+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:45:35.471604+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6938, 'log_dups_size': 0, 'ondisk_log_size': 6938, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46380300, 'num_objects': 5662, 'num_object_clones': 0, 'num_object_copies': 22648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5662, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6936, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3438}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:39.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:45:39.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7036", 'reported_seq': 8640, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:35.542067+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:45:35.542067+0000', 'last_peered': '2024-09-16T04:45:35.542067+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:45:35.542067+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7036, 'log_dups_size': 0, 'ondisk_log_size': 7036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46852632, 'num_objects': 5720, 'num_object_clones': 0, 'num_object_copies': 22880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1567, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7033, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 776, 'num_bytes_recovered': 6332416, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4153}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:39.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:45:39.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7016", 'reported_seq': 7099, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:34.955848+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:45:34.955848+0000', 'last_peered': '2024-09-16T04:45:34.955848+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:45:34.955848+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7016, 'log_dups_size': 0, 'ondisk_log_size': 7016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47035660, 'num_objects': 5742, 'num_object_clones': 0, 'num_object_copies': 22968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7014, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3544}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:39.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:45:39.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6857", 'reported_seq': 6932, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:35.455356+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:45:35.455356+0000', 'last_peered': '2024-09-16T04:45:35.455356+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:45:35.455356+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6857, 'log_dups_size': 0, 'ondisk_log_size': 6857, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46149522, 'num_objects': 5634, 'num_object_clones': 0, 'num_object_copies': 22536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2183, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6854, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3454}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:39.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:45:39.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6740", 'reported_seq': 6793, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:35.471500+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:45:35.471500+0000', 'last_peered': '2024-09-16T04:45:35.471500+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:45:35.471500+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6740, 'log_dups_size': 0, 'ondisk_log_size': 6740, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45800070, 'num_objects': 5591, 'num_object_clones': 0, 'num_object_copies': 22364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2123, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6739, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3469}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:39.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:45:39.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6878", 'reported_seq': 6953, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:34.785595+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:45:34.785595+0000', 'last_peered': '2024-09-16T04:45:34.785595+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:45:34.785595+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6878, 'log_dups_size': 0, 'ondisk_log_size': 6878, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46324358, 'num_objects': 5655, 'num_object_clones': 0, 'num_object_copies': 22620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5655, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6877, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3430}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:39.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:45:39.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6979", 'reported_seq': 7064, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:35.448834+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:45:35.448834+0000', 'last_peered': '2024-09-16T04:45:35.448834+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:45:35.448834+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6979, 'log_dups_size': 0, 'ondisk_log_size': 6979, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47535372, 'num_objects': 5803, 'num_object_clones': 0, 'num_object_copies': 23212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6977, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3584}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:39.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:45:39.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6829", 'reported_seq': 6913, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:35.064074+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:45:35.064074+0000', 'last_peered': '2024-09-16T04:45:35.064074+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:45:35.064074+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6829, 'log_dups_size': 0, 'ondisk_log_size': 6829, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46036236, 'num_objects': 5620, 'num_object_clones': 0, 'num_object_copies': 22480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5620, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6827, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3490}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:39.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:45:39.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6794", 'reported_seq': 6867, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:35.500049+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:45:35.500049+0000', 'last_peered': '2024-09-16T04:45:35.500049+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:45:35.500049+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6794, 'log_dups_size': 0, 'ondisk_log_size': 6794, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45159692, 'num_objects': 5513, 'num_object_clones': 0, 'num_object_copies': 22052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6792, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:39.257 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:45:39.991 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:39.992 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32465/362628 objects degraded (8.953%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":352,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":90657,"data_bytes":742614720,"bytes_used":2885849088,"bytes_avail":383661207552,"bytes_total":386547056640,"degraded_objects":32465,"degraded_total":362628,"degraded_ratio":0.089527008394277324,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":73025,"recovering_keys_per_sec":0,"num_objects_recovered":117,"num_bytes_recovered":876544,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":206},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:45:40.021 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:45:40.021 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:40.581 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:40.582 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:40.608 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-16T04:45:38.643944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":742614720,"num_objects":90657,"num_object_clones":0,"num_object_copies":362628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90657,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110830,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3114,"num_bytes_recovered":24666112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110857,"ondisk_log_size":110857,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2818212,"kb_used_data":1357836,"kb_used_omap":14,"kb_used_meta":1460273,"kb_avail":374669148,"statfs":{"total":386547056640,"available":383661207552,"internally_reserved":0,"allocated":1390424064,"data_stored":1389704188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1495320113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,2,15,11,4,3,0,2,0,0,0,0,0,5],"upper_bound":65536},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20275200,"num_objects":-2475,"num_object_clones":0,"num_object_copies":-9900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2475,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003186"},"pg_stats":[{"pgid":"1.f","version":"33'6955","reported_seq":7018,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.574793+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:35.574793+0000","last_peered":"2024-09-16T04:45:35.574793+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:35.574793+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6955,"log_dups_size":0,"ondisk_log_size":6955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46686208,"num_objects":5699,"num_object_clones":0,"num_object_copies":22796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6980","reported_seq":7053,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.462838+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:35.462838+0000","last_peered":"2024-09-16T04:45:35.462838+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:35.462838+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46642444,"num_objects":5694,"num_object_clones":0,"num_object_copies":22776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6978,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6959","reported_seq":7038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.039283+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:35.039283+0000","last_peered":"2024-09-16T04:45:35.039283+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:35.039283+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6959,"log_dups_size":0,"ondisk_log_size":6959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46783134,"num_objects":5712,"num_object_clones":0,"num_object_copies":22848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5712,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6958,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3486}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6990","reported_seq":7061,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.430403+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:35.430403+0000","last_peered":"2024-09-16T04:45:35.430403+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:35.430403+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6990,"log_dups_size":0,"ondisk_log_size":6990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46527756,"num_objects":5680,"num_object_clones":0,"num_object_copies":22720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6988,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7024","reported_seq":7103,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.451632+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:35.451632+0000","last_peered":"2024-09-16T04:45:35.451632+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:35.451632+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7024,"log_dups_size":0,"ondisk_log_size":7024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46830860,"num_objects":5717,"num_object_clones":0,"num_object_copies":22868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7022,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3445}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6797","reported_seq":6860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.545582+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:35.545582+0000","last_peered":"2024-09-16T04:45:35.545582+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:35.545582+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6797,"log_dups_size":0,"ondisk_log_size":6797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45178880,"num_objects":5515,"num_object_clones":0,"num_object_copies":22060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6938","reported_seq":7008,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.471604+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:35.471604+0000","last_peered":"2024-09-16T04:45:35.471604+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:35.471604+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6938,"log_dups_size":0,"ondisk_log_size":6938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46380300,"num_objects":5662,"num_object_clones":0,"num_object_copies":22648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6936,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3438}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7085","reported_seq":11641,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:37.438913+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:37.438913+0000","last_peered":"2024-09-16T04:45:37.438913+0000","last_clean":"2024-09-16T04:45:37.438913+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:37.438913+0000","last_undegraded":"2024-09-16T04:45:37.438913+0000","last_fullsized":"2024-09-16T04:45:37.438913+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7085,"log_dups_size":0,"ondisk_log_size":7085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:39.258821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46691596,"num_objects":5700,"num_object_clones":0,"num_object_copies":22800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7083,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7036","reported_seq":8640,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.542067+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:35.542067+0000","last_peered":"2024-09-16T04:45:35.542067+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:35.542067+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7036,"log_dups_size":0,"ondisk_log_size":7036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46852632,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7033,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":776,"num_bytes_recovered":6332416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7016","reported_seq":7099,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:34.955848+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:34.955848+0000","last_peered":"2024-09-16T04:45:34.955848+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:34.955848+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7016,"log_dups_size":0,"ondisk_log_size":7016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47035660,"num_objects":5742,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7014,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6857","reported_seq":6932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.455356+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:35.455356+0000","last_peered":"2024-09-16T04:45:35.455356+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:35.455356+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6857,"log_dups_size":0,"ondisk_log_size":6857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46149522,"num_objects":5634,"num_object_clones":0,"num_object_copies":22536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6854,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6740","reported_seq":6793,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.471500+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:35.471500+0000","last_peered":"2024-09-16T04:45:35.471500+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:35.471500+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6740,"log_dups_size":0,"ondisk_log_size":6740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45800070,"num_objects":5591,"num_object_clones":0,"num_object_copies":22364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6739,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3469}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6878","reported_seq":6953,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:34.785595+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:34.785595+0000","last_peered":"2024-09-16T04:45:34.785595+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:34.785595+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6878,"log_dups_size":0,"ondisk_log_size":6878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46324358,"num_objects":5655,"num_object_clones":0,"num_object_copies":22620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6877,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3430}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6979","reported_seq":7064,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.448834+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:35.448834+0000","last_peered":"2024-09-16T04:45:35.448834+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:35.448834+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6979,"log_dups_size":0,"ondisk_log_size":6979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47535372,"num_objects":5803,"num_object_clones":0,"num_object_copies":23212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6977,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3584}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6829","reported_seq":6913,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.064074+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:35.064074+0000","last_peered":"2024-09-16T04:45:35.064074+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:35.064074+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6829,"log_dups_size":0,"ondisk_log_size":6829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46036236,"num_objects":5620,"num_object_clones":0,"num_object_copies":22480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3490}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6794","reported_seq":6867,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:35.500049+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:35.500049+0000","last_peered":"2024-09-16T04:45:35.500049+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:35.500049+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6794,"log_dups_size":0,"ondisk_log_size":6794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45159692,"num_objects":5513,"num_object_clones":0,"num_object_copies":22052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6792,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3277}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":742614720,"num_objects":90657,"num_object_clones":0,"num_object_copies":362628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90657,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110830,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3114,"num_bytes_recovered":24666112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1388015616,"data_stored":1388015616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110857,"ondisk_log_size":110857,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764496,"kb_used_data":363516,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93607344,"statfs":{"total":96636764160,"available":95853920256,"internally_reserved":0,"allocated":372240384,"data_stored":372060415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410578639},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,2,3,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760088,"kb_used_data":363776,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93611752,"statfs":{"total":96636764160,"available":95858434048,"internally_reserved":0,"allocated":372506624,"data_stored":372326655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2,4,1,0,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051635,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538156,"kb_used_data":267676,"kb_used_omap":4,"kb_used_meta":270459,"kb_avail":93833684,"statfs":{"total":96636764160,"available":96085692416,"internally_reserved":0,"allocated":274100224,"data_stored":273920255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276950735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755472,"kb_used_data":362868,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93616368,"statfs":{"total":96636764160,"available":95863160832,"internally_reserved":0,"allocated":371576832,"data_stored":371396863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":401993422},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,6,2,0,0,0,0,0,0,0,0,0,3],"upper_bound":65536},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":371261440,"data_stored":371261440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273801216,"data_stored":273801216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":371716096,"data_stored":371716096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":371236864,"data_stored":371236864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:43.613 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:44.200 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:44.201 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:44.219 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-16T04:45:42.646692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":732341952,"num_objects":89403,"num_object_clones":0,"num_object_copies":357612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89403,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112084,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3157,"num_bytes_recovered":25018368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112111,"ondisk_log_size":112111,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2820800,"kb_used_data":1342384,"kb_used_omap":14,"kb_used_meta":1478321,"kb_avail":374666560,"statfs":{"total":386547056640,"available":383658557440,"internally_reserved":0,"allocated":1374601216,"data_stored":1373881340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1513801265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,8,6,6,13,11,6,0,0,0,0,2,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20717568,"num_objects":-2529,"num_object_clones":0,"num_object_copies":-10116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2529,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004918"},"pg_stats":[{"pgid":"1.f","version":"33'7032","reported_seq":7095,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.473640+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:40.473640+0000","last_peered":"2024-09-16T04:45:40.473640+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:40.473640+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7032,"log_dups_size":0,"ondisk_log_size":7032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46055424,"num_objects":5622,"num_object_clones":0,"num_object_copies":22488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7032,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7056","reported_seq":7127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.450659+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:40.450659+0000","last_peered":"2024-09-16T04:45:40.450659+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:40.450659+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7056,"log_dups_size":0,"ondisk_log_size":7056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46019852,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7054,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7028","reported_seq":7107,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.058707+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:40.058707+0000","last_peered":"2024-09-16T04:45:40.058707+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:40.058707+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7028,"log_dups_size":0,"ondisk_log_size":7028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46217886,"num_objects":5643,"num_object_clones":0,"num_object_copies":22572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5643,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7027,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3417}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7073","reported_seq":7144,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.544687+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:40.544687+0000","last_peered":"2024-09-16T04:45:40.544687+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:40.544687+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7073,"log_dups_size":0,"ondisk_log_size":7073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45847820,"num_objects":5597,"num_object_clones":0,"num_object_copies":22388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7071,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7125","reported_seq":7204,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.632354+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:40.632354+0000","last_peered":"2024-09-16T04:45:40.632354+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:40.632354+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46003468,"num_objects":5616,"num_object_clones":0,"num_object_copies":22464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7123,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3344}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6867","reported_seq":6930,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.282750+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:40.282750+0000","last_peered":"2024-09-16T04:45:40.282750+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:40.282750+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6867,"log_dups_size":0,"ondisk_log_size":6867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44605440,"num_objects":5445,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6867,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7015","reported_seq":7085,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.367910+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:40.367910+0000","last_peered":"2024-09-16T04:45:40.367910+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:40.367910+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7015,"log_dups_size":0,"ondisk_log_size":7015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45749516,"num_objects":5585,"num_object_clones":0,"num_object_copies":22340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7013,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7159","reported_seq":11715,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:42.444236+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:42.444236+0000","last_peered":"2024-09-16T04:45:42.444236+0000","last_clean":"2024-09-16T04:45:42.444236+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:42.444236+0000","last_undegraded":"2024-09-16T04:45:42.444236+0000","last_fullsized":"2024-09-16T04:45:42.444236+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7159,"log_dups_size":0,"ondisk_log_size":7159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:44.330959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46085388,"num_objects":5626,"num_object_clones":0,"num_object_copies":22504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7157,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7125","reported_seq":8816,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.579650+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:40.579650+0000","last_peered":"2024-09-16T04:45:40.579650+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:40.579650+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46123544,"num_objects":5631,"num_object_clones":0,"num_object_copies":22524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7122,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":819,"num_bytes_recovered":6684672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7091","reported_seq":7174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.125966+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:40.125966+0000","last_peered":"2024-09-16T04:45:40.125966+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:40.125966+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7091,"log_dups_size":0,"ondisk_log_size":7091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46421260,"num_objects":5667,"num_object_clones":0,"num_object_copies":22668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7089,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3469}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6930","reported_seq":7007,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.608292+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:40.608292+0000","last_peered":"2024-09-16T04:45:40.608292+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:40.608292+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6930,"log_dups_size":0,"ondisk_log_size":6930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45551506,"num_objects":5561,"num_object_clones":0,"num_object_copies":22244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6927,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3381}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6825","reported_seq":6878,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.635176+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:40.635176+0000","last_peered":"2024-09-16T04:45:40.635176+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:40.635176+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6825,"log_dups_size":0,"ondisk_log_size":6825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45103750,"num_objects":5506,"num_object_clones":0,"num_object_copies":22024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6824,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3384}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6951","reported_seq":7026,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.151415+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:40.151415+0000","last_peered":"2024-09-16T04:45:40.151415+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:40.151415+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6951,"log_dups_size":0,"ondisk_log_size":6951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45726342,"num_objects":5582,"num_object_clones":0,"num_object_copies":22328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6950,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3357}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7062","reported_seq":7147,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.562385+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:40.562385+0000","last_peered":"2024-09-16T04:45:40.562385+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:40.562385+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7062,"log_dups_size":0,"ondisk_log_size":7062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46855436,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7060,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3501}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6900","reported_seq":6984,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.013879+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:40.013879+0000","last_peered":"2024-09-16T04:45:40.013879+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:40.013879+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6900,"log_dups_size":0,"ondisk_log_size":6900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45454604,"num_objects":5549,"num_object_clones":0,"num_object_copies":22196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6898,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3419}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6872","reported_seq":6945,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.484257+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:40.484257+0000","last_peered":"2024-09-16T04:45:40.484257+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:40.484257+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6872,"log_dups_size":0,"ondisk_log_size":6872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44520716,"num_objects":5435,"num_object_clones":0,"num_object_copies":21740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6870,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3199}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":732341952,"num_objects":89403,"num_object_clones":0,"num_object_copies":357612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89403,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112084,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3157,"num_bytes_recovered":25018368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1372151808,"data_stored":1372151808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112111,"ondisk_log_size":112111,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746988,"kb_used_data":358484,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93624852,"statfs":{"total":96636764160,"available":95871848448,"internally_reserved":0,"allocated":367087616,"data_stored":366907647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397799119},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,4,3,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759196,"kb_used_data":358724,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93612644,"statfs":{"total":96636764160,"available":95859347456,"internally_reserved":0,"allocated":367333376,"data_stored":367153407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,1,1,0,2,4,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051636,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559916,"kb_used_data":267228,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93811924,"statfs":{"total":96636764160,"available":96063410176,"internally_reserved":0,"allocated":273641472,"data_stored":273461503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299691727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754700,"kb_used_data":357948,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93617140,"statfs":{"total":96636764160,"available":95863951360,"internally_reserved":0,"allocated":366538752,"data_stored":366358783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":406253262},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,0,0,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":366198784,"data_stored":366198784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273334272,"data_stored":273334272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366526464,"data_stored":366526464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":366092288,"data_stored":366092288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:44.224 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:44.753 INFO:tasks.ceph.osd.2.smithi152.stderr:2024-09-16T04:45:44.750+0000 7f3c61a1c640 -1 osd.2 33 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4991.0:104540 1.5s0 1:af063a43:::benchmark_data_smithi152_43478_object3836:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e33) 2024-09-16T04:45:44.860 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:44.864 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:44.890 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-16T04:45:44.647412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":732341952,"num_objects":89403,"num_object_clones":0,"num_object_copies":357612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89403,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112084,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3157,"num_bytes_recovered":25018368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112111,"ondisk_log_size":112111,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2820800,"kb_used_data":1342384,"kb_used_omap":14,"kb_used_meta":1478321,"kb_avail":374666560,"statfs":{"total":386547056640,"available":383658557440,"internally_reserved":0,"allocated":1374601216,"data_stored":1373881340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1513801265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,8,6,6,13,11,6,0,0,0,0,2,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20045824,"num_objects":-2447,"num_object_clones":0,"num_object_copies":-9788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2447,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005081"},"pg_stats":[{"pgid":"1.f","version":"33'7032","reported_seq":7095,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.473640+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:40.473640+0000","last_peered":"2024-09-16T04:45:40.473640+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:40.473640+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7032,"log_dups_size":0,"ondisk_log_size":7032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46055424,"num_objects":5622,"num_object_clones":0,"num_object_copies":22488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7032,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7056","reported_seq":7127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.450659+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:40.450659+0000","last_peered":"2024-09-16T04:45:40.450659+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:40.450659+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7056,"log_dups_size":0,"ondisk_log_size":7056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46019852,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7054,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7028","reported_seq":7107,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.058707+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:40.058707+0000","last_peered":"2024-09-16T04:45:40.058707+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:40.058707+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7028,"log_dups_size":0,"ondisk_log_size":7028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46217886,"num_objects":5643,"num_object_clones":0,"num_object_copies":22572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5643,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7027,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3417}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7073","reported_seq":7144,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.544687+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:40.544687+0000","last_peered":"2024-09-16T04:45:40.544687+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:40.544687+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7073,"log_dups_size":0,"ondisk_log_size":7073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45847820,"num_objects":5597,"num_object_clones":0,"num_object_copies":22388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7071,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7125","reported_seq":7204,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.632354+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:40.632354+0000","last_peered":"2024-09-16T04:45:40.632354+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:40.632354+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46003468,"num_objects":5616,"num_object_clones":0,"num_object_copies":22464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7123,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3344}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6867","reported_seq":6930,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.282750+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:40.282750+0000","last_peered":"2024-09-16T04:45:40.282750+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:40.282750+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6867,"log_dups_size":0,"ondisk_log_size":6867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44605440,"num_objects":5445,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6867,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7015","reported_seq":7085,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.367910+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:40.367910+0000","last_peered":"2024-09-16T04:45:40.367910+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:40.367910+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7015,"log_dups_size":0,"ondisk_log_size":7015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45749516,"num_objects":5585,"num_object_clones":0,"num_object_copies":22340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7013,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7159","reported_seq":11715,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:42.444236+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:42.444236+0000","last_peered":"2024-09-16T04:45:42.444236+0000","last_clean":"2024-09-16T04:45:42.444236+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:42.444236+0000","last_undegraded":"2024-09-16T04:45:42.444236+0000","last_fullsized":"2024-09-16T04:45:42.444236+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7159,"log_dups_size":0,"ondisk_log_size":7159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:44.330959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46085388,"num_objects":5626,"num_object_clones":0,"num_object_copies":22504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7157,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7125","reported_seq":8816,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.579650+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:40.579650+0000","last_peered":"2024-09-16T04:45:40.579650+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:40.579650+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46123544,"num_objects":5631,"num_object_clones":0,"num_object_copies":22524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7122,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":819,"num_bytes_recovered":6684672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7091","reported_seq":7174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.125966+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:40.125966+0000","last_peered":"2024-09-16T04:45:40.125966+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:40.125966+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7091,"log_dups_size":0,"ondisk_log_size":7091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46421260,"num_objects":5667,"num_object_clones":0,"num_object_copies":22668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7089,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3469}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6930","reported_seq":7007,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.608292+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:40.608292+0000","last_peered":"2024-09-16T04:45:40.608292+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:40.608292+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6930,"log_dups_size":0,"ondisk_log_size":6930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45551506,"num_objects":5561,"num_object_clones":0,"num_object_copies":22244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6927,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3381}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6825","reported_seq":6878,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.635176+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:40.635176+0000","last_peered":"2024-09-16T04:45:40.635176+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:40.635176+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6825,"log_dups_size":0,"ondisk_log_size":6825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45103750,"num_objects":5506,"num_object_clones":0,"num_object_copies":22024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6824,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3384}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6951","reported_seq":7026,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.151415+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:40.151415+0000","last_peered":"2024-09-16T04:45:40.151415+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:40.151415+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6951,"log_dups_size":0,"ondisk_log_size":6951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45726342,"num_objects":5582,"num_object_clones":0,"num_object_copies":22328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6950,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3357}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7062","reported_seq":7147,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.562385+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:40.562385+0000","last_peered":"2024-09-16T04:45:40.562385+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:40.562385+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7062,"log_dups_size":0,"ondisk_log_size":7062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46855436,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7060,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3501}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6900","reported_seq":6984,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.013879+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:40.013879+0000","last_peered":"2024-09-16T04:45:40.013879+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:40.013879+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6900,"log_dups_size":0,"ondisk_log_size":6900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45454604,"num_objects":5549,"num_object_clones":0,"num_object_copies":22196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6898,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3419}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6872","reported_seq":6945,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.484257+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:40.484257+0000","last_peered":"2024-09-16T04:45:40.484257+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:40.484257+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6872,"log_dups_size":0,"ondisk_log_size":6872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44520716,"num_objects":5435,"num_object_clones":0,"num_object_copies":21740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6870,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3199}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":732341952,"num_objects":89403,"num_object_clones":0,"num_object_copies":357612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89403,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112084,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3157,"num_bytes_recovered":25018368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1372151808,"data_stored":1372151808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112111,"ondisk_log_size":112111,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746988,"kb_used_data":358484,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93624852,"statfs":{"total":96636764160,"available":95871848448,"internally_reserved":0,"allocated":367087616,"data_stored":366907647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397799119},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,4,3,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759196,"kb_used_data":358724,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93612644,"statfs":{"total":96636764160,"available":95859347456,"internally_reserved":0,"allocated":367333376,"data_stored":367153407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,1,1,0,2,4,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051636,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559916,"kb_used_data":267228,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93811924,"statfs":{"total":96636764160,"available":96063410176,"internally_reserved":0,"allocated":273641472,"data_stored":273461503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299691727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754700,"kb_used_data":357948,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93617140,"statfs":{"total":96636764160,"available":95863951360,"internally_reserved":0,"allocated":366538752,"data_stored":366358783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":406253262},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,0,0,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":366198784,"data_stored":366198784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273334272,"data_stored":273334272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366526464,"data_stored":366526464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":366092288,"data_stored":366092288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:44.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:45:44.891 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7032", 'reported_seq': 7095, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:40.473640+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:45:40.473640+0000', 'last_peered': '2024-09-16T04:45:40.473640+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:45:40.473640+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7032, 'log_dups_size': 0, 'ondisk_log_size': 7032, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46055424, 'num_objects': 5622, 'num_object_clones': 0, 'num_object_copies': 22488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2210, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7032, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3412}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:44.891 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:45:44.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7056", 'reported_seq': 7127, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:40.450659+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:45:40.450659+0000', 'last_peered': '2024-09-16T04:45:40.450659+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:45:40.450659+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7056, 'log_dups_size': 0, 'ondisk_log_size': 7056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46019852, 'num_objects': 5618, 'num_object_clones': 0, 'num_object_copies': 22472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2210, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5618, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7054, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3410}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:44.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:45:44.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7028", 'reported_seq': 7107, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:40.058707+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:45:40.058707+0000', 'last_peered': '2024-09-16T04:45:40.058707+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:45:40.058707+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7028, 'log_dups_size': 0, 'ondisk_log_size': 7028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46217886, 'num_objects': 5643, 'num_object_clones': 0, 'num_object_copies': 22572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5643, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7027, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3417}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:44.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:45:44.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7073", 'reported_seq': 7144, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:40.544687+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:45:40.544687+0000', 'last_peered': '2024-09-16T04:45:40.544687+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:45:40.544687+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7073, 'log_dups_size': 0, 'ondisk_log_size': 7073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45847820, 'num_objects': 5597, 'num_object_clones': 0, 'num_object_copies': 22388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7071, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3370}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:44.892 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:45:44.892 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7125", 'reported_seq': 7204, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:40.632354+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:45:40.632354+0000', 'last_peered': '2024-09-16T04:45:40.632354+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:45:40.632354+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7125, 'log_dups_size': 0, 'ondisk_log_size': 7125, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46003468, 'num_objects': 5616, 'num_object_clones': 0, 'num_object_copies': 22464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2274, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7123, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3344}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:44.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:45:44.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6867", 'reported_seq': 6930, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:40.282750+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:45:40.282750+0000', 'last_peered': '2024-09-16T04:45:40.282750+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:45:40.282750+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6867, 'log_dups_size': 0, 'ondisk_log_size': 6867, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44605440, 'num_objects': 5445, 'num_object_clones': 0, 'num_object_copies': 21780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2199, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5445, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6867, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:44.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:45:44.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7015", 'reported_seq': 7085, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:40.367910+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:45:40.367910+0000', 'last_peered': '2024-09-16T04:45:40.367910+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:45:40.367910+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7015, 'log_dups_size': 0, 'ondisk_log_size': 7015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45749516, 'num_objects': 5585, 'num_object_clones': 0, 'num_object_copies': 22340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7013, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3361}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:44.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:45:44.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7125", 'reported_seq': 8816, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:40.579650+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:45:40.579650+0000', 'last_peered': '2024-09-16T04:45:40.579650+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:45:40.579650+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7125, 'log_dups_size': 0, 'ondisk_log_size': 7125, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46123544, 'num_objects': 5631, 'num_object_clones': 0, 'num_object_copies': 22524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1524, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5631, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7122, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 819, 'num_bytes_recovered': 6684672, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4107}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:44.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:45:44.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7091", 'reported_seq': 7174, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:40.125966+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:45:40.125966+0000', 'last_peered': '2024-09-16T04:45:40.125966+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:45:40.125966+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7091, 'log_dups_size': 0, 'ondisk_log_size': 7091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46421260, 'num_objects': 5667, 'num_object_clones': 0, 'num_object_copies': 22668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5667, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7089, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3469}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:44.893 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:45:44.893 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6930", 'reported_seq': 7007, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:40.608292+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:45:40.608292+0000', 'last_peered': '2024-09-16T04:45:40.608292+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:45:40.608292+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6930, 'log_dups_size': 0, 'ondisk_log_size': 6930, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45551506, 'num_objects': 5561, 'num_object_clones': 0, 'num_object_copies': 22244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2183, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5561, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6927, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3381}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:44.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:45:44.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6825", 'reported_seq': 6878, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:40.635176+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:45:40.635176+0000', 'last_peered': '2024-09-16T04:45:40.635176+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:45:40.635176+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6825, 'log_dups_size': 0, 'ondisk_log_size': 6825, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45103750, 'num_objects': 5506, 'num_object_clones': 0, 'num_object_copies': 22024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2123, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5506, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6824, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:44.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:45:44.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6951", 'reported_seq': 7026, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:40.151415+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:45:40.151415+0000', 'last_peered': '2024-09-16T04:45:40.151415+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:45:40.151415+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6951, 'log_dups_size': 0, 'ondisk_log_size': 6951, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45726342, 'num_objects': 5582, 'num_object_clones': 0, 'num_object_copies': 22328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5582, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6950, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3357}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:44.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:45:44.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7062", 'reported_seq': 7147, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:40.562385+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:45:40.562385+0000', 'last_peered': '2024-09-16T04:45:40.562385+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:45:40.562385+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7062, 'log_dups_size': 0, 'ondisk_log_size': 7062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46855436, 'num_objects': 5720, 'num_object_clones': 0, 'num_object_copies': 22880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7060, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:44.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:45:44.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6900", 'reported_seq': 6984, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:40.013879+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:45:40.013879+0000', 'last_peered': '2024-09-16T04:45:40.013879+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:45:40.013879+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6900, 'log_dups_size': 0, 'ondisk_log_size': 6900, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45454604, 'num_objects': 5549, 'num_object_clones': 0, 'num_object_copies': 22196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5549, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6898, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3419}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:44.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:45:44.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6872", 'reported_seq': 6945, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:40.484257+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:45:40.484257+0000', 'last_peered': '2024-09-16T04:45:40.484257+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:45:40.484257+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6872, 'log_dups_size': 0, 'ondisk_log_size': 6872, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44520716, 'num_objects': 5435, 'num_object_clones': 0, 'num_object_copies': 21740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5435, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6870, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3199}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:44.895 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:45:45.646 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:45.647 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32422/357612 objects degraded (9.066%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"0 slow ops, oldest one blocked for 30 sec, osd.2 has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":357,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":89403,"data_bytes":732341952,"bytes_used":2888499200,"bytes_avail":383658557440,"bytes_total":386547056640,"degraded_objects":32422,"degraded_total":357612,"degraded_ratio":0.090662505732469825,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":67555,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":811008,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":203},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:45:45.677 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:45:45.677 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:46.255 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:46.255 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:46.276 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-16T04:45:44.647412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":732341952,"num_objects":89403,"num_object_clones":0,"num_object_copies":357612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89403,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112084,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3157,"num_bytes_recovered":25018368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112111,"ondisk_log_size":112111,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2820800,"kb_used_data":1342384,"kb_used_omap":14,"kb_used_meta":1478321,"kb_avail":374666560,"statfs":{"total":386547056640,"available":383658557440,"internally_reserved":0,"allocated":1374601216,"data_stored":1373881340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1513801265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,8,6,6,13,11,6,0,0,0,0,2,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20045824,"num_objects":-2447,"num_object_clones":0,"num_object_copies":-9788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2447,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005081"},"pg_stats":[{"pgid":"1.f","version":"33'7032","reported_seq":7095,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.473640+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:40.473640+0000","last_peered":"2024-09-16T04:45:40.473640+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:40.473640+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7032,"log_dups_size":0,"ondisk_log_size":7032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46055424,"num_objects":5622,"num_object_clones":0,"num_object_copies":22488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7032,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7056","reported_seq":7127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.450659+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:40.450659+0000","last_peered":"2024-09-16T04:45:40.450659+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:40.450659+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7056,"log_dups_size":0,"ondisk_log_size":7056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46019852,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7054,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7028","reported_seq":7107,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.058707+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:40.058707+0000","last_peered":"2024-09-16T04:45:40.058707+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:40.058707+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7028,"log_dups_size":0,"ondisk_log_size":7028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46217886,"num_objects":5643,"num_object_clones":0,"num_object_copies":22572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5643,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7027,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3417}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7073","reported_seq":7144,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.544687+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:40.544687+0000","last_peered":"2024-09-16T04:45:40.544687+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:40.544687+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7073,"log_dups_size":0,"ondisk_log_size":7073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45847820,"num_objects":5597,"num_object_clones":0,"num_object_copies":22388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7071,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7125","reported_seq":7204,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.632354+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:40.632354+0000","last_peered":"2024-09-16T04:45:40.632354+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:40.632354+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46003468,"num_objects":5616,"num_object_clones":0,"num_object_copies":22464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7123,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3344}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6867","reported_seq":6930,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.282750+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:40.282750+0000","last_peered":"2024-09-16T04:45:40.282750+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:40.282750+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6867,"log_dups_size":0,"ondisk_log_size":6867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44605440,"num_objects":5445,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6867,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7015","reported_seq":7085,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.367910+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:40.367910+0000","last_peered":"2024-09-16T04:45:40.367910+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:40.367910+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7015,"log_dups_size":0,"ondisk_log_size":7015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45749516,"num_objects":5585,"num_object_clones":0,"num_object_copies":22340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7013,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7159","reported_seq":11715,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:42.444236+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:42.444236+0000","last_peered":"2024-09-16T04:45:42.444236+0000","last_clean":"2024-09-16T04:45:42.444236+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:42.444236+0000","last_undegraded":"2024-09-16T04:45:42.444236+0000","last_fullsized":"2024-09-16T04:45:42.444236+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7159,"log_dups_size":0,"ondisk_log_size":7159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:44.330959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46085388,"num_objects":5626,"num_object_clones":0,"num_object_copies":22504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7157,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7125","reported_seq":8816,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.579650+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:40.579650+0000","last_peered":"2024-09-16T04:45:40.579650+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:40.579650+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46123544,"num_objects":5631,"num_object_clones":0,"num_object_copies":22524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7122,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":819,"num_bytes_recovered":6684672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7091","reported_seq":7174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.125966+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:40.125966+0000","last_peered":"2024-09-16T04:45:40.125966+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:40.125966+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7091,"log_dups_size":0,"ondisk_log_size":7091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46421260,"num_objects":5667,"num_object_clones":0,"num_object_copies":22668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7089,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3469}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6930","reported_seq":7007,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.608292+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:40.608292+0000","last_peered":"2024-09-16T04:45:40.608292+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:40.608292+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6930,"log_dups_size":0,"ondisk_log_size":6930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45551506,"num_objects":5561,"num_object_clones":0,"num_object_copies":22244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6927,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3381}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6825","reported_seq":6878,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.635176+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:40.635176+0000","last_peered":"2024-09-16T04:45:40.635176+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:40.635176+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6825,"log_dups_size":0,"ondisk_log_size":6825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45103750,"num_objects":5506,"num_object_clones":0,"num_object_copies":22024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6824,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3384}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6951","reported_seq":7026,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.151415+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:40.151415+0000","last_peered":"2024-09-16T04:45:40.151415+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:40.151415+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6951,"log_dups_size":0,"ondisk_log_size":6951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45726342,"num_objects":5582,"num_object_clones":0,"num_object_copies":22328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6950,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3357}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7062","reported_seq":7147,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.562385+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:40.562385+0000","last_peered":"2024-09-16T04:45:40.562385+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:40.562385+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7062,"log_dups_size":0,"ondisk_log_size":7062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46855436,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7060,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3501}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6900","reported_seq":6984,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.013879+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:40.013879+0000","last_peered":"2024-09-16T04:45:40.013879+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:40.013879+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6900,"log_dups_size":0,"ondisk_log_size":6900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45454604,"num_objects":5549,"num_object_clones":0,"num_object_copies":22196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6898,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3419}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6872","reported_seq":6945,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:40.484257+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:40.484257+0000","last_peered":"2024-09-16T04:45:40.484257+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:40.484257+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6872,"log_dups_size":0,"ondisk_log_size":6872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44520716,"num_objects":5435,"num_object_clones":0,"num_object_copies":21740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6870,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3199}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":732341952,"num_objects":89403,"num_object_clones":0,"num_object_copies":357612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89403,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112084,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3157,"num_bytes_recovered":25018368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1372151808,"data_stored":1372151808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112111,"ondisk_log_size":112111,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746988,"kb_used_data":358484,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93624852,"statfs":{"total":96636764160,"available":95871848448,"internally_reserved":0,"allocated":367087616,"data_stored":366907647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397799119},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,4,3,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759196,"kb_used_data":358724,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93612644,"statfs":{"total":96636764160,"available":95859347456,"internally_reserved":0,"allocated":367333376,"data_stored":367153407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,1,1,0,2,4,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051636,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559916,"kb_used_data":267228,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93811924,"statfs":{"total":96636764160,"available":96063410176,"internally_reserved":0,"allocated":273641472,"data_stored":273461503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299691727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754700,"kb_used_data":357948,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93617140,"statfs":{"total":96636764160,"available":95863951360,"internally_reserved":0,"allocated":366538752,"data_stored":366358783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":406253262},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,0,0,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":366198784,"data_stored":366198784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273334272,"data_stored":273334272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366526464,"data_stored":366526464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":366092288,"data_stored":366092288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:49.279 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:49.926 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:49.927 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:49.956 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-16T04:45:48.648462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":722208448,"num_objects":88166,"num_object_clones":0,"num_object_copies":352664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88166,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113321,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3209,"num_bytes_recovered":25411584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113348,"ondisk_log_size":113348,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789016,"kb_used_data":1327224,"kb_used_omap":14,"kb_used_meta":1461681,"kb_avail":374698344,"statfs":{"total":386547056640,"available":383691104256,"internally_reserved":0,"allocated":1359077376,"data_stored":1358357500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1496761905},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,16,7,9,10,13,6,0,0,0,0,3,0,0,0,2],"upper_bound":131072},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21086208,"num_objects":-2574,"num_object_clones":0,"num_object_copies":-10296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":745472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005189"},"pg_stats":[{"pgid":"1.f","version":"33'7117","reported_seq":7182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.505042+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:45.505042+0000","last_peered":"2024-09-16T04:45:45.505042+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:45.505042+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7117,"log_dups_size":0,"ondisk_log_size":7117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45359104,"num_objects":5537,"num_object_clones":0,"num_object_copies":22148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7127","reported_seq":7200,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.537610+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:45.537610+0000","last_peered":"2024-09-16T04:45:45.537610+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:45.537610+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7127,"log_dups_size":0,"ondisk_log_size":7127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45438220,"num_objects":5547,"num_object_clones":0,"num_object_copies":22188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7125,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3339}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7111","reported_seq":7190,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.159459+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:45.159459+0000","last_peered":"2024-09-16T04:45:45.159459+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:45.159459+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7111,"log_dups_size":0,"ondisk_log_size":7111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45537950,"num_objects":5560,"num_object_clones":0,"num_object_copies":22240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5560,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7110,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3334}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7149","reported_seq":7220,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.536860+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:45.536860+0000","last_peered":"2024-09-16T04:45:45.536860+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:45.536860+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7149,"log_dups_size":0,"ondisk_log_size":7149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45225228,"num_objects":5521,"num_object_clones":0,"num_object_copies":22084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7147,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3294}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7205","reported_seq":7286,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.559449+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:45.559449+0000","last_peered":"2024-09-16T04:45:45.559449+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:45.559449+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7205,"log_dups_size":0,"ondisk_log_size":7205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45348108,"num_objects":5536,"num_object_clones":0,"num_object_copies":22144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7203,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6949","reported_seq":7012,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.682041+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:45.682041+0000","last_peered":"2024-09-16T04:45:45.682041+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:45.682041+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6949,"log_dups_size":0,"ondisk_log_size":6949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43933696,"num_objects":5363,"num_object_clones":0,"num_object_copies":21452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6949,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3164}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7086","reported_seq":7156,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.347203+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:45.347203+0000","last_peered":"2024-09-16T04:45:45.347203+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:45.347203+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7086,"log_dups_size":0,"ondisk_log_size":7086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45167884,"num_objects":5514,"num_object_clones":0,"num_object_copies":22056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7084,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3290}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7231","reported_seq":11787,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:47.551423+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:47.551423+0000","last_peered":"2024-09-16T04:45:47.551423+0000","last_clean":"2024-09-16T04:45:47.551423+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:47.551423+0000","last_undegraded":"2024-09-16T04:45:47.551423+0000","last_fullsized":"2024-09-16T04:45:47.551423+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7231,"log_dups_size":0,"ondisk_log_size":7231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:49.358775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45495564,"num_objects":5554,"num_object_clones":0,"num_object_copies":22216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7229,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7204","reported_seq":8990,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.641414+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:45.641414+0000","last_peered":"2024-09-16T04:45:45.641414+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:45.641414+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7204,"log_dups_size":0,"ondisk_log_size":7204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45476376,"num_objects":5552,"num_object_clones":0,"num_object_copies":22208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7201,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":867,"num_bytes_recovered":7077888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4076}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7169","reported_seq":7252,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.090216+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:45.090216+0000","last_peered":"2024-09-16T04:45:45.090216+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:45.090216+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7169,"log_dups_size":0,"ondisk_log_size":7169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45782284,"num_objects":5589,"num_object_clones":0,"num_object_copies":22356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7167,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7008","reported_seq":7085,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.632035+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:45.632035+0000","last_peered":"2024-09-16T04:45:45.632035+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:45.632035+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7008,"log_dups_size":0,"ondisk_log_size":7008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44912530,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7005,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6895","reported_seq":6950,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.670042+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:45.670042+0000","last_peered":"2024-09-16T04:45:45.670042+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:45.670042+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6895,"log_dups_size":0,"ondisk_log_size":6895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44530310,"num_objects":5436,"num_object_clones":0,"num_object_copies":21744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6894,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3315}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7024","reported_seq":7099,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.056548+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:45.056548+0000","last_peered":"2024-09-16T04:45:45.056548+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:45.056548+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7024,"log_dups_size":0,"ondisk_log_size":7024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45128326,"num_objects":5509,"num_object_clones":0,"num_object_copies":22036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7023,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3284}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7141","reported_seq":7228,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.600998+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:45.600998+0000","last_peered":"2024-09-16T04:45:45.600998+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:45.600998+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7141,"log_dups_size":0,"ondisk_log_size":7141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46208268,"num_objects":5641,"num_object_clones":0,"num_object_copies":22564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7139,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3423}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6983","reported_seq":7067,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.083075+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:45.083075+0000","last_peered":"2024-09-16T04:45:45.083075+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:45.083075+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6983,"log_dups_size":0,"ondisk_log_size":6983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44774668,"num_objects":5466,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6981,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3336}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6949","reported_seq":7021,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.519647+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:45.519647+0000","last_peered":"2024-09-16T04:45:45.519647+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:45.519647+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6949,"log_dups_size":0,"ondisk_log_size":6949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43889932,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6947,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3122}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":722208448,"num_objects":88166,"num_object_clones":0,"num_object_copies":352664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88166,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113321,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3209,"num_bytes_recovered":25411584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1356603392,"data_stored":1356603392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113348,"ondisk_log_size":113348,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746284,"kb_used_data":353604,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93625556,"statfs":{"total":96636764160,"available":95872569344,"internally_reserved":0,"allocated":362090496,"data_stored":361910527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":402058959},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,14,0,0,2,6,3,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741704,"kb_used_data":353712,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93630136,"statfs":{"total":96636764160,"available":95877259264,"internally_reserved":0,"allocated":362201088,"data_stored":362021119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,2,4,3,0,0,0,0,0,0,0,0,2],"upper_bound":131072},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051637,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547012,"kb_used_data":266804,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93824828,"statfs":{"total":96636764160,"available":96076623872,"internally_reserved":0,"allocated":273207296,"data_stored":273027327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":286912207},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754016,"kb_used_data":353104,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93617824,"statfs":{"total":96636764160,"available":95864651776,"internally_reserved":0,"allocated":361578496,"data_stored":361398527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":410513102},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":361152512,"data_stored":361152512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272891904,"data_stored":272891904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":361496576,"data_stored":361496576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":361062400,"data_stored":361062400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:49.958 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:50.556 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:50.556 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:50.581 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-16T04:45:48.648462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":722208448,"num_objects":88166,"num_object_clones":0,"num_object_copies":352664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88166,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113321,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3209,"num_bytes_recovered":25411584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113348,"ondisk_log_size":113348,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789016,"kb_used_data":1327224,"kb_used_omap":14,"kb_used_meta":1461681,"kb_avail":374698344,"statfs":{"total":386547056640,"available":383691104256,"internally_reserved":0,"allocated":1359077376,"data_stored":1358357500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1496761905},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,16,7,9,10,13,6,0,0,0,0,3,0,0,0,2],"upper_bound":131072},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21086208,"num_objects":-2574,"num_object_clones":0,"num_object_copies":-10296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":745472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005189"},"pg_stats":[{"pgid":"1.f","version":"33'7117","reported_seq":7182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.505042+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:45.505042+0000","last_peered":"2024-09-16T04:45:45.505042+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:45.505042+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7117,"log_dups_size":0,"ondisk_log_size":7117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45359104,"num_objects":5537,"num_object_clones":0,"num_object_copies":22148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7127","reported_seq":7200,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.537610+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:45.537610+0000","last_peered":"2024-09-16T04:45:45.537610+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:45.537610+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7127,"log_dups_size":0,"ondisk_log_size":7127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45438220,"num_objects":5547,"num_object_clones":0,"num_object_copies":22188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7125,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3339}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7111","reported_seq":7190,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.159459+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:45.159459+0000","last_peered":"2024-09-16T04:45:45.159459+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:45.159459+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7111,"log_dups_size":0,"ondisk_log_size":7111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45537950,"num_objects":5560,"num_object_clones":0,"num_object_copies":22240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5560,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7110,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3334}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7149","reported_seq":7220,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.536860+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:45.536860+0000","last_peered":"2024-09-16T04:45:45.536860+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:45.536860+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7149,"log_dups_size":0,"ondisk_log_size":7149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45225228,"num_objects":5521,"num_object_clones":0,"num_object_copies":22084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7147,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3294}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7205","reported_seq":7286,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.559449+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:45.559449+0000","last_peered":"2024-09-16T04:45:45.559449+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:45.559449+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7205,"log_dups_size":0,"ondisk_log_size":7205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45348108,"num_objects":5536,"num_object_clones":0,"num_object_copies":22144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7203,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6949","reported_seq":7012,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.682041+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:45.682041+0000","last_peered":"2024-09-16T04:45:45.682041+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:45.682041+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6949,"log_dups_size":0,"ondisk_log_size":6949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43933696,"num_objects":5363,"num_object_clones":0,"num_object_copies":21452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6949,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3164}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7086","reported_seq":7156,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.347203+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:45.347203+0000","last_peered":"2024-09-16T04:45:45.347203+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:45.347203+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7086,"log_dups_size":0,"ondisk_log_size":7086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45167884,"num_objects":5514,"num_object_clones":0,"num_object_copies":22056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7084,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3290}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7231","reported_seq":11787,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:47.551423+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:47.551423+0000","last_peered":"2024-09-16T04:45:47.551423+0000","last_clean":"2024-09-16T04:45:47.551423+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:47.551423+0000","last_undegraded":"2024-09-16T04:45:47.551423+0000","last_fullsized":"2024-09-16T04:45:47.551423+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7231,"log_dups_size":0,"ondisk_log_size":7231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:49.358775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45495564,"num_objects":5554,"num_object_clones":0,"num_object_copies":22216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7229,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7204","reported_seq":8990,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.641414+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:45.641414+0000","last_peered":"2024-09-16T04:45:45.641414+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:45.641414+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7204,"log_dups_size":0,"ondisk_log_size":7204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45476376,"num_objects":5552,"num_object_clones":0,"num_object_copies":22208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7201,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":867,"num_bytes_recovered":7077888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4076}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7169","reported_seq":7252,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.090216+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:45.090216+0000","last_peered":"2024-09-16T04:45:45.090216+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:45.090216+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7169,"log_dups_size":0,"ondisk_log_size":7169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45782284,"num_objects":5589,"num_object_clones":0,"num_object_copies":22356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7167,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7008","reported_seq":7085,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.632035+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:45.632035+0000","last_peered":"2024-09-16T04:45:45.632035+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:45.632035+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7008,"log_dups_size":0,"ondisk_log_size":7008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44912530,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7005,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6895","reported_seq":6950,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.670042+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:45.670042+0000","last_peered":"2024-09-16T04:45:45.670042+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:45.670042+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6895,"log_dups_size":0,"ondisk_log_size":6895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44530310,"num_objects":5436,"num_object_clones":0,"num_object_copies":21744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6894,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3315}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7024","reported_seq":7099,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.056548+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:45.056548+0000","last_peered":"2024-09-16T04:45:45.056548+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:45.056548+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7024,"log_dups_size":0,"ondisk_log_size":7024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45128326,"num_objects":5509,"num_object_clones":0,"num_object_copies":22036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7023,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3284}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7141","reported_seq":7228,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.600998+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:45.600998+0000","last_peered":"2024-09-16T04:45:45.600998+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:45.600998+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7141,"log_dups_size":0,"ondisk_log_size":7141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46208268,"num_objects":5641,"num_object_clones":0,"num_object_copies":22564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7139,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3423}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6983","reported_seq":7067,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.083075+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:45.083075+0000","last_peered":"2024-09-16T04:45:45.083075+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:45.083075+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6983,"log_dups_size":0,"ondisk_log_size":6983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44774668,"num_objects":5466,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6981,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3336}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6949","reported_seq":7021,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.519647+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:45.519647+0000","last_peered":"2024-09-16T04:45:45.519647+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:45.519647+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6949,"log_dups_size":0,"ondisk_log_size":6949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43889932,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6947,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3122}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":722208448,"num_objects":88166,"num_object_clones":0,"num_object_copies":352664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88166,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113321,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3209,"num_bytes_recovered":25411584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1356603392,"data_stored":1356603392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113348,"ondisk_log_size":113348,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746284,"kb_used_data":353604,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93625556,"statfs":{"total":96636764160,"available":95872569344,"internally_reserved":0,"allocated":362090496,"data_stored":361910527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":402058959},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,14,0,0,2,6,3,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741704,"kb_used_data":353712,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93630136,"statfs":{"total":96636764160,"available":95877259264,"internally_reserved":0,"allocated":362201088,"data_stored":362021119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,2,4,3,0,0,0,0,0,0,0,0,2],"upper_bound":131072},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051637,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547012,"kb_used_data":266804,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93824828,"statfs":{"total":96636764160,"available":96076623872,"internally_reserved":0,"allocated":273207296,"data_stored":273027327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":286912207},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754016,"kb_used_data":353104,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93617824,"statfs":{"total":96636764160,"available":95864651776,"internally_reserved":0,"allocated":361578496,"data_stored":361398527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":410513102},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":361152512,"data_stored":361152512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272891904,"data_stored":272891904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":361496576,"data_stored":361496576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":361062400,"data_stored":361062400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:50.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:45:50.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7117", 'reported_seq': 7182, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:45.505042+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:45:45.505042+0000', 'last_peered': '2024-09-16T04:45:45.505042+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:45:45.505042+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7117, 'log_dups_size': 0, 'ondisk_log_size': 7117, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45359104, 'num_objects': 5537, 'num_object_clones': 0, 'num_object_copies': 22148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2209, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5537, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7117, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:50.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:45:50.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7127", 'reported_seq': 7200, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:45.537610+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:45:45.537610+0000', 'last_peered': '2024-09-16T04:45:45.537610+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:45:45.537610+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7127, 'log_dups_size': 0, 'ondisk_log_size': 7127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45438220, 'num_objects': 5547, 'num_object_clones': 0, 'num_object_copies': 22188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2210, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5547, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7125, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3339}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:50.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:45:50.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7111", 'reported_seq': 7190, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:45.159459+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:45:45.159459+0000', 'last_peered': '2024-09-16T04:45:45.159459+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:45:45.159459+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7111, 'log_dups_size': 0, 'ondisk_log_size': 7111, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45537950, 'num_objects': 5560, 'num_object_clones': 0, 'num_object_copies': 22240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5560, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7110, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3334}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:50.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:45:50.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7149", 'reported_seq': 7220, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:45.536860+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:45:45.536860+0000', 'last_peered': '2024-09-16T04:45:45.536860+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:45:45.536860+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7149, 'log_dups_size': 0, 'ondisk_log_size': 7149, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45225228, 'num_objects': 5521, 'num_object_clones': 0, 'num_object_copies': 22084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7147, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3294}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:50.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:45:50.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7205", 'reported_seq': 7286, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:45.559449+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:45:45.559449+0000', 'last_peered': '2024-09-16T04:45:45.559449+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:45:45.559449+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7205, 'log_dups_size': 0, 'ondisk_log_size': 7205, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45348108, 'num_objects': 5536, 'num_object_clones': 0, 'num_object_copies': 22144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2273, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5536, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7203, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3265}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:50.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:45:50.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6949", 'reported_seq': 7012, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:45.682041+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:45:45.682041+0000', 'last_peered': '2024-09-16T04:45:45.682041+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:45:45.682041+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6949, 'log_dups_size': 0, 'ondisk_log_size': 6949, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43933696, 'num_objects': 5363, 'num_object_clones': 0, 'num_object_copies': 21452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2199, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6949, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3164}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:50.587 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:45:50.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7086", 'reported_seq': 7156, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:45.347203+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:45:45.347203+0000', 'last_peered': '2024-09-16T04:45:45.347203+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:45:45.347203+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7086, 'log_dups_size': 0, 'ondisk_log_size': 7086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45167884, 'num_objects': 5514, 'num_object_clones': 0, 'num_object_copies': 22056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5514, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7084, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3290}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:50.587 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:45:50.588 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7204", 'reported_seq': 8990, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:45.641414+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:45:45.641414+0000', 'last_peered': '2024-09-16T04:45:45.641414+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:45:45.641414+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7204, 'log_dups_size': 0, 'ondisk_log_size': 7204, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45476376, 'num_objects': 5552, 'num_object_clones': 0, 'num_object_copies': 22208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5552, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7201, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 867, 'num_bytes_recovered': 7077888, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4076}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:50.588 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:45:50.588 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7169", 'reported_seq': 7252, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:45.090216+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:45:45.090216+0000', 'last_peered': '2024-09-16T04:45:45.090216+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:45:45.090216+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7169, 'log_dups_size': 0, 'ondisk_log_size': 7169, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45782284, 'num_objects': 5589, 'num_object_clones': 0, 'num_object_copies': 22356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7167, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3391}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:50.589 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:45:50.589 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7008", 'reported_seq': 7085, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:45.632035+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:45:45.632035+0000', 'last_peered': '2024-09-16T04:45:45.632035+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:45:45.632035+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7008, 'log_dups_size': 0, 'ondisk_log_size': 7008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44912530, 'num_objects': 5483, 'num_object_clones': 0, 'num_object_copies': 21932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2183, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7005, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3303}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:50.589 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:45:50.589 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6895", 'reported_seq': 6950, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:45.670042+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:45:45.670042+0000', 'last_peered': '2024-09-16T04:45:45.670042+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:45:45.670042+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6895, 'log_dups_size': 0, 'ondisk_log_size': 6895, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44530310, 'num_objects': 5436, 'num_object_clones': 0, 'num_object_copies': 21744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2122, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5436, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6894, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3315}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:50.590 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:45:50.590 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7024", 'reported_seq': 7099, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:45.056548+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:45:45.056548+0000', 'last_peered': '2024-09-16T04:45:45.056548+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:45:45.056548+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7024, 'log_dups_size': 0, 'ondisk_log_size': 7024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45128326, 'num_objects': 5509, 'num_object_clones': 0, 'num_object_copies': 22036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7023, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3284}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:50.590 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:45:50.590 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7141", 'reported_seq': 7228, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:45.600998+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:45:45.600998+0000', 'last_peered': '2024-09-16T04:45:45.600998+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:45:45.600998+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7141, 'log_dups_size': 0, 'ondisk_log_size': 7141, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46208268, 'num_objects': 5641, 'num_object_clones': 0, 'num_object_copies': 22564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5641, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7139, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3423}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:50.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:45:50.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6983", 'reported_seq': 7067, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:45.083075+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:45:45.083075+0000', 'last_peered': '2024-09-16T04:45:45.083075+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:45:45.083075+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6983, 'log_dups_size': 0, 'ondisk_log_size': 6983, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44774668, 'num_objects': 5466, 'num_object_clones': 0, 'num_object_copies': 21864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6981, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3336}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:50.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:45:50.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6949", 'reported_seq': 7021, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:45.519647+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:45:45.519647+0000', 'last_peered': '2024-09-16T04:45:45.519647+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:45:45.519647+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6949, 'log_dups_size': 0, 'ondisk_log_size': 6949, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43889932, 'num_objects': 5358, 'num_object_clones': 0, 'num_object_copies': 21432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6947, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3122}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:50.592 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:45:51.342 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:51.342 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32368/350208 objects degraded (9.243%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":363,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":87552,"data_bytes":717178560,"bytes_used":2837274624,"bytes_avail":383709782016,"bytes_total":386547056640,"degraded_objects":32368,"degraded_total":350208,"degraded_ratio":0.092425073099415209,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":62095,"recovering_keys_per_sec":0,"num_objects_recovered":97,"num_bytes_recovered":745472,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":258},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:45:51.375 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:45:51.375 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:51.923 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:51.924 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:51.945 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-09-16T04:45:50.649147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":717178560,"num_objects":87552,"num_object_clones":0,"num_object_copies":350208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87552,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113935,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3211,"num_bytes_recovered":25411584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113962,"ondisk_log_size":113962,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2770776,"kb_used_data":1317312,"kb_used_omap":14,"kb_used_meta":1453361,"kb_avail":374716584,"statfs":{"total":386547056640,"available":383709782016,"internally_reserved":0,"allocated":1348927488,"data_stored":1348207612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1488242225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,14,11,6,10,12,3,0,0,0,0,3,0,0,0,2],"upper_bound":131072},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25436160,"num_objects":-3105,"num_object_clones":0,"num_object_copies":-12420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3105,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":745472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005202"},"pg_stats":[{"pgid":"1.f","version":"33'7117","reported_seq":7182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.505042+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:45.505042+0000","last_peered":"2024-09-16T04:45:45.505042+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:45.505042+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7117,"log_dups_size":0,"ondisk_log_size":7117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45359104,"num_objects":5537,"num_object_clones":0,"num_object_copies":22148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7200","reported_seq":7273,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.488385+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:50.488385+0000","last_peered":"2024-09-16T04:45:50.488385+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:50.488385+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44840204,"num_objects":5474,"num_object_clones":0,"num_object_copies":21896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7188","reported_seq":7266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.262288+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:50.262288+0000","last_peered":"2024-09-16T04:45:50.262288+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:50.262288+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7188,"log_dups_size":0,"ondisk_log_size":7188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44907166,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7187,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7228","reported_seq":7299,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.620040+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:50.620040+0000","last_peered":"2024-09-16T04:45:50.620040+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:50.620040+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7228,"log_dups_size":0,"ondisk_log_size":7228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44578060,"num_objects":5442,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7226,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3215}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7205","reported_seq":7286,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.559449+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:45.559449+0000","last_peered":"2024-09-16T04:45:45.559449+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:45.559449+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7205,"log_dups_size":0,"ondisk_log_size":7205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45348108,"num_objects":5536,"num_object_clones":0,"num_object_copies":22144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7203,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6949","reported_seq":7012,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.682041+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:45.682041+0000","last_peered":"2024-09-16T04:45:45.682041+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:45.682041+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6949,"log_dups_size":0,"ondisk_log_size":6949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43933696,"num_objects":5363,"num_object_clones":0,"num_object_copies":21452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6949,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3164}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7165","reported_seq":7235,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.350646+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:50.350646+0000","last_peered":"2024-09-16T04:45:50.350646+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:50.350646+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7165,"log_dups_size":0,"ondisk_log_size":7165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44520716,"num_objects":5435,"num_object_clones":0,"num_object_copies":21740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7163,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3211}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7231","reported_seq":11787,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:47.551423+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:47.551423+0000","last_peered":"2024-09-16T04:45:47.551423+0000","last_clean":"2024-09-16T04:45:47.551423+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:47.551423+0000","last_undegraded":"2024-09-16T04:45:47.551423+0000","last_fullsized":"2024-09-16T04:45:47.551423+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7231,"log_dups_size":0,"ondisk_log_size":7231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:49.358775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45495564,"num_objects":5554,"num_object_clones":0,"num_object_copies":22216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7229,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7204","reported_seq":8990,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.641414+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:45.641414+0000","last_peered":"2024-09-16T04:45:45.641414+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:45.641414+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7204,"log_dups_size":0,"ondisk_log_size":7204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45476376,"num_objects":5552,"num_object_clones":0,"num_object_copies":22208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7201,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":867,"num_bytes_recovered":7077888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4076}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7240","reported_seq":7326,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.257815+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:50.257815+0000","last_peered":"2024-09-16T04:45:50.257815+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:50.257815+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7240,"log_dups_size":0,"ondisk_log_size":7240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45200652,"num_objects":5518,"num_object_clones":0,"num_object_copies":22072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7238,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3322}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7008","reported_seq":7085,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.632035+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:45.632035+0000","last_peered":"2024-09-16T04:45:45.632035+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:45.632035+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7008,"log_dups_size":0,"ondisk_log_size":7008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44912530,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7005,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6895","reported_seq":6950,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.670042+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:45.670042+0000","last_peered":"2024-09-16T04:45:45.670042+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:45.670042+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6895,"log_dups_size":0,"ondisk_log_size":6895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44530310,"num_objects":5436,"num_object_clones":0,"num_object_copies":21744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6894,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3315}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7100","reported_seq":7175,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.151248+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:50.151248+0000","last_peered":"2024-09-16T04:45:50.151248+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:50.151248+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7100,"log_dups_size":0,"ondisk_log_size":7100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44505734,"num_objects":5433,"num_object_clones":0,"num_object_copies":21732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7099,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3208}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7141","reported_seq":7228,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:45.600998+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:45.600998+0000","last_peered":"2024-09-16T04:45:45.600998+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:45.600998+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7141,"log_dups_size":0,"ondisk_log_size":7141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46208268,"num_objects":5641,"num_object_clones":0,"num_object_copies":22564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7139,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3423}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7060","reported_seq":7144,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.172968+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:50.172968+0000","last_peered":"2024-09-16T04:45:50.172968+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:50.172968+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7060,"log_dups_size":0,"ondisk_log_size":7060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44143884,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7058,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7031","reported_seq":7104,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.598947+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:50.598947+0000","last_peered":"2024-09-16T04:45:50.598947+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:50.598947+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7031,"log_dups_size":0,"ondisk_log_size":7031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43218188,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7029,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3040}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":717178560,"num_objects":87552,"num_object_clones":0,"num_object_copies":350208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87552,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113935,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3211,"num_bytes_recovered":25411584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1346330624,"data_stored":1346330624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113962,"ondisk_log_size":113962,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746284,"kb_used_data":353604,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93625556,"statfs":{"total":96636764160,"available":95872569344,"internally_reserved":0,"allocated":362090496,"data_stored":361910527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":402058959},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,14,0,0,2,6,3,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740828,"kb_used_data":348684,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93631012,"statfs":{"total":96636764160,"available":95878156288,"internally_reserved":0,"allocated":357052416,"data_stored":356872447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,3,1,0,0,0,0,0,0,0,0,0,2],"upper_bound":131072},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051637,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547012,"kb_used_data":266804,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93824828,"statfs":{"total":96636764160,"available":96076623872,"internally_reserved":0,"allocated":273207296,"data_stored":273027327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":286912207},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736652,"kb_used_data":348220,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93635188,"statfs":{"total":96636764160,"available":95882432512,"internally_reserved":0,"allocated":356577280,"data_stored":356397311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":397733582},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356028416,"data_stored":356028416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272891904,"data_stored":272891904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":356347904,"data_stored":356347904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":361062400,"data_stored":361062400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:54.947 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:55.501 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:55.501 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:55.528 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-16T04:45:54.653434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":711886528,"num_objects":86906,"num_object_clones":0,"num_object_copies":347624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86906,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114581,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3259,"num_bytes_recovered":25772032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114608,"ondisk_log_size":114608,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773688,"kb_used_data":1311924,"kb_used_omap":14,"kb_used_meta":1461681,"kb_avail":374713672,"statfs":{"total":386547056640,"available":383706800128,"internally_reserved":0,"allocated":1343410176,"data_stored":1342690300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1496761905},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,8,7,10,11,3,0,0,0,0,1,0,0,0,2],"upper_bound":131072},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20455424,"num_objects":-2497,"num_object_clones":0,"num_object_copies":-9988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2497,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006741"},"pg_stats":[{"pgid":"1.f","version":"33'7196","reported_seq":7263,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.593466+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:50.593466+0000","last_peered":"2024-09-16T04:45:50.593466+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:50.593466+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7196,"log_dups_size":0,"ondisk_log_size":7196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44711936,"num_objects":5458,"num_object_clones":0,"num_object_copies":21832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7196,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7200","reported_seq":7273,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.488385+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:50.488385+0000","last_peered":"2024-09-16T04:45:50.488385+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:50.488385+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44840204,"num_objects":5474,"num_object_clones":0,"num_object_copies":21896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7188","reported_seq":7266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.262288+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:50.262288+0000","last_peered":"2024-09-16T04:45:50.262288+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:50.262288+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7188,"log_dups_size":0,"ondisk_log_size":7188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44907166,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7187,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7228","reported_seq":7299,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.620040+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:50.620040+0000","last_peered":"2024-09-16T04:45:50.620040+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:50.620040+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7228,"log_dups_size":0,"ondisk_log_size":7228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44578060,"num_objects":5442,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7226,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3215}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7297","reported_seq":7378,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.533026+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:50.533026+0000","last_peered":"2024-09-16T04:45:50.533026+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:50.533026+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7297,"log_dups_size":0,"ondisk_log_size":7297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44594444,"num_objects":5444,"num_object_clones":0,"num_object_copies":21776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7295,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7014","reported_seq":7080,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.737921+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:50.737921+0000","last_peered":"2024-09-16T04:45:50.737921+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:50.737921+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7014,"log_dups_size":0,"ondisk_log_size":7014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43401216,"num_objects":5298,"num_object_clones":0,"num_object_copies":21192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7014,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3101}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7165","reported_seq":7235,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.350646+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:50.350646+0000","last_peered":"2024-09-16T04:45:50.350646+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:50.350646+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7165,"log_dups_size":0,"ondisk_log_size":7165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44520716,"num_objects":5435,"num_object_clones":0,"num_object_copies":21740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7163,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3211}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7306","reported_seq":11862,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:52.555522+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:52.555522+0000","last_peered":"2024-09-16T04:45:52.555522+0000","last_clean":"2024-09-16T04:45:52.555522+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:52.555522+0000","last_undegraded":"2024-09-16T04:45:52.555522+0000","last_fullsized":"2024-09-16T04:45:52.555522+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:54.361783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44881164,"num_objects":5479,"num_object_clones":0,"num_object_copies":21916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7290","reported_seq":9165,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.635361+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:50.635361+0000","last_peered":"2024-09-16T04:45:50.635361+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:50.635361+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7290,"log_dups_size":0,"ondisk_log_size":7290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44771864,"num_objects":5466,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7287,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":911,"num_bytes_recovered":7438336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4034}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7240","reported_seq":7326,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.257815+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:50.257815+0000","last_peered":"2024-09-16T04:45:50.257815+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:50.257815+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7240,"log_dups_size":0,"ondisk_log_size":7240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45200652,"num_objects":5518,"num_object_clones":0,"num_object_copies":22072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7238,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3322}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7096","reported_seq":7173,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.656856+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:50.656856+0000","last_peered":"2024-09-16T04:45:50.656856+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:50.656856+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7096,"log_dups_size":0,"ondisk_log_size":7096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44191634,"num_objects":5395,"num_object_clones":0,"num_object_copies":21580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7093,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3215}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6972","reported_seq":7029,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.660245+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:50.660245+0000","last_peered":"2024-09-16T04:45:50.660245+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:50.660245+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6972,"log_dups_size":0,"ondisk_log_size":6972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43899526,"num_objects":5359,"num_object_clones":0,"num_object_copies":21436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6971,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3239}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7100","reported_seq":7175,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.151248+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:50.151248+0000","last_peered":"2024-09-16T04:45:50.151248+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:50.151248+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7100,"log_dups_size":0,"ondisk_log_size":7100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44505734,"num_objects":5433,"num_object_clones":0,"num_object_copies":21732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7099,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3208}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7225","reported_seq":7311,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.678040+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:50.678040+0000","last_peered":"2024-09-16T04:45:50.678040+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:50.678040+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7225,"log_dups_size":0,"ondisk_log_size":7225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45520140,"num_objects":5557,"num_object_clones":0,"num_object_copies":22228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7223,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3339}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7060","reported_seq":7144,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.172968+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:50.172968+0000","last_peered":"2024-09-16T04:45:50.172968+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:50.172968+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7060,"log_dups_size":0,"ondisk_log_size":7060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44143884,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7058,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7031","reported_seq":7104,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.598947+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:50.598947+0000","last_peered":"2024-09-16T04:45:50.598947+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:50.598947+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7031,"log_dups_size":0,"ondisk_log_size":7031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43218188,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7029,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3040}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":711886528,"num_objects":86906,"num_object_clones":0,"num_object_copies":347624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86906,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114581,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3259,"num_bytes_recovered":25772032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1340805120,"data_stored":1340805120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114608,"ondisk_log_size":114608,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745412,"kb_used_data":348596,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93626428,"statfs":{"total":96636764160,"available":95873462272,"internally_reserved":0,"allocated":356962304,"data_stored":356782335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406318799},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,0,0,6,3,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740828,"kb_used_data":348684,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93631012,"statfs":{"total":96636764160,"available":95878156288,"internally_reserved":0,"allocated":357052416,"data_stored":356872447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,3,1,0,0,0,0,0,0,0,0,0,2],"upper_bound":131072},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051638,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550796,"kb_used_data":266424,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93821044,"statfs":{"total":96636764160,"available":96072749056,"internally_reserved":0,"allocated":272818176,"data_stored":272638207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291172047},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736652,"kb_used_data":348220,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93635188,"statfs":{"total":96636764160,"available":95882432512,"internally_reserved":0,"allocated":356577280,"data_stored":356397311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":397733582},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356028416,"data_stored":356028416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272510976,"data_stored":272510976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":356347904,"data_stored":356347904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":355917824,"data_stored":355917824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:55.529 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:56.120 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:56.122 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:56.144 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-16T04:45:54.653434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":711886528,"num_objects":86906,"num_object_clones":0,"num_object_copies":347624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86906,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114581,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3259,"num_bytes_recovered":25772032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114608,"ondisk_log_size":114608,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773688,"kb_used_data":1311924,"kb_used_omap":14,"kb_used_meta":1461681,"kb_avail":374713672,"statfs":{"total":386547056640,"available":383706800128,"internally_reserved":0,"allocated":1343410176,"data_stored":1342690300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1496761905},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,8,7,10,11,3,0,0,0,0,1,0,0,0,2],"upper_bound":131072},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20455424,"num_objects":-2497,"num_object_clones":0,"num_object_copies":-9988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2497,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006741"},"pg_stats":[{"pgid":"1.f","version":"33'7196","reported_seq":7263,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.593466+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:50.593466+0000","last_peered":"2024-09-16T04:45:50.593466+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:50.593466+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7196,"log_dups_size":0,"ondisk_log_size":7196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44711936,"num_objects":5458,"num_object_clones":0,"num_object_copies":21832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7196,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7200","reported_seq":7273,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.488385+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:50.488385+0000","last_peered":"2024-09-16T04:45:50.488385+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:50.488385+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44840204,"num_objects":5474,"num_object_clones":0,"num_object_copies":21896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7188","reported_seq":7266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.262288+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:50.262288+0000","last_peered":"2024-09-16T04:45:50.262288+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:50.262288+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7188,"log_dups_size":0,"ondisk_log_size":7188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44907166,"num_objects":5483,"num_object_clones":0,"num_object_copies":21932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7187,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7228","reported_seq":7299,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.620040+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:50.620040+0000","last_peered":"2024-09-16T04:45:50.620040+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:50.620040+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7228,"log_dups_size":0,"ondisk_log_size":7228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44578060,"num_objects":5442,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7226,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3215}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7297","reported_seq":7378,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.533026+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:50.533026+0000","last_peered":"2024-09-16T04:45:50.533026+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:50.533026+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7297,"log_dups_size":0,"ondisk_log_size":7297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44594444,"num_objects":5444,"num_object_clones":0,"num_object_copies":21776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7295,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7014","reported_seq":7080,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.737921+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:50.737921+0000","last_peered":"2024-09-16T04:45:50.737921+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:50.737921+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7014,"log_dups_size":0,"ondisk_log_size":7014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43401216,"num_objects":5298,"num_object_clones":0,"num_object_copies":21192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7014,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3101}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7165","reported_seq":7235,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.350646+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:50.350646+0000","last_peered":"2024-09-16T04:45:50.350646+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:50.350646+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7165,"log_dups_size":0,"ondisk_log_size":7165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44520716,"num_objects":5435,"num_object_clones":0,"num_object_copies":21740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7163,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3211}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7306","reported_seq":11862,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:52.555522+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:52.555522+0000","last_peered":"2024-09-16T04:45:52.555522+0000","last_clean":"2024-09-16T04:45:52.555522+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:52.555522+0000","last_undegraded":"2024-09-16T04:45:52.555522+0000","last_fullsized":"2024-09-16T04:45:52.555522+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:54.361783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44881164,"num_objects":5479,"num_object_clones":0,"num_object_copies":21916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7290","reported_seq":9165,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.635361+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:50.635361+0000","last_peered":"2024-09-16T04:45:50.635361+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:50.635361+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7290,"log_dups_size":0,"ondisk_log_size":7290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44771864,"num_objects":5466,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7287,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":911,"num_bytes_recovered":7438336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4034}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7240","reported_seq":7326,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.257815+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:50.257815+0000","last_peered":"2024-09-16T04:45:50.257815+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:50.257815+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7240,"log_dups_size":0,"ondisk_log_size":7240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45200652,"num_objects":5518,"num_object_clones":0,"num_object_copies":22072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7238,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3322}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7096","reported_seq":7173,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.656856+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:50.656856+0000","last_peered":"2024-09-16T04:45:50.656856+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:50.656856+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7096,"log_dups_size":0,"ondisk_log_size":7096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44191634,"num_objects":5395,"num_object_clones":0,"num_object_copies":21580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7093,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3215}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6972","reported_seq":7029,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.660245+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:50.660245+0000","last_peered":"2024-09-16T04:45:50.660245+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:50.660245+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":6972,"log_dups_size":0,"ondisk_log_size":6972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43899526,"num_objects":5359,"num_object_clones":0,"num_object_copies":21436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6971,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3239}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7100","reported_seq":7175,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.151248+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:50.151248+0000","last_peered":"2024-09-16T04:45:50.151248+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:50.151248+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7100,"log_dups_size":0,"ondisk_log_size":7100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44505734,"num_objects":5433,"num_object_clones":0,"num_object_copies":21732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7099,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3208}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7225","reported_seq":7311,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.678040+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:50.678040+0000","last_peered":"2024-09-16T04:45:50.678040+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:50.678040+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7225,"log_dups_size":0,"ondisk_log_size":7225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45520140,"num_objects":5557,"num_object_clones":0,"num_object_copies":22228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7223,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3339}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7060","reported_seq":7144,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.172968+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:50.172968+0000","last_peered":"2024-09-16T04:45:50.172968+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:50.172968+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7060,"log_dups_size":0,"ondisk_log_size":7060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44143884,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7058,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3259}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7031","reported_seq":7104,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:50.598947+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:50.598947+0000","last_peered":"2024-09-16T04:45:50.598947+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:50.598947+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7031,"log_dups_size":0,"ondisk_log_size":7031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43218188,"num_objects":5276,"num_object_clones":0,"num_object_copies":21104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7029,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3040}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":711886528,"num_objects":86906,"num_object_clones":0,"num_object_copies":347624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86906,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114581,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3259,"num_bytes_recovered":25772032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1340805120,"data_stored":1340805120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114608,"ondisk_log_size":114608,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745412,"kb_used_data":348596,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93626428,"statfs":{"total":96636764160,"available":95873462272,"internally_reserved":0,"allocated":356962304,"data_stored":356782335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":406318799},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,0,0,6,3,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740828,"kb_used_data":348684,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93631012,"statfs":{"total":96636764160,"available":95878156288,"internally_reserved":0,"allocated":357052416,"data_stored":356872447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,3,1,0,0,0,0,0,0,0,0,0,2],"upper_bound":131072},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051638,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550796,"kb_used_data":266424,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93821044,"statfs":{"total":96636764160,"available":96072749056,"internally_reserved":0,"allocated":272818176,"data_stored":272638207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291172047},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736652,"kb_used_data":348220,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93635188,"statfs":{"total":96636764160,"available":95882432512,"internally_reserved":0,"allocated":356577280,"data_stored":356397311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":397733582},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356028416,"data_stored":356028416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272510976,"data_stored":272510976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":356347904,"data_stored":356347904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":355917824,"data_stored":355917824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:45:56.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:45:56.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7196", 'reported_seq': 7263, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:50.593466+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:45:50.593466+0000', 'last_peered': '2024-09-16T04:45:50.593466+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:45:50.593466+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7196, 'log_dups_size': 0, 'ondisk_log_size': 7196, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44711936, 'num_objects': 5458, 'num_object_clones': 0, 'num_object_copies': 21832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5458, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7196, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3250}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:56.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:45:56.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7200", 'reported_seq': 7273, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:50.488385+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:45:50.488385+0000', 'last_peered': '2024-09-16T04:45:50.488385+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:45:50.488385+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7200, 'log_dups_size': 0, 'ondisk_log_size': 7200, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44840204, 'num_objects': 5474, 'num_object_clones': 0, 'num_object_copies': 21896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2210, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7198, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3266}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:56.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:45:56.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7188", 'reported_seq': 7266, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:50.262288+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:45:50.262288+0000', 'last_peered': '2024-09-16T04:45:50.262288+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:45:50.262288+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7188, 'log_dups_size': 0, 'ondisk_log_size': 7188, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44907166, 'num_objects': 5483, 'num_object_clones': 0, 'num_object_copies': 21932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2227, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5483, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7187, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:56.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:45:56.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7228", 'reported_seq': 7299, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:50.620040+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:45:50.620040+0000', 'last_peered': '2024-09-16T04:45:50.620040+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:45:50.620040+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7228, 'log_dups_size': 0, 'ondisk_log_size': 7228, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44578060, 'num_objects': 5442, 'num_object_clones': 0, 'num_object_copies': 21768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7226, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3215}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:56.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:45:56.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7297", 'reported_seq': 7378, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:50.533026+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:45:50.533026+0000', 'last_peered': '2024-09-16T04:45:50.533026+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:45:50.533026+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7297, 'log_dups_size': 0, 'ondisk_log_size': 7297, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44594444, 'num_objects': 5444, 'num_object_clones': 0, 'num_object_copies': 21776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2273, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5444, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7295, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3173}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:56.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:45:56.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7014", 'reported_seq': 7080, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:50.737921+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:45:50.737921+0000', 'last_peered': '2024-09-16T04:45:50.737921+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:45:50.737921+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7014, 'log_dups_size': 0, 'ondisk_log_size': 7014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43401216, 'num_objects': 5298, 'num_object_clones': 0, 'num_object_copies': 21192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2197, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5298, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7014, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3101}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:56.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:45:56.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7165", 'reported_seq': 7235, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:50.350646+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:45:50.350646+0000', 'last_peered': '2024-09-16T04:45:50.350646+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:45:50.350646+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7165, 'log_dups_size': 0, 'ondisk_log_size': 7165, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44520716, 'num_objects': 5435, 'num_object_clones': 0, 'num_object_copies': 21740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5435, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7163, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3211}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:56.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:45:56.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7290", 'reported_seq': 9165, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:50.635361+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:45:50.635361+0000', 'last_peered': '2024-09-16T04:45:50.635361+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:45:50.635361+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7290, 'log_dups_size': 0, 'ondisk_log_size': 7290, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44771864, 'num_objects': 5466, 'num_object_clones': 0, 'num_object_copies': 21864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1432, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7287, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 911, 'num_bytes_recovered': 7438336, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4034}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:56.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:45:56.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7240", 'reported_seq': 7326, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:50.257815+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:45:50.257815+0000', 'last_peered': '2024-09-16T04:45:50.257815+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:45:50.257815+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7240, 'log_dups_size': 0, 'ondisk_log_size': 7240, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45200652, 'num_objects': 5518, 'num_object_clones': 0, 'num_object_copies': 22072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2198, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7238, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3322}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:56.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:45:56.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7096", 'reported_seq': 7173, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:50.656856+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:45:50.656856+0000', 'last_peered': '2024-09-16T04:45:50.656856+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:45:50.656856+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7096, 'log_dups_size': 0, 'ondisk_log_size': 7096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44191634, 'num_objects': 5395, 'num_object_clones': 0, 'num_object_copies': 21580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2183, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7093, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3215}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:56.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:45:56.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6972", 'reported_seq': 7029, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:50.660245+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:45:50.660245+0000', 'last_peered': '2024-09-16T04:45:50.660245+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:45:50.660245+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 6972, 'log_dups_size': 0, 'ondisk_log_size': 6972, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43899526, 'num_objects': 5359, 'num_object_clones': 0, 'num_object_copies': 21436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2121, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6971, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3239}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:56.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:45:56.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7100", 'reported_seq': 7175, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:50.151248+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:45:50.151248+0000', 'last_peered': '2024-09-16T04:45:50.151248+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:45:50.151248+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7100, 'log_dups_size': 0, 'ondisk_log_size': 7100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44505734, 'num_objects': 5433, 'num_object_clones': 0, 'num_object_copies': 21732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5433, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7099, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3208}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:56.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:45:56.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7225", 'reported_seq': 7311, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:50.678040+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:45:50.678040+0000', 'last_peered': '2024-09-16T04:45:50.678040+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:45:50.678040+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7225, 'log_dups_size': 0, 'ondisk_log_size': 7225, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45520140, 'num_objects': 5557, 'num_object_clones': 0, 'num_object_copies': 22228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7223, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3339}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:45:56.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:45:56.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7060", 'reported_seq': 7144, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:50.172968+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:45:50.172968+0000', 'last_peered': '2024-09-16T04:45:50.172968+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:45:50.172968+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7060, 'log_dups_size': 0, 'ondisk_log_size': 7060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44143884, 'num_objects': 5389, 'num_object_clones': 0, 'num_object_copies': 21556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7058, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3259}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:45:56.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:45:56.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7031", 'reported_seq': 7104, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:50.598947+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:45:50.598947+0000', 'last_peered': '2024-09-16T04:45:50.598947+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:45:50.598947+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7031, 'log_dups_size': 0, 'ondisk_log_size': 7031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43218188, 'num_objects': 5276, 'num_object_clones': 0, 'num_object_copies': 21104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7029, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3040}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:45:56.148 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:45:56.878 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:56.878 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32320/347624 objects degraded (9.297%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":369,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":86906,"data_bytes":711886528,"bytes_used":2840256512,"bytes_avail":383706800128,"bytes_total":386547056640,"degraded_objects":32320,"degraded_total":347624,"degraded_ratio":0.092974017904402462,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":62770,"recovering_keys_per_sec":0,"num_objects_recovered":102,"num_bytes_recovered":753664,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":207},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:45:56.909 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:45:56.910 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:45:57.519 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:45:57.519 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:45:57.541 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-09-16T04:45:56.653946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":703301312,"num_objects":85858,"num_object_clones":0,"num_object_copies":343432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85858,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115629,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3338,"num_bytes_recovered":26386432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115656,"ondisk_log_size":115656,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772300,"kb_used_data":1298028,"kb_used_omap":14,"kb_used_meta":1474161,"kb_avail":374715060,"statfs":{"total":386547056640,"available":383708221440,"internally_reserved":0,"allocated":1329180672,"data_stored":1328460796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1509541425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,10,17,1,1,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29040640,"num_objects":-3545,"num_object_clones":0,"num_object_copies":-14180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":1368064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006533"},"pg_stats":[{"pgid":"1.f","version":"33'7265","reported_seq":7332,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.691589+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:55.691589+0000","last_peered":"2024-09-16T04:45:55.691589+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:55.691589+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7265,"log_dups_size":0,"ondisk_log_size":7265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44146688,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7265,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7268","reported_seq":7341,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.516927+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:55.516927+0000","last_peered":"2024-09-16T04:45:55.516927+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:55.516927+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7268,"log_dups_size":0,"ondisk_log_size":7268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44283148,"num_objects":5406,"num_object_clones":0,"num_object_copies":21624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7266,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7245","reported_seq":7324,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:54.828941+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:45:54.828941+0000","last_peered":"2024-09-16T04:45:54.828941+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:45:54.828941+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7245,"log_dups_size":0,"ondisk_log_size":7245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44440222,"num_objects":5426,"num_object_clones":0,"num_object_copies":21704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5426,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7244,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7286","reported_seq":7357,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.413307+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:55.413307+0000","last_peered":"2024-09-16T04:45:55.413307+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:55.413307+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7286,"log_dups_size":0,"ondisk_log_size":7286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44102924,"num_objects":5384,"num_object_clones":0,"num_object_copies":21536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7284,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7366","reported_seq":7451,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.689170+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:55.689170+0000","last_peered":"2024-09-16T04:45:55.689170+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:55.689170+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7366,"log_dups_size":0,"ondisk_log_size":7366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44029196,"num_objects":5375,"num_object_clones":0,"num_object_copies":21500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7364,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7095","reported_seq":7161,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.629173+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:55.629173+0000","last_peered":"2024-09-16T04:45:55.629173+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:55.629173+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7095,"log_dups_size":0,"ondisk_log_size":7095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42737664,"num_objects":5217,"num_object_clones":0,"num_object_copies":20868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7095,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7244","reported_seq":7314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.594560+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:55.594560+0000","last_peered":"2024-09-16T04:45:55.594560+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:55.594560+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7244,"log_dups_size":0,"ondisk_log_size":7244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43873548,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7242,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3132}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7306","reported_seq":11862,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:52.555522+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:52.555522+0000","last_peered":"2024-09-16T04:45:52.555522+0000","last_clean":"2024-09-16T04:45:52.555522+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:52.555522+0000","last_undegraded":"2024-09-16T04:45:52.555522+0000","last_fullsized":"2024-09-16T04:45:52.555522+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:54.361783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44881164,"num_objects":5479,"num_object_clones":0,"num_object_copies":21916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7376","reported_seq":9401,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.765339+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:55.765339+0000","last_peered":"2024-09-16T04:45:55.765339+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:55.765339+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7376,"log_dups_size":0,"ondisk_log_size":7376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44067352,"num_objects":5380,"num_object_clones":0,"num_object_copies":21520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7373,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":986,"num_bytes_recovered":8052736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4023}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7300","reported_seq":7387,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.222074+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:45:55.222074+0000","last_peered":"2024-09-16T04:45:55.222074+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:45:55.222074+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7300,"log_dups_size":0,"ondisk_log_size":7300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44709132,"num_objects":5458,"num_object_clones":0,"num_object_copies":21832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7298,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3262}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7168","reported_seq":7247,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.779981+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:55.779981+0000","last_peered":"2024-09-16T04:45:55.779981+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:55.779981+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7168,"log_dups_size":0,"ondisk_log_size":7168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43601810,"num_objects":5323,"num_object_clones":0,"num_object_copies":21292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7165,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3144}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7041","reported_seq":7098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.591117+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:55.591117+0000","last_peered":"2024-09-16T04:45:55.591117+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:55.591117+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7041,"log_dups_size":0,"ondisk_log_size":7041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43334278,"num_objects":5290,"num_object_clones":0,"num_object_copies":21160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7040,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3170}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7163","reported_seq":7238,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.245159+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:45:55.245159+0000","last_peered":"2024-09-16T04:45:55.245159+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:45:55.245159+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7163,"log_dups_size":0,"ondisk_log_size":7163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43989638,"num_objects":5370,"num_object_clones":0,"num_object_copies":21480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7162,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3145}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7307","reported_seq":7396,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.748539+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:55.748539+0000","last_peered":"2024-09-16T04:45:55.748539+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:55.748539+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7307,"log_dups_size":0,"ondisk_log_size":7307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44848396,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7305,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3258}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7133","reported_seq":7217,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.014789+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:45:55.014789+0000","last_peered":"2024-09-16T04:45:55.014789+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:45:55.014789+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7133,"log_dups_size":0,"ondisk_log_size":7133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43545868,"num_objects":5316,"num_object_clones":0,"num_object_copies":21264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7131,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3186}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7093","reported_seq":7166,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.515982+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:55.515982+0000","last_peered":"2024-09-16T04:45:55.515982+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:55.515982+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7093,"log_dups_size":0,"ondisk_log_size":7093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710284,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7091,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2978}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":703301312,"num_objects":85858,"num_object_clones":0,"num_object_copies":343432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85858,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115629,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3338,"num_bytes_recovered":26386432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1326718976,"data_stored":1326718976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115656,"ondisk_log_size":115656,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745008,"kb_used_data":344012,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93626832,"statfs":{"total":96636764160,"available":95873875968,"internally_reserved":0,"allocated":352268288,"data_stored":352088319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410578639},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,6,0,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740280,"kb_used_data":343972,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93631560,"statfs":{"total":96636764160,"available":95878717440,"internally_reserved":0,"allocated":352227328,"data_stored":352047359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,2,5,0,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051638,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550796,"kb_used_data":266424,"kb_used_omap":4,"kb_used_meta":284347,"kb_avail":93821044,"statfs":{"total":96636764160,"available":96072749056,"internally_reserved":0,"allocated":272818176,"data_stored":272638207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291172047},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736216,"kb_used_data":343620,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93635624,"statfs":{"total":96636764160,"available":95882878976,"internally_reserved":0,"allocated":351866880,"data_stored":351686911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":401993422},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":351350784,"data_stored":351350784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272510976,"data_stored":272510976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351633408,"data_stored":351633408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":351223808,"data_stored":351223808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:00.543 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:01.157 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:01.158 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:01.184 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-09-16T04:46:00.655016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":700204736,"num_objects":85480,"num_object_clones":0,"num_object_copies":341920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116007,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3344,"num_bytes_recovered":26386432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116034,"ondisk_log_size":116034,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2775888,"kb_used_data":1293304,"kb_used_omap":14,"kb_used_meta":1482481,"kb_avail":374711472,"statfs":{"total":386547056640,"available":383704547328,"internally_reserved":0,"allocated":1324343296,"data_stored":1323623420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1518061105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,1,6,14,12,3,1,0,0,0,0,0,1,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22003712,"num_objects":-2686,"num_object_clones":0,"num_object_copies":-10744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2686,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006554"},"pg_stats":[{"pgid":"1.f","version":"33'7265","reported_seq":7332,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.691589+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:55.691589+0000","last_peered":"2024-09-16T04:45:55.691589+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:55.691589+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7265,"log_dups_size":0,"ondisk_log_size":7265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44146688,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7265,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7268","reported_seq":7341,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.516927+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:55.516927+0000","last_peered":"2024-09-16T04:45:55.516927+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:55.516927+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7268,"log_dups_size":0,"ondisk_log_size":7268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44283148,"num_objects":5406,"num_object_clones":0,"num_object_copies":21624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7266,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7313","reported_seq":7396,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.329004+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:00.329004+0000","last_peered":"2024-09-16T04:46:00.329004+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:00.329004+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7313,"log_dups_size":0,"ondisk_log_size":7313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43883166,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7312,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7286","reported_seq":7357,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.413307+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:55.413307+0000","last_peered":"2024-09-16T04:45:55.413307+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:55.413307+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7286,"log_dups_size":0,"ondisk_log_size":7286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44102924,"num_objects":5384,"num_object_clones":0,"num_object_copies":21536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7284,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7366","reported_seq":7451,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.689170+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:55.689170+0000","last_peered":"2024-09-16T04:45:55.689170+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:55.689170+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7366,"log_dups_size":0,"ondisk_log_size":7366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44029196,"num_objects":5375,"num_object_clones":0,"num_object_copies":21500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7364,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7095","reported_seq":7161,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.629173+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:55.629173+0000","last_peered":"2024-09-16T04:45:55.629173+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:55.629173+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7095,"log_dups_size":0,"ondisk_log_size":7095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42737664,"num_objects":5217,"num_object_clones":0,"num_object_copies":20868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7095,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7244","reported_seq":7314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.594560+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:55.594560+0000","last_peered":"2024-09-16T04:45:55.594560+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:55.594560+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7244,"log_dups_size":0,"ondisk_log_size":7244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43873548,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7242,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3132}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7384","reported_seq":11940,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:57.585669+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:57.585669+0000","last_peered":"2024-09-16T04:45:57.585669+0000","last_clean":"2024-09-16T04:45:57.585669+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:57.585669+0000","last_undegraded":"2024-09-16T04:45:57.585669+0000","last_fullsized":"2024-09-16T04:45:57.585669+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7384,"log_dups_size":0,"ondisk_log_size":7384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:00.364801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44242188,"num_objects":5401,"num_object_clones":0,"num_object_copies":21604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7382,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7376","reported_seq":9401,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.765339+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:55.765339+0000","last_peered":"2024-09-16T04:45:55.765339+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:55.765339+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7376,"log_dups_size":0,"ondisk_log_size":7376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44067352,"num_objects":5380,"num_object_clones":0,"num_object_copies":21520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7373,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":986,"num_bytes_recovered":8052736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4023}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7375","reported_seq":7468,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.320109+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:00.320109+0000","last_peered":"2024-09-16T04:46:00.320109+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:00.320109+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7375,"log_dups_size":0,"ondisk_log_size":7375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44094732,"num_objects":5383,"num_object_clones":0,"num_object_copies":21532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7373,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3190}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7168","reported_seq":7247,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.779981+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:55.779981+0000","last_peered":"2024-09-16T04:45:55.779981+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:55.779981+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7168,"log_dups_size":0,"ondisk_log_size":7168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43601810,"num_objects":5323,"num_object_clones":0,"num_object_copies":21292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7165,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3144}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7041","reported_seq":7098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.591117+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:55.591117+0000","last_peered":"2024-09-16T04:45:55.591117+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:55.591117+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7041,"log_dups_size":0,"ondisk_log_size":7041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43334278,"num_objects":5290,"num_object_clones":0,"num_object_copies":21160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7040,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3170}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7242","reported_seq":7319,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.306982+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:00.306982+0000","last_peered":"2024-09-16T04:46:00.306982+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:00.306982+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7242,"log_dups_size":0,"ondisk_log_size":7242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43342470,"num_objects":5291,"num_object_clones":0,"num_object_copies":21164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7241,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3067}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7307","reported_seq":7396,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.748539+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:55.748539+0000","last_peered":"2024-09-16T04:45:55.748539+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:55.748539+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7307,"log_dups_size":0,"ondisk_log_size":7307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44848396,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7305,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3258}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7211","reported_seq":7295,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.367787+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:00.367787+0000","last_peered":"2024-09-16T04:46:00.367787+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:00.367787+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42906892,"num_objects":5238,"num_object_clones":0,"num_object_copies":20952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7209,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3108}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7093","reported_seq":7166,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.515982+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:55.515982+0000","last_peered":"2024-09-16T04:45:55.515982+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:55.515982+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7093,"log_dups_size":0,"ondisk_log_size":7093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710284,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7091,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2978}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":700204736,"num_objects":85480,"num_object_clones":0,"num_object_copies":341920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116007,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3344,"num_bytes_recovered":26386432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1321775104,"data_stored":1321775104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116034,"ondisk_log_size":116034,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745008,"kb_used_data":344012,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93626832,"statfs":{"total":96636764160,"available":95873875968,"internally_reserved":0,"allocated":352268288,"data_stored":352088319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410578639},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,6,0,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739948,"kb_used_data":339484,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93631892,"statfs":{"total":96636764160,"available":95879057408,"internally_reserved":0,"allocated":347631616,"data_stored":347451647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,7,2,1,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051639,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554716,"kb_used_data":266188,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93817124,"statfs":{"total":96636764160,"available":96068734976,"internally_reserved":0,"allocated":272576512,"data_stored":272396543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295431887},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,0,1,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736216,"kb_used_data":343620,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93635624,"statfs":{"total":96636764160,"available":95882878976,"internally_reserved":0,"allocated":351866880,"data_stored":351686911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":401993422},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":351350784,"data_stored":351350784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272277504,"data_stored":272277504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346923008,"data_stored":346923008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":351223808,"data_stored":351223808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:01.186 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:01.776 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:01.776 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:01.805 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-09-16T04:46:00.655016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":700204736,"num_objects":85480,"num_object_clones":0,"num_object_copies":341920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116007,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3344,"num_bytes_recovered":26386432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116034,"ondisk_log_size":116034,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2775888,"kb_used_data":1293304,"kb_used_omap":14,"kb_used_meta":1482481,"kb_avail":374711472,"statfs":{"total":386547056640,"available":383704547328,"internally_reserved":0,"allocated":1324343296,"data_stored":1323623420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1518061105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,1,6,14,12,3,1,0,0,0,0,0,1,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22003712,"num_objects":-2686,"num_object_clones":0,"num_object_copies":-10744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2686,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006554"},"pg_stats":[{"pgid":"1.f","version":"33'7265","reported_seq":7332,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.691589+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:45:55.691589+0000","last_peered":"2024-09-16T04:45:55.691589+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:45:55.691589+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7265,"log_dups_size":0,"ondisk_log_size":7265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44146688,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7265,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7268","reported_seq":7341,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.516927+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:45:55.516927+0000","last_peered":"2024-09-16T04:45:55.516927+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:45:55.516927+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7268,"log_dups_size":0,"ondisk_log_size":7268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44283148,"num_objects":5406,"num_object_clones":0,"num_object_copies":21624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7266,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7313","reported_seq":7396,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.329004+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:00.329004+0000","last_peered":"2024-09-16T04:46:00.329004+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:00.329004+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7313,"log_dups_size":0,"ondisk_log_size":7313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43883166,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7312,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7286","reported_seq":7357,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.413307+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:45:55.413307+0000","last_peered":"2024-09-16T04:45:55.413307+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:45:55.413307+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7286,"log_dups_size":0,"ondisk_log_size":7286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44102924,"num_objects":5384,"num_object_clones":0,"num_object_copies":21536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7284,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7366","reported_seq":7451,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.689170+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:45:55.689170+0000","last_peered":"2024-09-16T04:45:55.689170+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:45:55.689170+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7366,"log_dups_size":0,"ondisk_log_size":7366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44029196,"num_objects":5375,"num_object_clones":0,"num_object_copies":21500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7364,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7095","reported_seq":7161,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.629173+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:45:55.629173+0000","last_peered":"2024-09-16T04:45:55.629173+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:45:55.629173+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7095,"log_dups_size":0,"ondisk_log_size":7095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42737664,"num_objects":5217,"num_object_clones":0,"num_object_copies":20868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7095,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7244","reported_seq":7314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.594560+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:45:55.594560+0000","last_peered":"2024-09-16T04:45:55.594560+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:45:55.594560+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7244,"log_dups_size":0,"ondisk_log_size":7244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43873548,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7242,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3132}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7384","reported_seq":11940,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:57.585669+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:57.585669+0000","last_peered":"2024-09-16T04:45:57.585669+0000","last_clean":"2024-09-16T04:45:57.585669+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:57.585669+0000","last_undegraded":"2024-09-16T04:45:57.585669+0000","last_fullsized":"2024-09-16T04:45:57.585669+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7384,"log_dups_size":0,"ondisk_log_size":7384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:00.364801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44242188,"num_objects":5401,"num_object_clones":0,"num_object_copies":21604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7382,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7376","reported_seq":9401,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.765339+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:45:55.765339+0000","last_peered":"2024-09-16T04:45:55.765339+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:45:55.765339+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7376,"log_dups_size":0,"ondisk_log_size":7376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44067352,"num_objects":5380,"num_object_clones":0,"num_object_copies":21520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7373,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":986,"num_bytes_recovered":8052736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4023}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7375","reported_seq":7468,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.320109+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:00.320109+0000","last_peered":"2024-09-16T04:46:00.320109+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:00.320109+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7375,"log_dups_size":0,"ondisk_log_size":7375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44094732,"num_objects":5383,"num_object_clones":0,"num_object_copies":21532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7373,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3190}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7168","reported_seq":7247,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.779981+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:45:55.779981+0000","last_peered":"2024-09-16T04:45:55.779981+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:45:55.779981+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7168,"log_dups_size":0,"ondisk_log_size":7168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43601810,"num_objects":5323,"num_object_clones":0,"num_object_copies":21292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7165,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3144}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7041","reported_seq":7098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.591117+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:45:55.591117+0000","last_peered":"2024-09-16T04:45:55.591117+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:45:55.591117+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7041,"log_dups_size":0,"ondisk_log_size":7041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43334278,"num_objects":5290,"num_object_clones":0,"num_object_copies":21160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7040,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3170}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7242","reported_seq":7319,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.306982+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:00.306982+0000","last_peered":"2024-09-16T04:46:00.306982+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:00.306982+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7242,"log_dups_size":0,"ondisk_log_size":7242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43342470,"num_objects":5291,"num_object_clones":0,"num_object_copies":21164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7241,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3067}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7307","reported_seq":7396,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.748539+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:45:55.748539+0000","last_peered":"2024-09-16T04:45:55.748539+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:45:55.748539+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7307,"log_dups_size":0,"ondisk_log_size":7307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44848396,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7305,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3258}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7211","reported_seq":7295,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.367787+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:00.367787+0000","last_peered":"2024-09-16T04:46:00.367787+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:00.367787+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42906892,"num_objects":5238,"num_object_clones":0,"num_object_copies":20952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7209,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3108}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7093","reported_seq":7166,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:45:55.515982+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:45:55.515982+0000","last_peered":"2024-09-16T04:45:55.515982+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:45:55.515982+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7093,"log_dups_size":0,"ondisk_log_size":7093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710284,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7091,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2978}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":700204736,"num_objects":85480,"num_object_clones":0,"num_object_copies":341920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116007,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3344,"num_bytes_recovered":26386432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1321775104,"data_stored":1321775104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116034,"ondisk_log_size":116034,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745008,"kb_used_data":344012,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93626832,"statfs":{"total":96636764160,"available":95873875968,"internally_reserved":0,"allocated":352268288,"data_stored":352088319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":410578639},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,6,0,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739948,"kb_used_data":339484,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93631892,"statfs":{"total":96636764160,"available":95879057408,"internally_reserved":0,"allocated":347631616,"data_stored":347451647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,7,2,1,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051639,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554716,"kb_used_data":266188,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93817124,"statfs":{"total":96636764160,"available":96068734976,"internally_reserved":0,"allocated":272576512,"data_stored":272396543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295431887},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,0,1,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736216,"kb_used_data":343620,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93635624,"statfs":{"total":96636764160,"available":95882878976,"internally_reserved":0,"allocated":351866880,"data_stored":351686911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":401993422},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":351350784,"data_stored":351350784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272277504,"data_stored":272277504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346923008,"data_stored":346923008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":351223808,"data_stored":351223808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:01.806 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:46:01.806 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7265", 'reported_seq': 7332, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:55.691589+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:45:55.691589+0000', 'last_peered': '2024-09-16T04:45:55.691589+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:45:55.691589+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7265, 'log_dups_size': 0, 'ondisk_log_size': 7265, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44146688, 'num_objects': 5389, 'num_object_clones': 0, 'num_object_copies': 21556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7265, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3181}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:01.806 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:46:01.806 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7268", 'reported_seq': 7341, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:55.516927+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:45:55.516927+0000', 'last_peered': '2024-09-16T04:45:55.516927+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:45:55.516927+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7268, 'log_dups_size': 0, 'ondisk_log_size': 7268, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44283148, 'num_objects': 5406, 'num_object_clones': 0, 'num_object_copies': 21624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2210, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7266, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3198}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:01.806 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:46:01.806 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7313", 'reported_seq': 7396, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:00.329004+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:46:00.329004+0000', 'last_peered': '2024-09-16T04:46:00.329004+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:46:00.329004+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7313, 'log_dups_size': 0, 'ondisk_log_size': 7313, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43883166, 'num_objects': 5358, 'num_object_clones': 0, 'num_object_copies': 21432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2225, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5358, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7312, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3134}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:01.807 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:46:01.807 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7286", 'reported_seq': 7357, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:55.413307+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:45:55.413307+0000', 'last_peered': '2024-09-16T04:45:55.413307+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:45:55.413307+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7286, 'log_dups_size': 0, 'ondisk_log_size': 7286, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44102924, 'num_objects': 5384, 'num_object_clones': 0, 'num_object_copies': 21536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5384, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7284, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3157}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:01.807 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:46:01.807 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7366", 'reported_seq': 7451, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:55.689170+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:45:55.689170+0000', 'last_peered': '2024-09-16T04:45:55.689170+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:45:55.689170+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7366, 'log_dups_size': 0, 'ondisk_log_size': 7366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44029196, 'num_objects': 5375, 'num_object_clones': 0, 'num_object_copies': 21500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2271, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5375, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7364, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3106}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:01.807 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:46:01.807 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7095", 'reported_seq': 7161, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:55.629173+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:45:55.629173+0000', 'last_peered': '2024-09-16T04:45:55.629173+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:45:55.629173+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7095, 'log_dups_size': 0, 'ondisk_log_size': 7095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42737664, 'num_objects': 5217, 'num_object_clones': 0, 'num_object_copies': 20868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2197, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5217, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7095, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3020}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:01.807 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:46:01.807 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7244", 'reported_seq': 7314, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:55.594560+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:45:55.594560+0000', 'last_peered': '2024-09-16T04:45:55.594560+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:45:55.594560+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7244, 'log_dups_size': 0, 'ondisk_log_size': 7244, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43873548, 'num_objects': 5356, 'num_object_clones': 0, 'num_object_copies': 21424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7242, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3132}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:01.807 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:46:01.807 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7376", 'reported_seq': 9401, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:55.765339+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:45:55.765339+0000', 'last_peered': '2024-09-16T04:45:55.765339+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:45:55.765339+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7376, 'log_dups_size': 0, 'ondisk_log_size': 7376, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44067352, 'num_objects': 5380, 'num_object_clones': 0, 'num_object_copies': 21520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1357, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5380, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7373, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 986, 'num_bytes_recovered': 8052736, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4023}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:01.807 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:46:01.808 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7375", 'reported_seq': 7468, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:00.320109+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:46:00.320109+0000', 'last_peered': '2024-09-16T04:46:00.320109+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:46:00.320109+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7375, 'log_dups_size': 0, 'ondisk_log_size': 7375, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44094732, 'num_objects': 5383, 'num_object_clones': 0, 'num_object_copies': 21532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2195, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7373, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3190}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:01.808 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:46:01.808 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7168", 'reported_seq': 7247, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:55.779981+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:45:55.779981+0000', 'last_peered': '2024-09-16T04:45:55.779981+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:45:55.779981+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7168, 'log_dups_size': 0, 'ondisk_log_size': 7168, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43601810, 'num_objects': 5323, 'num_object_clones': 0, 'num_object_copies': 21292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2182, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5323, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7165, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3144}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:01.808 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:46:01.808 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7041", 'reported_seq': 7098, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:55.591117+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:45:55.591117+0000', 'last_peered': '2024-09-16T04:45:55.591117+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:45:55.591117+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7041, 'log_dups_size': 0, 'ondisk_log_size': 7041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43334278, 'num_objects': 5290, 'num_object_clones': 0, 'num_object_copies': 21160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2121, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7040, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3170}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:01.808 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:46:01.808 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7242", 'reported_seq': 7319, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:00.306982+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:46:00.306982+0000', 'last_peered': '2024-09-16T04:46:00.306982+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:46:00.306982+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7242, 'log_dups_size': 0, 'ondisk_log_size': 7242, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43342470, 'num_objects': 5291, 'num_object_clones': 0, 'num_object_copies': 21164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2225, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5291, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7241, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3067}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:01.808 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:46:01.808 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7307", 'reported_seq': 7396, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:55.748539+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:45:55.748539+0000', 'last_peered': '2024-09-16T04:45:55.748539+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:45:55.748539+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7307, 'log_dups_size': 0, 'ondisk_log_size': 7307, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44848396, 'num_objects': 5475, 'num_object_clones': 0, 'num_object_copies': 21900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5475, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7305, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3258}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:01.808 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:46:01.808 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7211", 'reported_seq': 7295, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:00.367787+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:46:00.367787+0000', 'last_peered': '2024-09-16T04:46:00.367787+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:46:00.367787+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7211, 'log_dups_size': 0, 'ondisk_log_size': 7211, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42906892, 'num_objects': 5238, 'num_object_clones': 0, 'num_object_copies': 20952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7209, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3108}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:01.809 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:46:01.809 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7093", 'reported_seq': 7166, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:45:55.515982+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:45:55.515982+0000', 'last_peered': '2024-09-16T04:45:55.515982+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:45:55.515982+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7093, 'log_dups_size': 0, 'ondisk_log_size': 7093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42710284, 'num_objects': 5214, 'num_object_clones': 0, 'num_object_copies': 20856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7091, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2978}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:01.809 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:46:02.483 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:02.483 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32235/341920 objects degraded (9.428%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":374,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":85480,"data_bytes":700204736,"bytes_used":2842509312,"bytes_avail":383704547328,"bytes_total":386547056640,"degraded_objects":32235,"degraded_total":341920,"degraded_ratio":0.094276438933083767,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":81192,"recovering_keys_per_sec":0,"num_objects_recovered":135,"num_bytes_recovered":974848,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":223},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:46:02.514 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:46:02.514 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:03.101 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:03.102 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:03.143 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-16T04:46:02.657124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":693692096,"num_objects":84685,"num_object_clones":0,"num_object_copies":338740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116802,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3418,"num_bytes_recovered":26935296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116829,"ondisk_log_size":116829,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762872,"kb_used_data":1284460,"kb_used_omap":14,"kb_used_meta":1478321,"kb_avail":374724488,"statfs":{"total":386547056640,"available":383717875712,"internally_reserved":0,"allocated":1315287040,"data_stored":1314567164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1513801265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,4,5,14,9,4,0,0,0,0,0,2,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23486464,"num_objects":-2867,"num_object_clones":0,"num_object_copies":-11468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2867,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1523712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007976"},"pg_stats":[{"pgid":"1.f","version":"33'7332","reported_seq":7401,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.876562+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:00.876562+0000","last_peered":"2024-09-16T04:46:00.876562+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:00.876562+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7332,"log_dups_size":0,"ondisk_log_size":7332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43597824,"num_objects":5322,"num_object_clones":0,"num_object_copies":21288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7332,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3115}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7331","reported_seq":7404,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.635273+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:00.635273+0000","last_peered":"2024-09-16T04:46:00.635273+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:00.635273+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7331,"log_dups_size":0,"ondisk_log_size":7331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43767052,"num_objects":5343,"num_object_clones":0,"num_object_copies":21372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7329,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3135}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7313","reported_seq":7396,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.329004+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:00.329004+0000","last_peered":"2024-09-16T04:46:00.329004+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:00.329004+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7313,"log_dups_size":0,"ondisk_log_size":7313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43883166,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7312,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7362","reported_seq":7433,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.699383+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:00.699383+0000","last_peered":"2024-09-16T04:46:00.699383+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:00.699383+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7362,"log_dups_size":0,"ondisk_log_size":7362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43480332,"num_objects":5308,"num_object_clones":0,"num_object_copies":21232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7360,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3081}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7426","reported_seq":7511,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.769526+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:00.769526+0000","last_peered":"2024-09-16T04:46:00.769526+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:00.769526+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7426,"log_dups_size":0,"ondisk_log_size":7426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43537676,"num_objects":5315,"num_object_clones":0,"num_object_copies":21260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7424,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3046}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7159","reported_seq":7226,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.765628+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:00.765628+0000","last_peered":"2024-09-16T04:46:00.765628+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:00.765628+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7159,"log_dups_size":0,"ondisk_log_size":7159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42213376,"num_objects":5153,"num_object_clones":0,"num_object_copies":20612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7159,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2956}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7312","reported_seq":7382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.575298+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:00.575298+0000","last_peered":"2024-09-16T04:46:00.575298+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:00.575298+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7312,"log_dups_size":0,"ondisk_log_size":7312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43316492,"num_objects":5288,"num_object_clones":0,"num_object_copies":21152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7310,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3064}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7384","reported_seq":11940,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:45:57.585669+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:45:57.585669+0000","last_peered":"2024-09-16T04:45:57.585669+0000","last_clean":"2024-09-16T04:45:57.585669+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:45:57.585669+0000","last_undegraded":"2024-09-16T04:45:57.585669+0000","last_fullsized":"2024-09-16T04:45:57.585669+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7384,"log_dups_size":0,"ondisk_log_size":7384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:00.364801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44242188,"num_objects":5401,"num_object_clones":0,"num_object_copies":21604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7382,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7464","reported_seq":9629,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.878934+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:00.878934+0000","last_peered":"2024-09-16T04:46:00.878934+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:00.878934+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7464,"log_dups_size":0,"ondisk_log_size":7464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43346456,"num_objects":5292,"num_object_clones":0,"num_object_copies":21168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7461,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1056,"num_bytes_recovered":8601600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7375","reported_seq":7468,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.320109+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:00.320109+0000","last_peered":"2024-09-16T04:46:00.320109+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:00.320109+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7375,"log_dups_size":0,"ondisk_log_size":7375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44094732,"num_objects":5383,"num_object_clones":0,"num_object_copies":21532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7373,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3190}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7241","reported_seq":7320,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.536828+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:00.536828+0000","last_peered":"2024-09-16T04:46:00.536828+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:00.536828+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7241,"log_dups_size":0,"ondisk_log_size":7241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43003794,"num_objects":5250,"num_object_clones":0,"num_object_copies":21000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7238,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3071}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7111","reported_seq":7170,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.793353+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:00.793353+0000","last_peered":"2024-09-16T04:46:00.793353+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:00.793353+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7111,"log_dups_size":0,"ondisk_log_size":7111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42760838,"num_objects":5220,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7110,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3101}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7242","reported_seq":7319,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.306982+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:00.306982+0000","last_peered":"2024-09-16T04:46:00.306982+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:00.306982+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7242,"log_dups_size":0,"ondisk_log_size":7242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43342470,"num_objects":5291,"num_object_clones":0,"num_object_copies":21164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7241,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3067}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7397","reported_seq":7490,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.727952+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:00.727952+0000","last_peered":"2024-09-16T04:46:00.727952+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:00.727952+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7397,"log_dups_size":0,"ondisk_log_size":7397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44111116,"num_objects":5385,"num_object_clones":0,"num_object_copies":21540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3170}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7211","reported_seq":7295,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.367787+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:00.367787+0000","last_peered":"2024-09-16T04:46:00.367787+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:00.367787+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42906892,"num_objects":5238,"num_object_clones":0,"num_object_copies":20952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7209,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3108}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7169","reported_seq":7242,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:00.634647+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:00.634647+0000","last_peered":"2024-09-16T04:46:00.634647+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:00.634647+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7169,"log_dups_size":0,"ondisk_log_size":7169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42087692,"num_objects":5138,"num_object_clones":0,"num_object_copies":20552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7167,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":693692096,"num_objects":84685,"num_object_clones":0,"num_object_copies":338740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116802,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3418,"num_bytes_recovered":26935296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1312268288,"data_stored":1312268288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116829,"ondisk_log_size":116829,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732308,"kb_used_data":339640,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93639532,"statfs":{"total":96636764160,"available":95886880768,"internally_reserved":0,"allocated":347791360,"data_stored":347611391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":402058959},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,6,3,1,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739948,"kb_used_data":339484,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93631892,"statfs":{"total":96636764160,"available":95879057408,"internally_reserved":0,"allocated":347631616,"data_stored":347451647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,7,2,1,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051639,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554716,"kb_used_data":266188,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93817124,"statfs":{"total":96636764160,"available":96068734976,"internally_reserved":0,"allocated":272576512,"data_stored":272396543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295431887},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,0,1,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735900,"kb_used_data":339148,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93635940,"statfs":{"total":96636764160,"available":95883202560,"internally_reserved":0,"allocated":347287552,"data_stored":347107583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":406253262},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346624000,"data_stored":346624000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272277504,"data_stored":272277504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346923008,"data_stored":346923008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":346443776,"data_stored":346443776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:06.145 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:06.772 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:06.773 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:06.810 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-16T04:46:06.658123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":684123840,"num_objects":83517,"num_object_clones":0,"num_object_copies":334068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83517,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117970,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3486,"num_bytes_recovered":27377664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117997,"ondisk_log_size":117997,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2802288,"kb_used_data":1269732,"kb_used_omap":14,"kb_used_meta":1532465,"kb_avail":374685072,"statfs":{"total":386547056640,"available":383677513728,"internally_reserved":0,"allocated":1300205568,"data_stored":1299485692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1569244721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,4,19,10,7,2,2,0,0,4,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27762688,"num_objects":-3389,"num_object_clones":0,"num_object_copies":-13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3389,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":227,"num_bytes_recovered":1605632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004689"},"pg_stats":[{"pgid":"1.f","version":"33'7395","reported_seq":7466,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.776253+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:05.776253+0000","last_peered":"2024-09-16T04:46:05.776253+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:05.776253+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7395,"log_dups_size":0,"ondisk_log_size":7395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43081728,"num_objects":5259,"num_object_clones":0,"num_object_copies":21036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3053}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7397","reported_seq":7470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.662835+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:05.662835+0000","last_peered":"2024-09-16T04:46:05.662835+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:05.662835+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7397,"log_dups_size":0,"ondisk_log_size":7397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43226380,"num_objects":5277,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3069}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7398","reported_seq":7481,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.250477+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:05.250477+0000","last_peered":"2024-09-16T04:46:05.250477+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:05.250477+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7398,"log_dups_size":0,"ondisk_log_size":7398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43186846,"num_objects":5273,"num_object_clones":0,"num_object_copies":21092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7397,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3049}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7425","reported_seq":7496,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.622069+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:05.622069+0000","last_peered":"2024-09-16T04:46:05.622069+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:05.622069+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7425,"log_dups_size":0,"ondisk_log_size":7425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42964236,"num_objects":5245,"num_object_clones":0,"num_object_copies":20980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7423,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3018}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7490","reported_seq":7579,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.668187+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:05.668187+0000","last_peered":"2024-09-16T04:46:05.668187+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:05.668187+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43013388,"num_objects":5251,"num_object_clones":0,"num_object_copies":21004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7233","reported_seq":7302,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.831404+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:05.831404+0000","last_peered":"2024-09-16T04:46:05.831404+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:05.831404+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7233,"log_dups_size":0,"ondisk_log_size":7233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41607168,"num_objects":5079,"num_object_clones":0,"num_object_copies":20316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7233,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7375","reported_seq":7445,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.325850+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:05.325850+0000","last_peered":"2024-09-16T04:46:05.325850+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:05.325850+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7375,"log_dups_size":0,"ondisk_log_size":7375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42800396,"num_objects":5225,"num_object_clones":0,"num_object_copies":20900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7373,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3001}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7465","reported_seq":12021,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:02.627327+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:02.627327+0000","last_peered":"2024-09-16T04:46:02.627327+0000","last_clean":"2024-09-16T04:46:02.627327+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:02.627327+0000","last_undegraded":"2024-09-16T04:46:02.627327+0000","last_fullsized":"2024-09-16T04:46:02.627327+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:05.436918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43578636,"num_objects":5320,"num_object_clones":0,"num_object_copies":21280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7463,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7543","reported_seq":9822,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.922621+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:05.922621+0000","last_peered":"2024-09-16T04:46:05.922621+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:05.922621+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42699288,"num_objects":5213,"num_object_clones":0,"num_object_copies":20852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7540,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1113,"num_bytes_recovered":9043968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3983}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7458","reported_seq":7551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.291569+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:05.291569+0000","last_peered":"2024-09-16T04:46:05.291569+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:05.291569+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7458,"log_dups_size":0,"ondisk_log_size":7458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43414796,"num_objects":5300,"num_object_clones":0,"num_object_copies":21200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7456,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3107}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7316","reported_seq":7397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.721195+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:05.721195+0000","last_peered":"2024-09-16T04:46:05.721195+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:05.721195+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42389394,"num_objects":5175,"num_object_clones":0,"num_object_copies":20700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7313,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7183","reported_seq":7239,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.844717+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:05.844717+0000","last_peered":"2024-09-16T04:46:05.844717+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:05.844717+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42171014,"num_objects":5148,"num_object_clones":0,"num_object_copies":20592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7182,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3029}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7325","reported_seq":7404,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.273248+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:05.273248+0000","last_peered":"2024-09-16T04:46:05.273248+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:05.273248+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7325,"log_dups_size":0,"ondisk_log_size":7325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42662534,"num_objects":5208,"num_object_clones":0,"num_object_copies":20832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7324,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2985}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7464","reported_seq":7561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.856207+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:05.856207+0000","last_peered":"2024-09-16T04:46:05.856207+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:05.856207+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7464,"log_dups_size":0,"ondisk_log_size":7464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43562252,"num_objects":5318,"num_object_clones":0,"num_object_copies":21272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7462,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7288","reported_seq":7376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.401300+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:05.401300+0000","last_peered":"2024-09-16T04:46:05.401300+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:05.401300+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7288,"log_dups_size":0,"ondisk_log_size":7288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42276108,"num_objects":5161,"num_object_clones":0,"num_object_copies":20644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7286,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3033}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7242","reported_seq":7315,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.637002+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:05.637002+0000","last_peered":"2024-09-16T04:46:05.637002+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:05.637002+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7242,"log_dups_size":0,"ondisk_log_size":7242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41489676,"num_objects":5065,"num_object_clones":0,"num_object_copies":20260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":684123840,"num_objects":83517,"num_object_clones":0,"num_object_copies":334068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83517,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117970,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3486,"num_bytes_recovered":27377664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1297346560,"data_stored":1297346560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117997,"ondisk_log_size":117997,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749708,"kb_used_data":334844,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93622132,"statfs":{"total":96636764160,"available":95869063168,"internally_reserved":0,"allocated":342880256,"data_stored":342700287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424799951},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,6,3,1,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740736,"kb_used_data":334692,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93631104,"statfs":{"total":96636764160,"available":95878250496,"internally_reserved":0,"allocated":342724608,"data_stored":342544639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,2,2,2,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051640,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558480,"kb_used_data":265792,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93813360,"statfs":{"total":96636764160,"available":96064880640,"internally_reserved":0,"allocated":272171008,"data_stored":271991039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299691727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753364,"kb_used_data":334404,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93618476,"statfs":{"total":96636764160,"available":95865319424,"internally_reserved":0,"allocated":342429696,"data_stored":342249727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":428994254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":341794816,"data_stored":341794816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":271876096,"data_stored":271876096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":342085632,"data_stored":342085632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":341590016,"data_stored":341590016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:06.812 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:07.457 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:07.458 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:07.484 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-16T04:46:06.658123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":684123840,"num_objects":83517,"num_object_clones":0,"num_object_copies":334068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83517,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117970,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3486,"num_bytes_recovered":27377664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117997,"ondisk_log_size":117997,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2802288,"kb_used_data":1269732,"kb_used_omap":14,"kb_used_meta":1532465,"kb_avail":374685072,"statfs":{"total":386547056640,"available":383677513728,"internally_reserved":0,"allocated":1300205568,"data_stored":1299485692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1569244721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,4,19,10,7,2,2,0,0,4,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27762688,"num_objects":-3389,"num_object_clones":0,"num_object_copies":-13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3389,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":227,"num_bytes_recovered":1605632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004689"},"pg_stats":[{"pgid":"1.f","version":"33'7395","reported_seq":7466,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.776253+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:05.776253+0000","last_peered":"2024-09-16T04:46:05.776253+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:05.776253+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7395,"log_dups_size":0,"ondisk_log_size":7395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43081728,"num_objects":5259,"num_object_clones":0,"num_object_copies":21036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3053}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7397","reported_seq":7470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.662835+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:05.662835+0000","last_peered":"2024-09-16T04:46:05.662835+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:05.662835+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7397,"log_dups_size":0,"ondisk_log_size":7397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43226380,"num_objects":5277,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3069}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7398","reported_seq":7481,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.250477+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:05.250477+0000","last_peered":"2024-09-16T04:46:05.250477+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:05.250477+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7398,"log_dups_size":0,"ondisk_log_size":7398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43186846,"num_objects":5273,"num_object_clones":0,"num_object_copies":21092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7397,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3049}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7425","reported_seq":7496,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.622069+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:05.622069+0000","last_peered":"2024-09-16T04:46:05.622069+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:05.622069+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7425,"log_dups_size":0,"ondisk_log_size":7425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42964236,"num_objects":5245,"num_object_clones":0,"num_object_copies":20980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7423,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3018}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7490","reported_seq":7579,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.668187+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:05.668187+0000","last_peered":"2024-09-16T04:46:05.668187+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:05.668187+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43013388,"num_objects":5251,"num_object_clones":0,"num_object_copies":21004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7233","reported_seq":7302,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.831404+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:05.831404+0000","last_peered":"2024-09-16T04:46:05.831404+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:05.831404+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7233,"log_dups_size":0,"ondisk_log_size":7233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41607168,"num_objects":5079,"num_object_clones":0,"num_object_copies":20316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7233,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7375","reported_seq":7445,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.325850+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:05.325850+0000","last_peered":"2024-09-16T04:46:05.325850+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:05.325850+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7375,"log_dups_size":0,"ondisk_log_size":7375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42800396,"num_objects":5225,"num_object_clones":0,"num_object_copies":20900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7373,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3001}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7465","reported_seq":12021,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:02.627327+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:02.627327+0000","last_peered":"2024-09-16T04:46:02.627327+0000","last_clean":"2024-09-16T04:46:02.627327+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:02.627327+0000","last_undegraded":"2024-09-16T04:46:02.627327+0000","last_fullsized":"2024-09-16T04:46:02.627327+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:05.436918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43578636,"num_objects":5320,"num_object_clones":0,"num_object_copies":21280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7463,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7543","reported_seq":9822,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.922621+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:05.922621+0000","last_peered":"2024-09-16T04:46:05.922621+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:05.922621+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42699288,"num_objects":5213,"num_object_clones":0,"num_object_copies":20852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7540,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1113,"num_bytes_recovered":9043968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3983}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7458","reported_seq":7551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.291569+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:05.291569+0000","last_peered":"2024-09-16T04:46:05.291569+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:05.291569+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7458,"log_dups_size":0,"ondisk_log_size":7458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43414796,"num_objects":5300,"num_object_clones":0,"num_object_copies":21200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7456,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3107}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7316","reported_seq":7397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.721195+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:05.721195+0000","last_peered":"2024-09-16T04:46:05.721195+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:05.721195+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42389394,"num_objects":5175,"num_object_clones":0,"num_object_copies":20700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7313,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7183","reported_seq":7239,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.844717+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:05.844717+0000","last_peered":"2024-09-16T04:46:05.844717+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:05.844717+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42171014,"num_objects":5148,"num_object_clones":0,"num_object_copies":20592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7182,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3029}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7325","reported_seq":7404,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.273248+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:05.273248+0000","last_peered":"2024-09-16T04:46:05.273248+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:05.273248+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7325,"log_dups_size":0,"ondisk_log_size":7325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42662534,"num_objects":5208,"num_object_clones":0,"num_object_copies":20832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7324,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2985}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7464","reported_seq":7561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.856207+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:05.856207+0000","last_peered":"2024-09-16T04:46:05.856207+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:05.856207+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7464,"log_dups_size":0,"ondisk_log_size":7464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43562252,"num_objects":5318,"num_object_clones":0,"num_object_copies":21272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7462,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7288","reported_seq":7376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.401300+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:05.401300+0000","last_peered":"2024-09-16T04:46:05.401300+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:05.401300+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7288,"log_dups_size":0,"ondisk_log_size":7288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42276108,"num_objects":5161,"num_object_clones":0,"num_object_copies":20644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7286,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3033}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7242","reported_seq":7315,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.637002+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:05.637002+0000","last_peered":"2024-09-16T04:46:05.637002+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:05.637002+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7242,"log_dups_size":0,"ondisk_log_size":7242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41489676,"num_objects":5065,"num_object_clones":0,"num_object_copies":20260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":684123840,"num_objects":83517,"num_object_clones":0,"num_object_copies":334068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83517,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117970,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3486,"num_bytes_recovered":27377664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1297346560,"data_stored":1297346560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117997,"ondisk_log_size":117997,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749708,"kb_used_data":334844,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93622132,"statfs":{"total":96636764160,"available":95869063168,"internally_reserved":0,"allocated":342880256,"data_stored":342700287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424799951},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,6,3,1,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740736,"kb_used_data":334692,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93631104,"statfs":{"total":96636764160,"available":95878250496,"internally_reserved":0,"allocated":342724608,"data_stored":342544639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,2,2,2,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051640,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558480,"kb_used_data":265792,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93813360,"statfs":{"total":96636764160,"available":96064880640,"internally_reserved":0,"allocated":272171008,"data_stored":271991039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299691727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753364,"kb_used_data":334404,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93618476,"statfs":{"total":96636764160,"available":95865319424,"internally_reserved":0,"allocated":342429696,"data_stored":342249727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":428994254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":341794816,"data_stored":341794816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":271876096,"data_stored":271876096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":342085632,"data_stored":342085632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":341590016,"data_stored":341590016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:07.485 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:46:07.485 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7395", 'reported_seq': 7466, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:05.776253+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:46:05.776253+0000', 'last_peered': '2024-09-16T04:46:05.776253+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:46:05.776253+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7395, 'log_dups_size': 0, 'ondisk_log_size': 7395, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43081728, 'num_objects': 5259, 'num_object_clones': 0, 'num_object_copies': 21036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2206, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7395, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3053}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:07.485 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:46:07.485 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7397", 'reported_seq': 7470, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:05.662835+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:46:05.662835+0000', 'last_peered': '2024-09-16T04:46:05.662835+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:46:05.662835+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7397, 'log_dups_size': 0, 'ondisk_log_size': 7397, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43226380, 'num_objects': 5277, 'num_object_clones': 0, 'num_object_copies': 21108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2210, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5277, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7395, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3069}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:07.485 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:46:07.485 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7398", 'reported_seq': 7481, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:05.250477+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:46:05.250477+0000', 'last_peered': '2024-09-16T04:46:05.250477+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:46:05.250477+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7398, 'log_dups_size': 0, 'ondisk_log_size': 7398, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43186846, 'num_objects': 5273, 'num_object_clones': 0, 'num_object_copies': 21092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2225, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5273, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7397, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3049}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:07.485 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:46:07.485 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7425", 'reported_seq': 7496, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:05.622069+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:46:05.622069+0000', 'last_peered': '2024-09-16T04:46:05.622069+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:46:05.622069+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7425, 'log_dups_size': 0, 'ondisk_log_size': 7425, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42964236, 'num_objects': 5245, 'num_object_clones': 0, 'num_object_copies': 20980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5245, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7423, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3018}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:07.486 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:46:07.486 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7490", 'reported_seq': 7579, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:05.668187+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:46:05.668187+0000', 'last_peered': '2024-09-16T04:46:05.668187+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:46:05.668187+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7490, 'log_dups_size': 0, 'ondisk_log_size': 7490, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43013388, 'num_objects': 5251, 'num_object_clones': 0, 'num_object_copies': 21004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7488, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2984}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:07.486 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:46:07.486 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7233", 'reported_seq': 7302, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:05.831404+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:46:05.831404+0000', 'last_peered': '2024-09-16T04:46:05.831404+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:46:05.831404+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7233, 'log_dups_size': 0, 'ondisk_log_size': 7233, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41607168, 'num_objects': 5079, 'num_object_clones': 0, 'num_object_copies': 20316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2196, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5079, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7233, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2883}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:07.486 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:46:07.486 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7375", 'reported_seq': 7445, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:05.325850+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:46:05.325850+0000', 'last_peered': '2024-09-16T04:46:05.325850+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:46:05.325850+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7375, 'log_dups_size': 0, 'ondisk_log_size': 7375, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42800396, 'num_objects': 5225, 'num_object_clones': 0, 'num_object_copies': 20900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5225, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7373, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3001}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:07.486 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:46:07.486 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7543", 'reported_seq': 9822, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:05.922621+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:46:05.922621+0000', 'last_peered': '2024-09-16T04:46:05.922621+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:46:05.922621+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7543, 'log_dups_size': 0, 'ondisk_log_size': 7543, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42699288, 'num_objects': 5213, 'num_object_clones': 0, 'num_object_copies': 20852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5213, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7540, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1113, 'num_bytes_recovered': 9043968, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3983}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:07.486 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:46:07.486 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7458", 'reported_seq': 7551, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:05.291569+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:46:05.291569+0000', 'last_peered': '2024-09-16T04:46:05.291569+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:46:05.291569+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7458, 'log_dups_size': 0, 'ondisk_log_size': 7458, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43414796, 'num_objects': 5300, 'num_object_clones': 0, 'num_object_copies': 21200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2195, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7456, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3107}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:07.486 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:46:07.487 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7316", 'reported_seq': 7397, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:05.721195+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:46:05.721195+0000', 'last_peered': '2024-09-16T04:46:05.721195+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:46:05.721195+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7316, 'log_dups_size': 0, 'ondisk_log_size': 7316, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42389394, 'num_objects': 5175, 'num_object_clones': 0, 'num_object_copies': 20700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2180, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5175, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7313, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2998}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:07.487 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:46:07.487 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7183", 'reported_seq': 7239, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:05.844717+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:46:05.844717+0000', 'last_peered': '2024-09-16T04:46:05.844717+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:46:05.844717+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7183, 'log_dups_size': 0, 'ondisk_log_size': 7183, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42171014, 'num_objects': 5148, 'num_object_clones': 0, 'num_object_copies': 20592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2120, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5148, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7182, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3029}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:07.487 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:46:07.487 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7325", 'reported_seq': 7404, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:05.273248+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:46:05.273248+0000', 'last_peered': '2024-09-16T04:46:05.273248+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:46:05.273248+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7325, 'log_dups_size': 0, 'ondisk_log_size': 7325, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42662534, 'num_objects': 5208, 'num_object_clones': 0, 'num_object_copies': 20832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5208, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7324, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2985}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:07.487 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:46:07.487 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7464", 'reported_seq': 7561, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:05.856207+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:46:05.856207+0000', 'last_peered': '2024-09-16T04:46:05.856207+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:46:05.856207+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7464, 'log_dups_size': 0, 'ondisk_log_size': 7464, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43562252, 'num_objects': 5318, 'num_object_clones': 0, 'num_object_copies': 21272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7462, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3105}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:07.487 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:46:07.487 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7288", 'reported_seq': 7376, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:05.401300+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:46:05.401300+0000', 'last_peered': '2024-09-16T04:46:05.401300+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:46:05.401300+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7288, 'log_dups_size': 0, 'ondisk_log_size': 7288, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42276108, 'num_objects': 5161, 'num_object_clones': 0, 'num_object_copies': 20644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5161, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7286, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3033}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:07.487 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:46:07.487 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7242", 'reported_seq': 7315, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:05.637002+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:46:05.637002+0000', 'last_peered': '2024-09-16T04:46:05.637002+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:46:05.637002+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7242, 'log_dups_size': 0, 'ondisk_log_size': 7242, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41489676, 'num_objects': 5065, 'num_object_clones': 0, 'num_object_copies': 20260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7240, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2829}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:07.488 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:46:08.178 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:08.178 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32093/334068 objects degraded (9.607%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":380,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":83517,"data_bytes":684123840,"bytes_used":2869542912,"bytes_avail":383677513728,"bytes_total":386547056640,"degraded_objects":32093,"degraded_total":334068,"degraded_ratio":0.096067267741896861,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":133750,"recovering_keys_per_sec":0,"num_objects_recovered":227,"num_bytes_recovered":1605632,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":282},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:46:08.207 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:46:08.207 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:08.769 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:08.769 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:08.796 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-09-16T04:46:08.658655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":683534016,"num_objects":83445,"num_object_clones":0,"num_object_copies":333780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83445,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118042,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3486,"num_bytes_recovered":27377664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118069,"ondisk_log_size":118069,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789552,"kb_used_data":1269476,"kb_used_omap":14,"kb_used_meta":1519985,"kb_avail":374697808,"statfs":{"total":386547056640,"available":383690555392,"internally_reserved":0,"allocated":1299943424,"data_stored":1299223548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1556465201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,5,21,9,8,2,2,0,0,4,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19767296,"num_objects":-2413,"num_object_clones":0,"num_object_copies":-9652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2413,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004709"},"pg_stats":[{"pgid":"1.f","version":"33'7395","reported_seq":7466,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.776253+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:05.776253+0000","last_peered":"2024-09-16T04:46:05.776253+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:05.776253+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7395,"log_dups_size":0,"ondisk_log_size":7395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43081728,"num_objects":5259,"num_object_clones":0,"num_object_copies":21036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3053}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7397","reported_seq":7470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.662835+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:05.662835+0000","last_peered":"2024-09-16T04:46:05.662835+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:05.662835+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7397,"log_dups_size":0,"ondisk_log_size":7397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43226380,"num_objects":5277,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3069}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7398","reported_seq":7481,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.250477+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:05.250477+0000","last_peered":"2024-09-16T04:46:05.250477+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:05.250477+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7398,"log_dups_size":0,"ondisk_log_size":7398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43186846,"num_objects":5273,"num_object_clones":0,"num_object_copies":21092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7397,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3049}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7425","reported_seq":7496,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.622069+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:05.622069+0000","last_peered":"2024-09-16T04:46:05.622069+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:05.622069+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7425,"log_dups_size":0,"ondisk_log_size":7425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42964236,"num_objects":5245,"num_object_clones":0,"num_object_copies":20980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7423,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3018}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7490","reported_seq":7579,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.668187+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:05.668187+0000","last_peered":"2024-09-16T04:46:05.668187+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:05.668187+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43013388,"num_objects":5251,"num_object_clones":0,"num_object_copies":21004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7233","reported_seq":7302,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.831404+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:05.831404+0000","last_peered":"2024-09-16T04:46:05.831404+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:05.831404+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7233,"log_dups_size":0,"ondisk_log_size":7233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41607168,"num_objects":5079,"num_object_clones":0,"num_object_copies":20316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7233,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7375","reported_seq":7445,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.325850+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:05.325850+0000","last_peered":"2024-09-16T04:46:05.325850+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:05.325850+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7375,"log_dups_size":0,"ondisk_log_size":7375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42800396,"num_objects":5225,"num_object_clones":0,"num_object_copies":20900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7373,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3001}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7537","reported_seq":12093,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:07.452852+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:07.452852+0000","last_peered":"2024-09-16T04:46:07.452852+0000","last_clean":"2024-09-16T04:46:07.452852+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:07.452852+0000","last_undegraded":"2024-09-16T04:46:07.452852+0000","last_fullsized":"2024-09-16T04:46:07.452852+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:10.439959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42988812,"num_objects":5248,"num_object_clones":0,"num_object_copies":20992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7535,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7543","reported_seq":9822,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.922621+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:05.922621+0000","last_peered":"2024-09-16T04:46:05.922621+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:05.922621+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42699288,"num_objects":5213,"num_object_clones":0,"num_object_copies":20852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7540,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1113,"num_bytes_recovered":9043968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3983}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7458","reported_seq":7551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.291569+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:05.291569+0000","last_peered":"2024-09-16T04:46:05.291569+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:05.291569+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7458,"log_dups_size":0,"ondisk_log_size":7458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43414796,"num_objects":5300,"num_object_clones":0,"num_object_copies":21200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7456,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3107}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7316","reported_seq":7397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.721195+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:05.721195+0000","last_peered":"2024-09-16T04:46:05.721195+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:05.721195+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42389394,"num_objects":5175,"num_object_clones":0,"num_object_copies":20700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7313,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7183","reported_seq":7239,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.844717+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:05.844717+0000","last_peered":"2024-09-16T04:46:05.844717+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:05.844717+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42171014,"num_objects":5148,"num_object_clones":0,"num_object_copies":20592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7182,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3029}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7325","reported_seq":7404,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.273248+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:05.273248+0000","last_peered":"2024-09-16T04:46:05.273248+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:05.273248+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7325,"log_dups_size":0,"ondisk_log_size":7325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42662534,"num_objects":5208,"num_object_clones":0,"num_object_copies":20832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7324,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2985}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7464","reported_seq":7561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.856207+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:05.856207+0000","last_peered":"2024-09-16T04:46:05.856207+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:05.856207+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7464,"log_dups_size":0,"ondisk_log_size":7464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43562252,"num_objects":5318,"num_object_clones":0,"num_object_copies":21272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7462,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7288","reported_seq":7376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.401300+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:05.401300+0000","last_peered":"2024-09-16T04:46:05.401300+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:05.401300+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7288,"log_dups_size":0,"ondisk_log_size":7288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42276108,"num_objects":5161,"num_object_clones":0,"num_object_copies":20644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7286,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3033}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7242","reported_seq":7315,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.637002+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:05.637002+0000","last_peered":"2024-09-16T04:46:05.637002+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:05.637002+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7242,"log_dups_size":0,"ondisk_log_size":7242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41489676,"num_objects":5065,"num_object_clones":0,"num_object_copies":20260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":683534016,"num_objects":83445,"num_object_clones":0,"num_object_copies":333780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83445,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118042,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3486,"num_bytes_recovered":27377664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1297076224,"data_stored":1297076224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118069,"ondisk_log_size":118069,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749708,"kb_used_data":334844,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93622132,"statfs":{"total":96636764160,"available":95869063168,"internally_reserved":0,"allocated":342880256,"data_stored":342700287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424799951},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,6,3,1,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740736,"kb_used_data":334692,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93631104,"statfs":{"total":96636764160,"available":95878250496,"internally_reserved":0,"allocated":342724608,"data_stored":342544639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,2,2,2,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545744,"kb_used_data":265536,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93826096,"statfs":{"total":96636764160,"available":96077922304,"internally_reserved":0,"allocated":271908864,"data_stored":271728895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":286912207},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753364,"kb_used_data":334404,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93618476,"statfs":{"total":96636764160,"available":95865319424,"internally_reserved":0,"allocated":342429696,"data_stored":342249727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":428994254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":341794816,"data_stored":341794816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":271605760,"data_stored":271605760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":342085632,"data_stored":342085632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":341590016,"data_stored":341590016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:11.798 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:12.439 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:12.439 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:12.462 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-16T04:46:10.659171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":681100992,"num_objects":83148,"num_object_clones":0,"num_object_copies":332592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83148,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118339,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3491,"num_bytes_recovered":27377664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118366,"ondisk_log_size":118366,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793428,"kb_used_data":1265044,"kb_used_omap":14,"kb_used_meta":1528305,"kb_avail":374693932,"statfs":{"total":386547056640,"available":383686586368,"internally_reserved":0,"allocated":1295405056,"data_stored":1294685180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1564984881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,20,9,6,0,0,0,0,0,2,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21561344,"num_objects":-2632,"num_object_clones":0,"num_object_copies":-10528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2632,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004657"},"pg_stats":[{"pgid":"1.f","version":"33'7395","reported_seq":7466,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.776253+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:05.776253+0000","last_peered":"2024-09-16T04:46:05.776253+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:05.776253+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7395,"log_dups_size":0,"ondisk_log_size":7395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43081728,"num_objects":5259,"num_object_clones":0,"num_object_copies":21036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3053}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7397","reported_seq":7470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.662835+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:05.662835+0000","last_peered":"2024-09-16T04:46:05.662835+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:05.662835+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7397,"log_dups_size":0,"ondisk_log_size":7397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43226380,"num_objects":5277,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3069}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7461","reported_seq":7545,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.465678+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:10.465678+0000","last_peered":"2024-09-16T04:46:10.465678+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:10.465678+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7461,"log_dups_size":0,"ondisk_log_size":7461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42670750,"num_objects":5210,"num_object_clones":0,"num_object_copies":20840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7460,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2987}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7425","reported_seq":7496,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.622069+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:05.622069+0000","last_peered":"2024-09-16T04:46:05.622069+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:05.622069+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7425,"log_dups_size":0,"ondisk_log_size":7425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42964236,"num_objects":5245,"num_object_clones":0,"num_object_copies":20980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7423,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3018}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7490","reported_seq":7579,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.668187+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:05.668187+0000","last_peered":"2024-09-16T04:46:05.668187+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:05.668187+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43013388,"num_objects":5251,"num_object_clones":0,"num_object_copies":21004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7233","reported_seq":7302,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.831404+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:05.831404+0000","last_peered":"2024-09-16T04:46:05.831404+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:05.831404+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7233,"log_dups_size":0,"ondisk_log_size":7233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41607168,"num_objects":5079,"num_object_clones":0,"num_object_copies":20316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7233,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7375","reported_seq":7445,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.325850+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:05.325850+0000","last_peered":"2024-09-16T04:46:05.325850+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:05.325850+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7375,"log_dups_size":0,"ondisk_log_size":7375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42800396,"num_objects":5225,"num_object_clones":0,"num_object_copies":20900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7373,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3001}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7537","reported_seq":12093,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:07.452852+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:07.452852+0000","last_peered":"2024-09-16T04:46:07.452852+0000","last_clean":"2024-09-16T04:46:07.452852+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:07.452852+0000","last_undegraded":"2024-09-16T04:46:07.452852+0000","last_fullsized":"2024-09-16T04:46:07.452852+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:10.439959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42988812,"num_objects":5248,"num_object_clones":0,"num_object_copies":20992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7535,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7543","reported_seq":9822,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.922621+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:05.922621+0000","last_peered":"2024-09-16T04:46:05.922621+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:05.922621+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42699288,"num_objects":5213,"num_object_clones":0,"num_object_copies":20852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7540,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1113,"num_bytes_recovered":9043968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3983}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7537","reported_seq":7634,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.463764+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:10.463764+0000","last_peered":"2024-09-16T04:46:10.463764+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:10.463764+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42767628,"num_objects":5221,"num_object_clones":0,"num_object_copies":20884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7535,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3030}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7316","reported_seq":7397,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.721195+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:05.721195+0000","last_peered":"2024-09-16T04:46:05.721195+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:05.721195+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42389394,"num_objects":5175,"num_object_clones":0,"num_object_copies":20700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7313,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7183","reported_seq":7239,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.844717+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:05.844717+0000","last_peered":"2024-09-16T04:46:05.844717+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:05.844717+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42171014,"num_objects":5148,"num_object_clones":0,"num_object_copies":20592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7182,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3029}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7413","reported_seq":7494,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.475047+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:10.475047+0000","last_peered":"2024-09-16T04:46:10.475047+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:10.475047+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7413,"log_dups_size":0,"ondisk_log_size":7413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41941638,"num_objects":5120,"num_object_clones":0,"num_object_copies":20480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2898}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7464","reported_seq":7561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.856207+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:05.856207+0000","last_peered":"2024-09-16T04:46:05.856207+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:05.856207+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7464,"log_dups_size":0,"ondisk_log_size":7464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43562252,"num_objects":5318,"num_object_clones":0,"num_object_copies":21272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7462,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7355","reported_seq":7444,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.407989+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:10.407989+0000","last_peered":"2024-09-16T04:46:10.407989+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:10.407989+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7355,"log_dups_size":0,"ondisk_log_size":7355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41727244,"num_objects":5094,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2967}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7242","reported_seq":7315,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:05.637002+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:05.637002+0000","last_peered":"2024-09-16T04:46:05.637002+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:05.637002+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7242,"log_dups_size":0,"ondisk_log_size":7242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41489676,"num_objects":5065,"num_object_clones":0,"num_object_copies":20260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":681100992,"num_objects":83148,"num_object_clones":0,"num_object_copies":332592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83148,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118339,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3491,"num_bytes_recovered":27377664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1292369920,"data_stored":1292369920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118366,"ondisk_log_size":118366,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749708,"kb_used_data":334844,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93622132,"statfs":{"total":96636764160,"available":95869063168,"internally_reserved":0,"allocated":342880256,"data_stored":342700287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424799951},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,6,3,1,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744612,"kb_used_data":330260,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93627228,"statfs":{"total":96636764160,"available":95874281472,"internally_reserved":0,"allocated":338186240,"data_stored":338006271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,0,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545744,"kb_used_data":265536,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93826096,"statfs":{"total":96636764160,"available":96077922304,"internally_reserved":0,"allocated":271908864,"data_stored":271728895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":286912207},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753364,"kb_used_data":334404,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93618476,"statfs":{"total":96636764160,"available":95865319424,"internally_reserved":0,"allocated":342429696,"data_stored":342249727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":428994254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":341794816,"data_stored":341794816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":271605760,"data_stored":271605760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":337379328,"data_stored":337379328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":341590016,"data_stored":341590016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:12.464 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:13.082 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:13.082 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:13.111 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-09-16T04:46:12.659689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":674686656,"num_objects":82365,"num_object_clones":0,"num_object_copies":329460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119122,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3553,"num_bytes_recovered":27860992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119149,"ondisk_log_size":119149,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2776048,"kb_used_data":1255976,"kb_used_omap":14,"kb_used_meta":1519985,"kb_avail":374711312,"statfs":{"total":386547056640,"available":383704383488,"internally_reserved":0,"allocated":1286119424,"data_stored":1285399548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1556465201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,8,2,12,12,8,1,0,0,0,0,3,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25518080,"num_objects":-3115,"num_object_clones":0,"num_object_copies":-12460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3115,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1474560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004673"},"pg_stats":[{"pgid":"1.f","version":"33'7480","reported_seq":7551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.879485+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:10.879485+0000","last_peered":"2024-09-16T04:46:10.879485+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:10.879485+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7480,"log_dups_size":0,"ondisk_log_size":7480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42385408,"num_objects":5174,"num_object_clones":0,"num_object_copies":20696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7480,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7463","reported_seq":7536,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.699430+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:10.699430+0000","last_peered":"2024-09-16T04:46:10.699430+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:10.699430+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7463,"log_dups_size":0,"ondisk_log_size":7463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42685708,"num_objects":5211,"num_object_clones":0,"num_object_copies":20844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7461,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3003}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7461","reported_seq":7545,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.465678+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:10.465678+0000","last_peered":"2024-09-16T04:46:10.465678+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:10.465678+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7461,"log_dups_size":0,"ondisk_log_size":7461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42670750,"num_objects":5210,"num_object_clones":0,"num_object_copies":20840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7460,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2987}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7480","reported_seq":7551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.642834+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:10.642834+0000","last_peered":"2024-09-16T04:46:10.642834+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:10.642834+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7480,"log_dups_size":0,"ondisk_log_size":7480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42513676,"num_objects":5190,"num_object_clones":0,"num_object_copies":20760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7478,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2963}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7567","reported_seq":7656,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.837838+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:10.837838+0000","last_peered":"2024-09-16T04:46:10.837838+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:10.837838+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7567,"log_dups_size":0,"ondisk_log_size":7567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42382604,"num_objects":5174,"num_object_clones":0,"num_object_copies":20696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7565,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7299","reported_seq":7368,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.683330+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:10.683330+0000","last_peered":"2024-09-16T04:46:10.683330+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:10.683330+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7299,"log_dups_size":0,"ondisk_log_size":7299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41066496,"num_objects":5013,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7299,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7440","reported_seq":7510,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.726387+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:10.726387+0000","last_peered":"2024-09-16T04:46:10.726387+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:10.726387+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7440,"log_dups_size":0,"ondisk_log_size":7440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42267916,"num_objects":5160,"num_object_clones":0,"num_object_copies":20640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7438,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2936}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7537","reported_seq":12093,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:07.452852+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:07.452852+0000","last_peered":"2024-09-16T04:46:07.452852+0000","last_clean":"2024-09-16T04:46:07.452852+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:07.452852+0000","last_undegraded":"2024-09-16T04:46:07.452852+0000","last_fullsized":"2024-09-16T04:46:07.452852+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:10.439959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42988812,"num_objects":5248,"num_object_clones":0,"num_object_copies":20992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7535,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7615","reported_seq":10012,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.947265+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:10.947265+0000","last_peered":"2024-09-16T04:46:10.947265+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:10.947265+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7615,"log_dups_size":0,"ondisk_log_size":7615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42109464,"num_objects":5141,"num_object_clones":0,"num_object_copies":20564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7612,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1172,"num_bytes_recovered":9527296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7537","reported_seq":7634,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.463764+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:10.463764+0000","last_peered":"2024-09-16T04:46:10.463764+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:10.463764+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42767628,"num_objects":5221,"num_object_clones":0,"num_object_copies":20884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7535,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3030}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7399","reported_seq":7488,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.877348+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:10.877348+0000","last_peered":"2024-09-16T04:46:10.877348+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:10.877348+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7399,"log_dups_size":0,"ondisk_log_size":7399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41709458,"num_objects":5092,"num_object_clones":0,"num_object_copies":20368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7396,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7264","reported_seq":7323,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.772242+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:10.772242+0000","last_peered":"2024-09-16T04:46:10.772242+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:10.772242+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7264,"log_dups_size":0,"ondisk_log_size":7264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41507462,"num_objects":5067,"num_object_clones":0,"num_object_copies":20268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7263,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2948}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7413","reported_seq":7494,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.475047+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:10.475047+0000","last_peered":"2024-09-16T04:46:10.475047+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:10.475047+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7413,"log_dups_size":0,"ondisk_log_size":7413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41941638,"num_objects":5120,"num_object_clones":0,"num_object_copies":20480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2898}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7531","reported_seq":7628,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.907579+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:10.907579+0000","last_peered":"2024-09-16T04:46:10.907579+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:10.907579+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7531,"log_dups_size":0,"ondisk_log_size":7531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43013388,"num_objects":5251,"num_object_clones":0,"num_object_copies":21004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7529,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3038}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7355","reported_seq":7444,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.407989+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:10.407989+0000","last_peered":"2024-09-16T04:46:10.407989+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:10.407989+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7355,"log_dups_size":0,"ondisk_log_size":7355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41727244,"num_objects":5094,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2967}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7308","reported_seq":7381,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.450731+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:10.450731+0000","last_peered":"2024-09-16T04:46:10.450731+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:10.450731+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7308,"log_dups_size":0,"ondisk_log_size":7308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40949004,"num_objects":4999,"num_object_clones":0,"num_object_copies":19996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7306,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":674686656,"num_objects":82365,"num_object_clones":0,"num_object_copies":329460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119122,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3553,"num_bytes_recovered":27860992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1283072000,"data_stored":1283072000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119149,"ondisk_log_size":119149,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749300,"kb_used_data":330276,"kb_used_omap":4,"kb_used_meta":419003,"kb_avail":93622540,"statfs":{"total":96636764160,"available":95869480960,"internally_reserved":0,"allocated":338202624,"data_stored":338022655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":429059791},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,1,4,3,4,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744612,"kb_used_data":330260,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93627228,"statfs":{"total":96636764160,"available":95874281472,"internally_reserved":0,"allocated":338186240,"data_stored":338006271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,0,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545744,"kb_used_data":265536,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93826096,"statfs":{"total":96636764160,"available":96077922304,"internally_reserved":0,"allocated":271908864,"data_stored":271728895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":286912207},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736392,"kb_used_data":329904,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93635448,"statfs":{"total":96636764160,"available":95882698752,"internally_reserved":0,"allocated":337821696,"data_stored":337641727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":416214734},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,4,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":337154048,"data_stored":337154048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":271605760,"data_stored":271605760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":337379328,"data_stored":337379328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":336932864,"data_stored":336932864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:13.112 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:46:13.112 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7480", 'reported_seq': 7551, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:10.879485+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:46:10.879485+0000', 'last_peered': '2024-09-16T04:46:10.879485+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:46:10.879485+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7480, 'log_dups_size': 0, 'ondisk_log_size': 7480, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42385408, 'num_objects': 5174, 'num_object_clones': 0, 'num_object_copies': 20696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2206, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7480, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2968}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:13.112 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:46:13.113 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7463", 'reported_seq': 7536, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:10.699430+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:46:10.699430+0000', 'last_peered': '2024-09-16T04:46:10.699430+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:46:10.699430+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7463, 'log_dups_size': 0, 'ondisk_log_size': 7463, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42685708, 'num_objects': 5211, 'num_object_clones': 0, 'num_object_copies': 20844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2210, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5211, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7461, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3003}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:13.113 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:46:13.113 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7461", 'reported_seq': 7545, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:10.465678+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:46:10.465678+0000', 'last_peered': '2024-09-16T04:46:10.465678+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:46:10.465678+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7461, 'log_dups_size': 0, 'ondisk_log_size': 7461, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42670750, 'num_objects': 5210, 'num_object_clones': 0, 'num_object_copies': 20840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5210, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7460, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2987}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:13.113 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:46:13.113 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7480", 'reported_seq': 7551, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:10.642834+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:46:10.642834+0000', 'last_peered': '2024-09-16T04:46:10.642834+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:46:10.642834+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7480, 'log_dups_size': 0, 'ondisk_log_size': 7480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42513676, 'num_objects': 5190, 'num_object_clones': 0, 'num_object_copies': 20760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5190, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7478, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2963}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:13.113 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:46:13.113 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7567", 'reported_seq': 7656, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:10.837838+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:46:10.837838+0000', 'last_peered': '2024-09-16T04:46:10.837838+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:46:10.837838+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7567, 'log_dups_size': 0, 'ondisk_log_size': 7567, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42382604, 'num_objects': 5174, 'num_object_clones': 0, 'num_object_copies': 20696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7565, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2907}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:13.113 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:46:13.113 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7299", 'reported_seq': 7368, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:10.683330+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:46:10.683330+0000', 'last_peered': '2024-09-16T04:46:10.683330+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:46:10.683330+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7299, 'log_dups_size': 0, 'ondisk_log_size': 7299, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41066496, 'num_objects': 5013, 'num_object_clones': 0, 'num_object_copies': 20052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2196, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7299, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2817}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:13.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:46:13.114 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7440", 'reported_seq': 7510, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:10.726387+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:46:10.726387+0000', 'last_peered': '2024-09-16T04:46:10.726387+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:46:10.726387+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7440, 'log_dups_size': 0, 'ondisk_log_size': 7440, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42267916, 'num_objects': 5160, 'num_object_clones': 0, 'num_object_copies': 20640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5160, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7438, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2936}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:13.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:46:13.114 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7615", 'reported_seq': 10012, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:10.947265+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:46:10.947265+0000', 'last_peered': '2024-09-16T04:46:10.947265+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:46:10.947265+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7615, 'log_dups_size': 0, 'ondisk_log_size': 7615, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42109464, 'num_objects': 5141, 'num_object_clones': 0, 'num_object_copies': 20564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1171, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7612, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1172, 'num_bytes_recovered': 9527296, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3970}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:13.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:46:13.114 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7537", 'reported_seq': 7634, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:10.463764+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:46:10.463764+0000', 'last_peered': '2024-09-16T04:46:10.463764+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:46:10.463764+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7537, 'log_dups_size': 0, 'ondisk_log_size': 7537, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42767628, 'num_objects': 5221, 'num_object_clones': 0, 'num_object_copies': 20884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2193, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5221, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7535, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3030}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:13.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:46:13.114 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7399", 'reported_seq': 7488, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:10.877348+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:46:10.877348+0000', 'last_peered': '2024-09-16T04:46:10.877348+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:46:10.877348+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7399, 'log_dups_size': 0, 'ondisk_log_size': 7399, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41709458, 'num_objects': 5092, 'num_object_clones': 0, 'num_object_copies': 20368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2177, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5092, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7396, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2918}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:13.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:46:13.115 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7264", 'reported_seq': 7323, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:10.772242+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:46:10.772242+0000', 'last_peered': '2024-09-16T04:46:10.772242+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:46:10.772242+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7264, 'log_dups_size': 0, 'ondisk_log_size': 7264, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41507462, 'num_objects': 5067, 'num_object_clones': 0, 'num_object_copies': 20268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2120, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5067, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7263, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2948}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:13.115 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:46:13.115 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7413", 'reported_seq': 7494, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:10.475047+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:46:10.475047+0000', 'last_peered': '2024-09-16T04:46:10.475047+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:46:10.475047+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7413, 'log_dups_size': 0, 'ondisk_log_size': 7413, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41941638, 'num_objects': 5120, 'num_object_clones': 0, 'num_object_copies': 20480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7412, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2898}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:13.115 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:46:13.115 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7531", 'reported_seq': 7628, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:10.907579+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:46:10.907579+0000', 'last_peered': '2024-09-16T04:46:10.907579+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:46:10.907579+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7531, 'log_dups_size': 0, 'ondisk_log_size': 7531, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43013388, 'num_objects': 5251, 'num_object_clones': 0, 'num_object_copies': 21004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7529, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3038}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:13.115 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:46:13.115 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7355", 'reported_seq': 7444, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:10.407989+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:46:10.407989+0000', 'last_peered': '2024-09-16T04:46:10.407989+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:46:10.407989+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7355, 'log_dups_size': 0, 'ondisk_log_size': 7355, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41727244, 'num_objects': 5094, 'num_object_clones': 0, 'num_object_copies': 20376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5094, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7353, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2967}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:13.115 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:46:13.115 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7308", 'reported_seq': 7381, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:10.450731+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:46:10.450731+0000', 'last_peered': '2024-09-16T04:46:10.450731+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:46:10.450731+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7308, 'log_dups_size': 0, 'ondisk_log_size': 7308, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40949004, 'num_objects': 4999, 'num_object_clones': 0, 'num_object_copies': 19996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7306, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2763}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:13.116 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:46:13.871 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:13.871 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32026/329460 objects degraded (9.721%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":386,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":82365,"data_bytes":674686656,"bytes_used":2842673152,"bytes_avail":383704383488,"bytes_total":386547056640,"degraded_objects":32026,"degraded_total":329460,"degraded_ratio":0.097207551751350693,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":122832,"recovering_keys_per_sec":0,"num_objects_recovered":209,"num_bytes_recovered":1474560,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":259},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:46:13.902 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:46:13.902 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:14.517 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:14.518 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:14.545 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-09-16T04:46:12.659689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":674686656,"num_objects":82365,"num_object_clones":0,"num_object_copies":329460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119122,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3553,"num_bytes_recovered":27860992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119149,"ondisk_log_size":119149,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2776048,"kb_used_data":1255976,"kb_used_omap":14,"kb_used_meta":1519985,"kb_avail":374711312,"statfs":{"total":386547056640,"available":383704383488,"internally_reserved":0,"allocated":1286119424,"data_stored":1285399548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1556465201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,8,2,12,12,8,1,0,0,0,0,3,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25518080,"num_objects":-3115,"num_object_clones":0,"num_object_copies":-12460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3115,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1474560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004673"},"pg_stats":[{"pgid":"1.f","version":"33'7480","reported_seq":7551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.879485+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:10.879485+0000","last_peered":"2024-09-16T04:46:10.879485+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:10.879485+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7480,"log_dups_size":0,"ondisk_log_size":7480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42385408,"num_objects":5174,"num_object_clones":0,"num_object_copies":20696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7480,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7463","reported_seq":7536,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.699430+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:10.699430+0000","last_peered":"2024-09-16T04:46:10.699430+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:10.699430+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7463,"log_dups_size":0,"ondisk_log_size":7463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42685708,"num_objects":5211,"num_object_clones":0,"num_object_copies":20844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7461,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3003}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7461","reported_seq":7545,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.465678+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:10.465678+0000","last_peered":"2024-09-16T04:46:10.465678+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:10.465678+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7461,"log_dups_size":0,"ondisk_log_size":7461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42670750,"num_objects":5210,"num_object_clones":0,"num_object_copies":20840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7460,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2987}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7480","reported_seq":7551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.642834+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:10.642834+0000","last_peered":"2024-09-16T04:46:10.642834+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:10.642834+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7480,"log_dups_size":0,"ondisk_log_size":7480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42513676,"num_objects":5190,"num_object_clones":0,"num_object_copies":20760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7478,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2963}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7567","reported_seq":7656,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.837838+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:10.837838+0000","last_peered":"2024-09-16T04:46:10.837838+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:10.837838+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7567,"log_dups_size":0,"ondisk_log_size":7567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42382604,"num_objects":5174,"num_object_clones":0,"num_object_copies":20696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7565,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7299","reported_seq":7368,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.683330+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:10.683330+0000","last_peered":"2024-09-16T04:46:10.683330+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:10.683330+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7299,"log_dups_size":0,"ondisk_log_size":7299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41066496,"num_objects":5013,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7299,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7440","reported_seq":7510,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.726387+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:10.726387+0000","last_peered":"2024-09-16T04:46:10.726387+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:10.726387+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7440,"log_dups_size":0,"ondisk_log_size":7440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42267916,"num_objects":5160,"num_object_clones":0,"num_object_copies":20640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7438,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2936}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7537","reported_seq":12093,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:07.452852+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:07.452852+0000","last_peered":"2024-09-16T04:46:07.452852+0000","last_clean":"2024-09-16T04:46:07.452852+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:07.452852+0000","last_undegraded":"2024-09-16T04:46:07.452852+0000","last_fullsized":"2024-09-16T04:46:07.452852+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:10.439959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42988812,"num_objects":5248,"num_object_clones":0,"num_object_copies":20992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7535,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7615","reported_seq":10012,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.947265+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:10.947265+0000","last_peered":"2024-09-16T04:46:10.947265+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:10.947265+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7615,"log_dups_size":0,"ondisk_log_size":7615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42109464,"num_objects":5141,"num_object_clones":0,"num_object_copies":20564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7612,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1172,"num_bytes_recovered":9527296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7537","reported_seq":7634,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.463764+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:10.463764+0000","last_peered":"2024-09-16T04:46:10.463764+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:10.463764+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42767628,"num_objects":5221,"num_object_clones":0,"num_object_copies":20884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7535,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3030}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7399","reported_seq":7488,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.877348+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:10.877348+0000","last_peered":"2024-09-16T04:46:10.877348+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:10.877348+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7399,"log_dups_size":0,"ondisk_log_size":7399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41709458,"num_objects":5092,"num_object_clones":0,"num_object_copies":20368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7396,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7264","reported_seq":7323,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.772242+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:10.772242+0000","last_peered":"2024-09-16T04:46:10.772242+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:10.772242+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7264,"log_dups_size":0,"ondisk_log_size":7264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41507462,"num_objects":5067,"num_object_clones":0,"num_object_copies":20268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7263,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2948}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7413","reported_seq":7494,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.475047+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:10.475047+0000","last_peered":"2024-09-16T04:46:10.475047+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:10.475047+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7413,"log_dups_size":0,"ondisk_log_size":7413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41941638,"num_objects":5120,"num_object_clones":0,"num_object_copies":20480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2898}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7531","reported_seq":7628,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.907579+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:10.907579+0000","last_peered":"2024-09-16T04:46:10.907579+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:10.907579+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7531,"log_dups_size":0,"ondisk_log_size":7531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43013388,"num_objects":5251,"num_object_clones":0,"num_object_copies":21004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7529,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3038}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7355","reported_seq":7444,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.407989+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:10.407989+0000","last_peered":"2024-09-16T04:46:10.407989+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:10.407989+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7355,"log_dups_size":0,"ondisk_log_size":7355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41727244,"num_objects":5094,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7353,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2967}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7308","reported_seq":7381,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:10.450731+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:10.450731+0000","last_peered":"2024-09-16T04:46:10.450731+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:10.450731+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7308,"log_dups_size":0,"ondisk_log_size":7308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40949004,"num_objects":4999,"num_object_clones":0,"num_object_copies":19996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7306,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":674686656,"num_objects":82365,"num_object_clones":0,"num_object_copies":329460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119122,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3553,"num_bytes_recovered":27860992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1283072000,"data_stored":1283072000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119149,"ondisk_log_size":119149,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749300,"kb_used_data":330276,"kb_used_omap":4,"kb_used_meta":419003,"kb_avail":93622540,"statfs":{"total":96636764160,"available":95869480960,"internally_reserved":0,"allocated":338202624,"data_stored":338022655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":429059791},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,1,4,3,4,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744612,"kb_used_data":330260,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93627228,"statfs":{"total":96636764160,"available":95874281472,"internally_reserved":0,"allocated":338186240,"data_stored":338006271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,0,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545744,"kb_used_data":265536,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93826096,"statfs":{"total":96636764160,"available":96077922304,"internally_reserved":0,"allocated":271908864,"data_stored":271728895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":286912207},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736392,"kb_used_data":329904,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93635448,"statfs":{"total":96636764160,"available":95882698752,"internally_reserved":0,"allocated":337821696,"data_stored":337641727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":416214734},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,4,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":337154048,"data_stored":337154048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":271605760,"data_stored":271605760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":337379328,"data_stored":337379328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":336932864,"data_stored":336932864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:17.547 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:18.154 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:18.154 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:18.189 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-16T04:46:16.660709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":665003712,"num_objects":81183,"num_object_clones":0,"num_object_copies":324732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81183,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120304,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3606,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120331,"ondisk_log_size":120331,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2819848,"kb_used_data":1241476,"kb_used_omap":14,"kb_used_meta":1578289,"kb_avail":374667512,"statfs":{"total":386547056640,"available":383659532288,"internally_reserved":0,"allocated":1271271424,"data_stored":1270551548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1616168497},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,7,8,6,14,6,3,4,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28024832,"num_objects":-3421,"num_object_clones":0,"num_object_copies":-13684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003038"},"pg_stats":[{"pgid":"1.f","version":"33'7544","reported_seq":7619,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.916969+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:15.916969+0000","last_peered":"2024-09-16T04:46:15.916969+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:15.916969+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7544,"log_dups_size":0,"ondisk_log_size":7544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41861120,"num_objects":5110,"num_object_clones":0,"num_object_copies":20440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7544,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7549","reported_seq":7622,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.611275+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:15.611275+0000","last_peered":"2024-09-16T04:46:15.611275+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:15.611275+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7549,"log_dups_size":0,"ondisk_log_size":7549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41981196,"num_objects":5125,"num_object_clones":0,"num_object_copies":20500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7547,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2917}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7537","reported_seq":7622,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.475996+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:15.475996+0000","last_peered":"2024-09-16T04:46:15.475996+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:15.475996+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42048158,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7536,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2911}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7551","reported_seq":7622,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.675212+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:15.675212+0000","last_peered":"2024-09-16T04:46:15.675212+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:15.675212+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7551,"log_dups_size":0,"ondisk_log_size":7551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41932044,"num_objects":5119,"num_object_clones":0,"num_object_copies":20476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7549,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7648","reported_seq":7739,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.900108+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:15.900108+0000","last_peered":"2024-09-16T04:46:15.900108+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:15.900108+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7648,"log_dups_size":0,"ondisk_log_size":7648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41719052,"num_objects":5093,"num_object_clones":0,"num_object_copies":20372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7646,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2827}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7366","reported_seq":7437,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.864987+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:15.864987+0000","last_peered":"2024-09-16T04:46:15.864987+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:15.864987+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7366,"log_dups_size":0,"ondisk_log_size":7366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40517632,"num_objects":4946,"num_object_clones":0,"num_object_copies":19784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7366,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7511","reported_seq":7581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.636287+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:15.636287+0000","last_peered":"2024-09-16T04:46:15.636287+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:15.636287+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7511,"log_dups_size":0,"ondisk_log_size":7511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41686284,"num_objects":5089,"num_object_clones":0,"num_object_copies":20356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7509,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2865}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7608","reported_seq":12164,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:12.330937+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:12.330937+0000","last_peered":"2024-09-16T04:46:12.330937+0000","last_clean":"2024-09-16T04:46:12.330937+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:12.330937+0000","last_undegraded":"2024-09-16T04:46:12.330937+0000","last_fullsized":"2024-09-16T04:46:12.330937+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7608,"log_dups_size":0,"ondisk_log_size":7608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:16.260366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42407180,"num_objects":5177,"num_object_clones":0,"num_object_copies":20708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7606,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7694","reported_seq":10175,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.906572+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:15.906572+0000","last_peered":"2024-09-16T04:46:15.906572+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:15.906572+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7694,"log_dups_size":0,"ondisk_log_size":7694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41462296,"num_objects":5062,"num_object_clones":0,"num_object_copies":20248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7691,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1214,"num_bytes_recovered":9846784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3930}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7615","reported_seq":7712,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.473355+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:15.473355+0000","last_peered":"2024-09-16T04:46:15.473355+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:15.473355+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7615,"log_dups_size":0,"ondisk_log_size":7615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42128652,"num_objects":5143,"num_object_clones":0,"num_object_copies":20572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2952}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7475","reported_seq":7572,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.841454+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:15.841454+0000","last_peered":"2024-09-16T04:46:15.841454+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:15.841454+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7475,"log_dups_size":0,"ondisk_log_size":7475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41086866,"num_objects":5016,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7472,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7339","reported_seq":7402,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.937855+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:15.937855+0000","last_peered":"2024-09-16T04:46:15.937855+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:15.937855+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7339,"log_dups_size":0,"ondisk_log_size":7339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40893062,"num_objects":4992,"num_object_clones":0,"num_object_copies":19968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7338,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2875}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7481","reported_seq":7561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.374110+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:15.374110+0000","last_peered":"2024-09-16T04:46:15.374110+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:15.374110+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7481,"log_dups_size":0,"ondisk_log_size":7481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41384582,"num_objects":5052,"num_object_clones":0,"num_object_copies":20208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7480,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7608","reported_seq":7707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.781030+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:15.781030+0000","last_peered":"2024-09-16T04:46:15.781030+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:15.781030+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7608,"log_dups_size":0,"ondisk_log_size":7608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42382604,"num_objects":5174,"num_object_clones":0,"num_object_copies":20696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7606,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2962}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7436","reported_seq":7525,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.484444+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:15.484444+0000","last_peered":"2024-09-16T04:46:15.484444+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:15.484444+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7436,"log_dups_size":0,"ondisk_log_size":7436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41063692,"num_objects":5013,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7434,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7369","reported_seq":7442,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.611538+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:15.611538+0000","last_peered":"2024-09-16T04:46:15.611538+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:15.611538+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7369,"log_dups_size":0,"ondisk_log_size":7369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40449292,"num_objects":4938,"num_object_clones":0,"num_object_copies":19752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7367,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2702}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":665003712,"num_objects":81183,"num_object_clones":0,"num_object_copies":324732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81183,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120304,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3606,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1268330496,"data_stored":1268330496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120331,"ondisk_log_size":120331,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750120,"kb_used_data":325528,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93621720,"statfs":{"total":96636764160,"available":95868641280,"internally_reserved":0,"allocated":333340672,"data_stored":333160703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434761423},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,1,2,7,3,1,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762084,"kb_used_data":325520,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93609756,"statfs":{"total":96636764160,"available":95856390144,"internally_reserved":0,"allocated":333332480,"data_stored":333152511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051642,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553788,"kb_used_data":265260,"kb_used_omap":4,"kb_used_meta":288507,"kb_avail":93818052,"statfs":{"total":96636764160,"available":96069685248,"internally_reserved":0,"allocated":271626240,"data_stored":271446271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295431887},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,7,1,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753856,"kb_used_data":325168,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93617984,"statfs":{"total":96636764160,"available":95864815616,"internally_reserved":0,"allocated":332972032,"data_stored":332792063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":438955726},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,4,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":332328960,"data_stored":332328960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":271343616,"data_stored":271343616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":332578816,"data_stored":332578816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":332079104,"data_stored":332079104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:18.191 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:18.782 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:18.783 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:18.808 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-16T04:46:18.661307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664438464,"num_objects":81114,"num_object_clones":0,"num_object_copies":324456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81114,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120373,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3606,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120400,"ondisk_log_size":120400,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2823796,"kb_used_data":1241264,"kb_used_omap":14,"kb_used_meta":1582449,"kb_avail":374663564,"statfs":{"total":386547056640,"available":383655489536,"internally_reserved":0,"allocated":1271054336,"data_stored":1270334460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1620428337},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,7,3,5,16,5,3,4,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19685376,"num_objects":-2403,"num_object_clones":0,"num_object_copies":-9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2403,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003183"},"pg_stats":[{"pgid":"1.f","version":"33'7544","reported_seq":7619,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.916969+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:15.916969+0000","last_peered":"2024-09-16T04:46:15.916969+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:15.916969+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7544,"log_dups_size":0,"ondisk_log_size":7544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41861120,"num_objects":5110,"num_object_clones":0,"num_object_copies":20440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7544,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7549","reported_seq":7622,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.611275+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:15.611275+0000","last_peered":"2024-09-16T04:46:15.611275+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:15.611275+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7549,"log_dups_size":0,"ondisk_log_size":7549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41981196,"num_objects":5125,"num_object_clones":0,"num_object_copies":20500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7547,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2917}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7537","reported_seq":7622,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.475996+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:15.475996+0000","last_peered":"2024-09-16T04:46:15.475996+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:15.475996+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42048158,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7536,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2911}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7551","reported_seq":7622,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.675212+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:15.675212+0000","last_peered":"2024-09-16T04:46:15.675212+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:15.675212+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7551,"log_dups_size":0,"ondisk_log_size":7551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41932044,"num_objects":5119,"num_object_clones":0,"num_object_copies":20476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7549,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7648","reported_seq":7739,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.900108+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:15.900108+0000","last_peered":"2024-09-16T04:46:15.900108+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:15.900108+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7648,"log_dups_size":0,"ondisk_log_size":7648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41719052,"num_objects":5093,"num_object_clones":0,"num_object_copies":20372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7646,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2827}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7366","reported_seq":7437,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.864987+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:15.864987+0000","last_peered":"2024-09-16T04:46:15.864987+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:15.864987+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7366,"log_dups_size":0,"ondisk_log_size":7366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40517632,"num_objects":4946,"num_object_clones":0,"num_object_copies":19784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7366,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7511","reported_seq":7581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.636287+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:15.636287+0000","last_peered":"2024-09-16T04:46:15.636287+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:15.636287+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7511,"log_dups_size":0,"ondisk_log_size":7511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41686284,"num_objects":5089,"num_object_clones":0,"num_object_copies":20356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7509,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2865}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7677","reported_seq":12234,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:17.697932+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:17.697932+0000","last_peered":"2024-09-16T04:46:17.697932+0000","last_clean":"2024-09-16T04:46:17.697932+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:17.697932+0000","last_undegraded":"2024-09-16T04:46:17.697932+0000","last_fullsized":"2024-09-16T04:46:17.697932+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7677,"log_dups_size":0,"ondisk_log_size":7677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:21.276340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41841932,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7694","reported_seq":10175,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.906572+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:15.906572+0000","last_peered":"2024-09-16T04:46:15.906572+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:15.906572+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7694,"log_dups_size":0,"ondisk_log_size":7694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41462296,"num_objects":5062,"num_object_clones":0,"num_object_copies":20248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7691,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1214,"num_bytes_recovered":9846784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3930}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7615","reported_seq":7712,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.473355+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:15.473355+0000","last_peered":"2024-09-16T04:46:15.473355+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:15.473355+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7615,"log_dups_size":0,"ondisk_log_size":7615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42128652,"num_objects":5143,"num_object_clones":0,"num_object_copies":20572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2952}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7475","reported_seq":7572,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.841454+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:15.841454+0000","last_peered":"2024-09-16T04:46:15.841454+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:15.841454+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7475,"log_dups_size":0,"ondisk_log_size":7475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41086866,"num_objects":5016,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7472,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7339","reported_seq":7402,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.937855+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:15.937855+0000","last_peered":"2024-09-16T04:46:15.937855+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:15.937855+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7339,"log_dups_size":0,"ondisk_log_size":7339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40893062,"num_objects":4992,"num_object_clones":0,"num_object_copies":19968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7338,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2875}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7481","reported_seq":7561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.374110+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:15.374110+0000","last_peered":"2024-09-16T04:46:15.374110+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:15.374110+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7481,"log_dups_size":0,"ondisk_log_size":7481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41384582,"num_objects":5052,"num_object_clones":0,"num_object_copies":20208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7480,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7608","reported_seq":7707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.781030+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:15.781030+0000","last_peered":"2024-09-16T04:46:15.781030+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:15.781030+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7608,"log_dups_size":0,"ondisk_log_size":7608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42382604,"num_objects":5174,"num_object_clones":0,"num_object_copies":20696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7606,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2962}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7436","reported_seq":7525,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.484444+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:15.484444+0000","last_peered":"2024-09-16T04:46:15.484444+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:15.484444+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7436,"log_dups_size":0,"ondisk_log_size":7436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41063692,"num_objects":5013,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7434,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7369","reported_seq":7442,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.611538+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:15.611538+0000","last_peered":"2024-09-16T04:46:15.611538+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:15.611538+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7369,"log_dups_size":0,"ondisk_log_size":7369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40449292,"num_objects":4938,"num_object_clones":0,"num_object_copies":19752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7367,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2702}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":664438464,"num_objects":81114,"num_object_clones":0,"num_object_copies":324456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81114,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120373,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3606,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1268088832,"data_stored":1268088832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120400,"ondisk_log_size":120400,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750120,"kb_used_data":325528,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93621720,"statfs":{"total":96636764160,"available":95868641280,"internally_reserved":0,"allocated":333340672,"data_stored":333160703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434761423},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,1,2,7,3,1,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762084,"kb_used_data":325520,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93609756,"statfs":{"total":96636764160,"available":95856390144,"internally_reserved":0,"allocated":333332480,"data_stored":333152511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557736,"kb_used_data":265048,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93814104,"statfs":{"total":96636764160,"available":96065642496,"internally_reserved":0,"allocated":271409152,"data_stored":271229183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299691727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753856,"kb_used_data":325168,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93617984,"statfs":{"total":96636764160,"available":95864815616,"internally_reserved":0,"allocated":332972032,"data_stored":332792063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":438955726},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,4,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":332328960,"data_stored":332328960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":271101952,"data_stored":271101952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":332578816,"data_stored":332578816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":332079104,"data_stored":332079104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:18.809 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:46:18.809 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7544", 'reported_seq': 7619, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:15.916969+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:46:15.916969+0000', 'last_peered': '2024-09-16T04:46:15.916969+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:46:15.916969+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7544, 'log_dups_size': 0, 'ondisk_log_size': 7544, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41861120, 'num_objects': 5110, 'num_object_clones': 0, 'num_object_copies': 20440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2204, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5110, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7544, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2906}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:18.810 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:46:18.810 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7549", 'reported_seq': 7622, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:15.611275+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:46:15.611275+0000', 'last_peered': '2024-09-16T04:46:15.611275+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:46:15.611275+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7549, 'log_dups_size': 0, 'ondisk_log_size': 7549, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41981196, 'num_objects': 5125, 'num_object_clones': 0, 'num_object_copies': 20500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2210, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5125, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7547, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2917}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:18.810 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:46:18.810 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7537", 'reported_seq': 7622, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:15.475996+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:46:15.475996+0000', 'last_peered': '2024-09-16T04:46:15.475996+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:46:15.475996+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7537, 'log_dups_size': 0, 'ondisk_log_size': 7537, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42048158, 'num_objects': 5134, 'num_object_clones': 0, 'num_object_copies': 20536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5134, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7536, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2911}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:18.810 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:46:18.810 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7551", 'reported_seq': 7622, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:15.675212+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:46:15.675212+0000', 'last_peered': '2024-09-16T04:46:15.675212+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:46:15.675212+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7551, 'log_dups_size': 0, 'ondisk_log_size': 7551, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41932044, 'num_objects': 5119, 'num_object_clones': 0, 'num_object_copies': 20476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7549, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2892}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:18.810 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:46:18.810 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7648", 'reported_seq': 7739, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:15.900108+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:46:15.900108+0000', 'last_peered': '2024-09-16T04:46:15.900108+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:46:15.900108+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7648, 'log_dups_size': 0, 'ondisk_log_size': 7648, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41719052, 'num_objects': 5093, 'num_object_clones': 0, 'num_object_copies': 20372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2268, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7646, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2827}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:18.810 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:46:18.810 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7366", 'reported_seq': 7437, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:15.864987+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:46:15.864987+0000', 'last_peered': '2024-09-16T04:46:15.864987+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:46:15.864987+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7366, 'log_dups_size': 0, 'ondisk_log_size': 7366, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40517632, 'num_objects': 4946, 'num_object_clones': 0, 'num_object_copies': 19784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2195, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4946, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7366, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2751}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:18.811 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:46:18.811 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7511", 'reported_seq': 7581, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:15.636287+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:46:15.636287+0000', 'last_peered': '2024-09-16T04:46:15.636287+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:46:15.636287+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7511, 'log_dups_size': 0, 'ondisk_log_size': 7511, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41686284, 'num_objects': 5089, 'num_object_clones': 0, 'num_object_copies': 20356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5089, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7509, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2865}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:18.811 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:46:18.811 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7694", 'reported_seq': 10175, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:15.906572+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:46:15.906572+0000', 'last_peered': '2024-09-16T04:46:15.906572+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:46:15.906572+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7694, 'log_dups_size': 0, 'ondisk_log_size': 7694, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41462296, 'num_objects': 5062, 'num_object_clones': 0, 'num_object_copies': 20248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7691, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1214, 'num_bytes_recovered': 9846784, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3930}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:18.811 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:46:18.811 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7615", 'reported_seq': 7712, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:15.473355+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:46:15.473355+0000', 'last_peered': '2024-09-16T04:46:15.473355+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:46:15.473355+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7615, 'log_dups_size': 0, 'ondisk_log_size': 7615, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42128652, 'num_objects': 5143, 'num_object_clones': 0, 'num_object_copies': 20572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2193, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5143, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7613, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2952}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:18.811 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:46:18.811 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7475", 'reported_seq': 7572, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:15.841454+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:46:15.841454+0000', 'last_peered': '2024-09-16T04:46:15.841454+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:46:15.841454+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7475, 'log_dups_size': 0, 'ondisk_log_size': 7475, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41086866, 'num_objects': 5016, 'num_object_clones': 0, 'num_object_copies': 20064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2173, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5016, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7472, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2846}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:18.811 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:46:18.811 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7339", 'reported_seq': 7402, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:15.937855+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:46:15.937855+0000', 'last_peered': '2024-09-16T04:46:15.937855+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:46:15.937855+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7339, 'log_dups_size': 0, 'ondisk_log_size': 7339, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40893062, 'num_objects': 4992, 'num_object_clones': 0, 'num_object_copies': 19968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7338, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2875}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:18.811 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:46:18.812 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7481", 'reported_seq': 7561, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:15.374110+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:46:15.374110+0000', 'last_peered': '2024-09-16T04:46:15.374110+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:46:15.374110+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7481, 'log_dups_size': 0, 'ondisk_log_size': 7481, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41384582, 'num_objects': 5052, 'num_object_clones': 0, 'num_object_copies': 20208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5052, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7480, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2830}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:18.812 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:46:18.812 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7608", 'reported_seq': 7707, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:15.781030+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:46:15.781030+0000', 'last_peered': '2024-09-16T04:46:15.781030+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:46:15.781030+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7608, 'log_dups_size': 0, 'ondisk_log_size': 7608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42382604, 'num_objects': 5174, 'num_object_clones': 0, 'num_object_copies': 20696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2214, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7606, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2962}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:18.812 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:46:18.812 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7436", 'reported_seq': 7525, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:15.484444+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:46:15.484444+0000', 'last_peered': '2024-09-16T04:46:15.484444+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:46:15.484444+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7436, 'log_dups_size': 0, 'ondisk_log_size': 7436, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41063692, 'num_objects': 5013, 'num_object_clones': 0, 'num_object_copies': 20052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7434, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2886}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:18.812 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:46:18.812 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7369", 'reported_seq': 7442, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:15.611538+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:46:15.611538+0000', 'last_peered': '2024-09-16T04:46:15.611538+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:46:15.611538+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7369, 'log_dups_size': 0, 'ondisk_log_size': 7369, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40449292, 'num_objects': 4938, 'num_object_clones': 0, 'num_object_copies': 19752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7367, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2702}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:18.812 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:46:19.573 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:19.574 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31976/324456 objects degraded (9.855%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":391,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":81114,"data_bytes":664438464,"bytes_used":2891567104,"bytes_avail":383655489536,"bytes_total":386547056640,"degraded_objects":31976,"degraded_total":324456,"degraded_ratio":0.098552654289025321,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":66883,"recovering_keys_per_sec":0,"num_objects_recovered":120,"num_bytes_recovered":802816,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":200},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:46:19.595 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:46:19.596 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:20.162 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:20.162 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:20.193 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-16T04:46:18.661307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664438464,"num_objects":81114,"num_object_clones":0,"num_object_copies":324456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81114,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120373,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3606,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120400,"ondisk_log_size":120400,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2823796,"kb_used_data":1241264,"kb_used_omap":14,"kb_used_meta":1582449,"kb_avail":374663564,"statfs":{"total":386547056640,"available":383655489536,"internally_reserved":0,"allocated":1271054336,"data_stored":1270334460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1620428337},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,7,3,5,16,5,3,4,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19685376,"num_objects":-2403,"num_object_clones":0,"num_object_copies":-9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2403,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003183"},"pg_stats":[{"pgid":"1.f","version":"33'7544","reported_seq":7619,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.916969+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:15.916969+0000","last_peered":"2024-09-16T04:46:15.916969+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:15.916969+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7544,"log_dups_size":0,"ondisk_log_size":7544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41861120,"num_objects":5110,"num_object_clones":0,"num_object_copies":20440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7544,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7549","reported_seq":7622,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.611275+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:15.611275+0000","last_peered":"2024-09-16T04:46:15.611275+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:15.611275+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7549,"log_dups_size":0,"ondisk_log_size":7549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41981196,"num_objects":5125,"num_object_clones":0,"num_object_copies":20500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7547,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2917}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7537","reported_seq":7622,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.475996+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:15.475996+0000","last_peered":"2024-09-16T04:46:15.475996+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:15.475996+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42048158,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7536,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2911}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7551","reported_seq":7622,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.675212+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:15.675212+0000","last_peered":"2024-09-16T04:46:15.675212+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:15.675212+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7551,"log_dups_size":0,"ondisk_log_size":7551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41932044,"num_objects":5119,"num_object_clones":0,"num_object_copies":20476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7549,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7648","reported_seq":7739,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.900108+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:15.900108+0000","last_peered":"2024-09-16T04:46:15.900108+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:15.900108+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7648,"log_dups_size":0,"ondisk_log_size":7648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41719052,"num_objects":5093,"num_object_clones":0,"num_object_copies":20372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7646,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2827}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7366","reported_seq":7437,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.864987+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:15.864987+0000","last_peered":"2024-09-16T04:46:15.864987+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:15.864987+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7366,"log_dups_size":0,"ondisk_log_size":7366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40517632,"num_objects":4946,"num_object_clones":0,"num_object_copies":19784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7366,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7511","reported_seq":7581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.636287+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:15.636287+0000","last_peered":"2024-09-16T04:46:15.636287+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:15.636287+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7511,"log_dups_size":0,"ondisk_log_size":7511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41686284,"num_objects":5089,"num_object_clones":0,"num_object_copies":20356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7509,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2865}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7677","reported_seq":12234,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:17.697932+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:17.697932+0000","last_peered":"2024-09-16T04:46:17.697932+0000","last_clean":"2024-09-16T04:46:17.697932+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:17.697932+0000","last_undegraded":"2024-09-16T04:46:17.697932+0000","last_fullsized":"2024-09-16T04:46:17.697932+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7677,"log_dups_size":0,"ondisk_log_size":7677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:21.276340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41841932,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7694","reported_seq":10175,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.906572+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:15.906572+0000","last_peered":"2024-09-16T04:46:15.906572+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:15.906572+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7694,"log_dups_size":0,"ondisk_log_size":7694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41462296,"num_objects":5062,"num_object_clones":0,"num_object_copies":20248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7691,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1214,"num_bytes_recovered":9846784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3930}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7615","reported_seq":7712,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.473355+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:15.473355+0000","last_peered":"2024-09-16T04:46:15.473355+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:15.473355+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7615,"log_dups_size":0,"ondisk_log_size":7615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42128652,"num_objects":5143,"num_object_clones":0,"num_object_copies":20572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2952}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7475","reported_seq":7572,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.841454+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:15.841454+0000","last_peered":"2024-09-16T04:46:15.841454+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:15.841454+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7475,"log_dups_size":0,"ondisk_log_size":7475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41086866,"num_objects":5016,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7472,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7339","reported_seq":7402,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.937855+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:15.937855+0000","last_peered":"2024-09-16T04:46:15.937855+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:15.937855+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7339,"log_dups_size":0,"ondisk_log_size":7339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40893062,"num_objects":4992,"num_object_clones":0,"num_object_copies":19968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7338,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2875}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7481","reported_seq":7561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.374110+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:15.374110+0000","last_peered":"2024-09-16T04:46:15.374110+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:15.374110+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7481,"log_dups_size":0,"ondisk_log_size":7481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41384582,"num_objects":5052,"num_object_clones":0,"num_object_copies":20208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7480,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7608","reported_seq":7707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.781030+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:15.781030+0000","last_peered":"2024-09-16T04:46:15.781030+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:15.781030+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7608,"log_dups_size":0,"ondisk_log_size":7608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42382604,"num_objects":5174,"num_object_clones":0,"num_object_copies":20696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7606,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2962}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7436","reported_seq":7525,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.484444+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:15.484444+0000","last_peered":"2024-09-16T04:46:15.484444+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:15.484444+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7436,"log_dups_size":0,"ondisk_log_size":7436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41063692,"num_objects":5013,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7434,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7369","reported_seq":7442,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:15.611538+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:15.611538+0000","last_peered":"2024-09-16T04:46:15.611538+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:15.611538+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7369,"log_dups_size":0,"ondisk_log_size":7369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40449292,"num_objects":4938,"num_object_clones":0,"num_object_copies":19752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7367,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2702}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":664438464,"num_objects":81114,"num_object_clones":0,"num_object_copies":324456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81114,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120373,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3606,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1268088832,"data_stored":1268088832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120400,"ondisk_log_size":120400,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750120,"kb_used_data":325528,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93621720,"statfs":{"total":96636764160,"available":95868641280,"internally_reserved":0,"allocated":333340672,"data_stored":333160703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434761423},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,1,2,7,3,1,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762084,"kb_used_data":325520,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93609756,"statfs":{"total":96636764160,"available":95856390144,"internally_reserved":0,"allocated":333332480,"data_stored":333152511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557736,"kb_used_data":265048,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93814104,"statfs":{"total":96636764160,"available":96065642496,"internally_reserved":0,"allocated":271409152,"data_stored":271229183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299691727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753856,"kb_used_data":325168,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93617984,"statfs":{"total":96636764160,"available":95864815616,"internally_reserved":0,"allocated":332972032,"data_stored":332792063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":438955726},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,4,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":332328960,"data_stored":332328960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":271101952,"data_stored":271101952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":332578816,"data_stored":332578816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":332079104,"data_stored":332079104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:23.195 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:23.811 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:23.812 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:23.847 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-16T04:46:22.662262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655328960,"num_objects":80002,"num_object_clones":0,"num_object_copies":320008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121485,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3673,"num_bytes_recovered":28598272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121512,"ondisk_log_size":121512,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2822220,"kb_used_data":1227192,"kb_used_omap":14,"kb_used_meta":1594929,"kb_avail":374665140,"statfs":{"total":386547056640,"available":383657103360,"internally_reserved":0,"allocated":1256644608,"data_stored":1255924732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1633207857},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,3,11,14,24,2,0,0,0,0,4,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25772032,"num_objects":-3146,"num_object_clones":0,"num_object_copies":-12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3146,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1220608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003090"},"pg_stats":[{"pgid":"1.f","version":"33'7621","reported_seq":7700,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.888371+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:20.888371+0000","last_peered":"2024-09-16T04:46:20.888371+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:20.888371+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7621,"log_dups_size":0,"ondisk_log_size":7621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41230336,"num_objects":5033,"num_object_clones":0,"num_object_copies":20132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7621,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7634","reported_seq":7707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.794103+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:20.794103+0000","last_peered":"2024-09-16T04:46:20.794103+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:20.794103+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7634,"log_dups_size":0,"ondisk_log_size":7634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41284876,"num_objects":5040,"num_object_clones":0,"num_object_copies":20160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7632,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2832}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7609","reported_seq":7694,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.539192+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:20.539192+0000","last_peered":"2024-09-16T04:46:20.539192+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:20.539192+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7609,"log_dups_size":0,"ondisk_log_size":7609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41458334,"num_objects":5062,"num_object_clones":0,"num_object_copies":20248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5062,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7608,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2839}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7622","reported_seq":7693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.641067+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:20.641067+0000","last_peered":"2024-09-16T04:46:20.641067+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:20.641067+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7622,"log_dups_size":0,"ondisk_log_size":7622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41350412,"num_objects":5048,"num_object_clones":0,"num_object_copies":20192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7620,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2821}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7726","reported_seq":7819,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.961325+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:20.961325+0000","last_peered":"2024-09-16T04:46:20.961325+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:20.961325+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7726,"log_dups_size":0,"ondisk_log_size":7726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41080076,"num_objects":5015,"num_object_clones":0,"num_object_copies":20060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7724,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7443","reported_seq":7522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.967230+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:20.967230+0000","last_peered":"2024-09-16T04:46:20.967230+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:20.967230+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7443,"log_dups_size":0,"ondisk_log_size":7443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39886848,"num_objects":4869,"num_object_clones":0,"num_object_copies":19476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7443,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2678}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7580","reported_seq":7650,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.707319+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:20.707319+0000","last_peered":"2024-09-16T04:46:20.707319+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:20.707319+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7580,"log_dups_size":0,"ondisk_log_size":7580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41121036,"num_objects":5020,"num_object_clones":0,"num_object_copies":20080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7578,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2796}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7677","reported_seq":12234,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:17.697932+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:17.697932+0000","last_peered":"2024-09-16T04:46:17.697932+0000","last_clean":"2024-09-16T04:46:17.697932+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:17.697932+0000","last_undegraded":"2024-09-16T04:46:17.697932+0000","last_fullsized":"2024-09-16T04:46:17.697932+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7677,"log_dups_size":0,"ondisk_log_size":7677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:21.276340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41841932,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7783","reported_seq":10370,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.994393+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:20.994393+0000","last_peered":"2024-09-16T04:46:20.994393+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:20.994393+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7783,"log_dups_size":0,"ondisk_log_size":7783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40733208,"num_objects":4973,"num_object_clones":0,"num_object_copies":19892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7780,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1267,"num_bytes_recovered":10264576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3894}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7685","reported_seq":7782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.519648+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:20.519648+0000","last_peered":"2024-09-16T04:46:20.519648+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:20.519648+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7685,"log_dups_size":0,"ondisk_log_size":7685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41555212,"num_objects":5073,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7683,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2882}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7553","reported_seq":7656,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.894738+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:20.894738+0000","last_peered":"2024-09-16T04:46:20.894738+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:20.894738+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7553,"log_dups_size":0,"ondisk_log_size":7553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40447890,"num_objects":4938,"num_object_clones":0,"num_object_copies":19752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7550,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2771}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7403","reported_seq":7470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.556851+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:20.556851+0000","last_peered":"2024-09-16T04:46:20.556851+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:20.556851+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7403,"log_dups_size":0,"ondisk_log_size":7403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40368774,"num_objects":4928,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7402,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2813}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7554","reported_seq":7635,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.450095+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:20.450095+0000","last_peered":"2024-09-16T04:46:20.450095+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:20.450095+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7554,"log_dups_size":0,"ondisk_log_size":7554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40786566,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7553,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7679","reported_seq":7780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.990200+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:20.990200+0000","last_peered":"2024-09-16T04:46:20.990200+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:20.990200+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7679,"log_dups_size":0,"ondisk_log_size":7679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41800972,"num_objects":5103,"num_object_clones":0,"num_object_copies":20412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7677,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2893}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7513","reported_seq":7603,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.415842+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:20.415842+0000","last_peered":"2024-09-16T04:46:20.415842+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:20.415842+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7513,"log_dups_size":0,"ondisk_log_size":7513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40432908,"num_objects":4936,"num_object_clones":0,"num_object_copies":19744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7511,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2809}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7430","reported_seq":7503,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.640004+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:20.640004+0000","last_peered":"2024-09-16T04:46:20.640004+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:20.640004+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7430,"log_dups_size":0,"ondisk_log_size":7430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39949580,"num_objects":4877,"num_object_clones":0,"num_object_copies":19508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7428,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":655328960,"num_objects":80002,"num_object_clones":0,"num_object_copies":320008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121485,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3673,"num_bytes_recovered":28598272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1253568512,"data_stored":1253568512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121512,"ondisk_log_size":121512,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749592,"kb_used_data":320828,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93622248,"statfs":{"total":96636764160,"available":95869181952,"internally_reserved":0,"allocated":328527872,"data_stored":328347903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":439021263},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,4,16,1,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761532,"kb_used_data":320812,"kb_used_omap":1,"kb_used_meta":440702,"kb_avail":93610308,"statfs":{"total":96636764160,"available":95856955392,"internally_reserved":0,"allocated":328511488,"data_stored":328331519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":451279301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557736,"kb_used_data":265048,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93814104,"statfs":{"total":96636764160,"available":96065642496,"internally_reserved":0,"allocated":271409152,"data_stored":271229183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299691727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753360,"kb_used_data":320504,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93618480,"statfs":{"total":96636764160,"available":95865323520,"internally_reserved":0,"allocated":328196096,"data_stored":328016127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":443215566},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,0,1,2,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327483392,"data_stored":327483392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":271101952,"data_stored":271101952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327737344,"data_stored":327737344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327245824,"data_stored":327245824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:23.848 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:24.395 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:24.395 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:24.416 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-16T04:46:22.662262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655328960,"num_objects":80002,"num_object_clones":0,"num_object_copies":320008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121485,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3673,"num_bytes_recovered":28598272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121512,"ondisk_log_size":121512,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2822220,"kb_used_data":1227192,"kb_used_omap":14,"kb_used_meta":1594929,"kb_avail":374665140,"statfs":{"total":386547056640,"available":383657103360,"internally_reserved":0,"allocated":1256644608,"data_stored":1255924732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1633207857},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,3,11,14,24,2,0,0,0,0,4,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25772032,"num_objects":-3146,"num_object_clones":0,"num_object_copies":-12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3146,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1220608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003090"},"pg_stats":[{"pgid":"1.f","version":"33'7621","reported_seq":7700,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.888371+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:20.888371+0000","last_peered":"2024-09-16T04:46:20.888371+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:20.888371+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7621,"log_dups_size":0,"ondisk_log_size":7621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41230336,"num_objects":5033,"num_object_clones":0,"num_object_copies":20132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7621,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7634","reported_seq":7707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.794103+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:20.794103+0000","last_peered":"2024-09-16T04:46:20.794103+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:20.794103+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7634,"log_dups_size":0,"ondisk_log_size":7634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41284876,"num_objects":5040,"num_object_clones":0,"num_object_copies":20160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7632,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2832}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7609","reported_seq":7694,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.539192+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:20.539192+0000","last_peered":"2024-09-16T04:46:20.539192+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:20.539192+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7609,"log_dups_size":0,"ondisk_log_size":7609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41458334,"num_objects":5062,"num_object_clones":0,"num_object_copies":20248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5062,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7608,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2839}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7622","reported_seq":7693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.641067+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:20.641067+0000","last_peered":"2024-09-16T04:46:20.641067+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:20.641067+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7622,"log_dups_size":0,"ondisk_log_size":7622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41350412,"num_objects":5048,"num_object_clones":0,"num_object_copies":20192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7620,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2821}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7726","reported_seq":7819,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.961325+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:20.961325+0000","last_peered":"2024-09-16T04:46:20.961325+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:20.961325+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7726,"log_dups_size":0,"ondisk_log_size":7726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41080076,"num_objects":5015,"num_object_clones":0,"num_object_copies":20060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7724,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7443","reported_seq":7522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.967230+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:20.967230+0000","last_peered":"2024-09-16T04:46:20.967230+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:20.967230+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7443,"log_dups_size":0,"ondisk_log_size":7443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39886848,"num_objects":4869,"num_object_clones":0,"num_object_copies":19476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7443,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2678}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7580","reported_seq":7650,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.707319+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:20.707319+0000","last_peered":"2024-09-16T04:46:20.707319+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:20.707319+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7580,"log_dups_size":0,"ondisk_log_size":7580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41121036,"num_objects":5020,"num_object_clones":0,"num_object_copies":20080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7578,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2796}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7677","reported_seq":12234,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:17.697932+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:17.697932+0000","last_peered":"2024-09-16T04:46:17.697932+0000","last_clean":"2024-09-16T04:46:17.697932+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:17.697932+0000","last_undegraded":"2024-09-16T04:46:17.697932+0000","last_fullsized":"2024-09-16T04:46:17.697932+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7677,"log_dups_size":0,"ondisk_log_size":7677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:21.276340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41841932,"num_objects":5108,"num_object_clones":0,"num_object_copies":20432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7783","reported_seq":10370,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.994393+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:20.994393+0000","last_peered":"2024-09-16T04:46:20.994393+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:20.994393+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7783,"log_dups_size":0,"ondisk_log_size":7783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40733208,"num_objects":4973,"num_object_clones":0,"num_object_copies":19892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7780,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1267,"num_bytes_recovered":10264576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3894}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7685","reported_seq":7782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.519648+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:20.519648+0000","last_peered":"2024-09-16T04:46:20.519648+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:20.519648+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7685,"log_dups_size":0,"ondisk_log_size":7685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41555212,"num_objects":5073,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7683,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2882}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7553","reported_seq":7656,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.894738+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:20.894738+0000","last_peered":"2024-09-16T04:46:20.894738+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:20.894738+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7553,"log_dups_size":0,"ondisk_log_size":7553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40447890,"num_objects":4938,"num_object_clones":0,"num_object_copies":19752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7550,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2771}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7403","reported_seq":7470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.556851+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:20.556851+0000","last_peered":"2024-09-16T04:46:20.556851+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:20.556851+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7403,"log_dups_size":0,"ondisk_log_size":7403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40368774,"num_objects":4928,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7402,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2813}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7554","reported_seq":7635,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.450095+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:20.450095+0000","last_peered":"2024-09-16T04:46:20.450095+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:20.450095+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7554,"log_dups_size":0,"ondisk_log_size":7554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40786566,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7553,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7679","reported_seq":7780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.990200+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:20.990200+0000","last_peered":"2024-09-16T04:46:20.990200+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:20.990200+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7679,"log_dups_size":0,"ondisk_log_size":7679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41800972,"num_objects":5103,"num_object_clones":0,"num_object_copies":20412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7677,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2893}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7513","reported_seq":7603,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.415842+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:20.415842+0000","last_peered":"2024-09-16T04:46:20.415842+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:20.415842+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7513,"log_dups_size":0,"ondisk_log_size":7513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40432908,"num_objects":4936,"num_object_clones":0,"num_object_copies":19744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7511,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2809}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7430","reported_seq":7503,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.640004+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:20.640004+0000","last_peered":"2024-09-16T04:46:20.640004+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:20.640004+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7430,"log_dups_size":0,"ondisk_log_size":7430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39949580,"num_objects":4877,"num_object_clones":0,"num_object_copies":19508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7428,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":655328960,"num_objects":80002,"num_object_clones":0,"num_object_copies":320008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121485,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3673,"num_bytes_recovered":28598272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1253568512,"data_stored":1253568512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121512,"ondisk_log_size":121512,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749592,"kb_used_data":320828,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93622248,"statfs":{"total":96636764160,"available":95869181952,"internally_reserved":0,"allocated":328527872,"data_stored":328347903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":439021263},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,4,16,1,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761532,"kb_used_data":320812,"kb_used_omap":1,"kb_used_meta":440702,"kb_avail":93610308,"statfs":{"total":96636764160,"available":95856955392,"internally_reserved":0,"allocated":328511488,"data_stored":328331519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":451279301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557736,"kb_used_data":265048,"kb_used_omap":4,"kb_used_meta":292667,"kb_avail":93814104,"statfs":{"total":96636764160,"available":96065642496,"internally_reserved":0,"allocated":271409152,"data_stored":271229183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299691727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753360,"kb_used_data":320504,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93618480,"statfs":{"total":96636764160,"available":95865323520,"internally_reserved":0,"allocated":328196096,"data_stored":328016127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":443215566},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,0,1,2,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327483392,"data_stored":327483392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":271101952,"data_stored":271101952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327737344,"data_stored":327737344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327245824,"data_stored":327245824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:24.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:46:24.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7621", 'reported_seq': 7700, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:20.888371+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:46:20.888371+0000', 'last_peered': '2024-09-16T04:46:20.888371+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:46:20.888371+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7621, 'log_dups_size': 0, 'ondisk_log_size': 7621, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41230336, 'num_objects': 5033, 'num_object_clones': 0, 'num_object_copies': 20132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5033, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7621, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2831}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:24.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:46:24.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7634", 'reported_seq': 7707, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:20.794103+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:46:20.794103+0000', 'last_peered': '2024-09-16T04:46:20.794103+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:46:20.794103+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7634, 'log_dups_size': 0, 'ondisk_log_size': 7634, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41284876, 'num_objects': 5040, 'num_object_clones': 0, 'num_object_copies': 20160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2210, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5040, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7632, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2832}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:24.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:46:24.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7609", 'reported_seq': 7694, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:20.539192+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:46:20.539192+0000', 'last_peered': '2024-09-16T04:46:20.539192+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:46:20.539192+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7609, 'log_dups_size': 0, 'ondisk_log_size': 7609, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41458334, 'num_objects': 5062, 'num_object_clones': 0, 'num_object_copies': 20248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5062, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7608, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2839}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:24.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:46:24.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7622", 'reported_seq': 7693, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:20.641067+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:46:20.641067+0000', 'last_peered': '2024-09-16T04:46:20.641067+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:46:20.641067+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7622, 'log_dups_size': 0, 'ondisk_log_size': 7622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41350412, 'num_objects': 5048, 'num_object_clones': 0, 'num_object_copies': 20192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5048, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7620, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2821}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:24.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:46:24.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7726", 'reported_seq': 7819, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:20.961325+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:46:20.961325+0000', 'last_peered': '2024-09-16T04:46:20.961325+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:46:20.961325+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7726, 'log_dups_size': 0, 'ondisk_log_size': 7726, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41080076, 'num_objects': 5015, 'num_object_clones': 0, 'num_object_copies': 20060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2267, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5015, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7724, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2750}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:24.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:46:24.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7443", 'reported_seq': 7522, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:20.967230+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:46:20.967230+0000', 'last_peered': '2024-09-16T04:46:20.967230+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:46:20.967230+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7443, 'log_dups_size': 0, 'ondisk_log_size': 7443, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39886848, 'num_objects': 4869, 'num_object_clones': 0, 'num_object_copies': 19476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4869, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7443, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2678}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:24.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:46:24.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7580", 'reported_seq': 7650, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:20.707319+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:46:20.707319+0000', 'last_peered': '2024-09-16T04:46:20.707319+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:46:20.707319+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7580, 'log_dups_size': 0, 'ondisk_log_size': 7580, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41121036, 'num_objects': 5020, 'num_object_clones': 0, 'num_object_copies': 20080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7578, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2796}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:24.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:46:24.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7783", 'reported_seq': 10370, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:20.994393+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:46:20.994393+0000', 'last_peered': '2024-09-16T04:46:20.994393+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:46:20.994393+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7783, 'log_dups_size': 0, 'ondisk_log_size': 7783, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40733208, 'num_objects': 4973, 'num_object_clones': 0, 'num_object_copies': 19892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1079, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7780, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1267, 'num_bytes_recovered': 10264576, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3894}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:24.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:46:24.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7685", 'reported_seq': 7782, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:20.519648+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:46:20.519648+0000', 'last_peered': '2024-09-16T04:46:20.519648+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:46:20.519648+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7685, 'log_dups_size': 0, 'ondisk_log_size': 7685, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41555212, 'num_objects': 5073, 'num_object_clones': 0, 'num_object_copies': 20292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2193, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5073, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7683, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2882}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:24.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:46:24.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7553", 'reported_seq': 7656, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:20.894738+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:46:20.894738+0000', 'last_peered': '2024-09-16T04:46:20.894738+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:46:20.894738+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7553, 'log_dups_size': 0, 'ondisk_log_size': 7553, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40447890, 'num_objects': 4938, 'num_object_clones': 0, 'num_object_copies': 19752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2170, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7550, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2771}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:24.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:46:24.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7403", 'reported_seq': 7470, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:20.556851+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:46:20.556851+0000', 'last_peered': '2024-09-16T04:46:20.556851+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:46:20.556851+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7403, 'log_dups_size': 0, 'ondisk_log_size': 7403, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40368774, 'num_objects': 4928, 'num_object_clones': 0, 'num_object_copies': 19712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4928, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7402, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2813}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:24.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:46:24.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7554", 'reported_seq': 7635, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:20.450095+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:46:20.450095+0000', 'last_peered': '2024-09-16T04:46:20.450095+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:46:20.450095+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7554, 'log_dups_size': 0, 'ondisk_log_size': 7554, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40786566, 'num_objects': 4979, 'num_object_clones': 0, 'num_object_copies': 19916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7553, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2757}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:24.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:46:24.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7679", 'reported_seq': 7780, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:20.990200+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:46:20.990200+0000', 'last_peered': '2024-09-16T04:46:20.990200+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:46:20.990200+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7679, 'log_dups_size': 0, 'ondisk_log_size': 7679, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41800972, 'num_objects': 5103, 'num_object_clones': 0, 'num_object_copies': 20412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5103, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7677, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2893}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:24.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:46:24.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7513", 'reported_seq': 7603, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:20.415842+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:46:20.415842+0000', 'last_peered': '2024-09-16T04:46:20.415842+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:46:20.415842+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7513, 'log_dups_size': 0, 'ondisk_log_size': 7513, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40432908, 'num_objects': 4936, 'num_object_clones': 0, 'num_object_copies': 19744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7511, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2809}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:24.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:46:24.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7430", 'reported_seq': 7503, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:20.640004+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:46:20.640004+0000', 'last_peered': '2024-09-16T04:46:20.640004+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:46:20.640004+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7430, 'log_dups_size': 0, 'ondisk_log_size': 7430, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39949580, 'num_objects': 4877, 'num_object_clones': 0, 'num_object_copies': 19508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4877, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7428, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2641}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:24.420 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:46:25.134 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:25.134 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31909/320008 objects degraded (9.971%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":397,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":80002,"data_bytes":655328960,"bytes_used":2889953280,"bytes_avail":383657103360,"bytes_total":386547056640,"degraded_objects":31909,"degraded_total":320008,"degraded_ratio":0.099713132171695709,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":101691,"recovering_keys_per_sec":0,"num_objects_recovered":182,"num_bytes_recovered":1220608,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":262},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:46:25.170 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:46:25.171 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:25.741 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:25.742 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:25.768 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-16T04:46:24.662775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654755520,"num_objects":79932,"num_object_clones":0,"num_object_copies":319728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79932,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121555,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3673,"num_bytes_recovered":28598272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121582,"ondisk_log_size":121582,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2827540,"kb_used_data":1226944,"kb_used_omap":14,"kb_used_meta":1600497,"kb_avail":374659820,"statfs":{"total":386547056640,"available":383651655680,"internally_reserved":0,"allocated":1256390656,"data_stored":1255670780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1638909489},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,2,15,12,27,2,0,0,0,0,4,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19931136,"num_objects":-2433,"num_object_clones":0,"num_object_copies":-9732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2433,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":737280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003085"},"pg_stats":[{"pgid":"1.f","version":"33'7621","reported_seq":7700,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.888371+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:20.888371+0000","last_peered":"2024-09-16T04:46:20.888371+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:20.888371+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7621,"log_dups_size":0,"ondisk_log_size":7621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41230336,"num_objects":5033,"num_object_clones":0,"num_object_copies":20132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7621,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7634","reported_seq":7707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.794103+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:20.794103+0000","last_peered":"2024-09-16T04:46:20.794103+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:20.794103+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7634,"log_dups_size":0,"ondisk_log_size":7634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41284876,"num_objects":5040,"num_object_clones":0,"num_object_copies":20160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7632,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2832}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7609","reported_seq":7694,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.539192+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:20.539192+0000","last_peered":"2024-09-16T04:46:20.539192+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:20.539192+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7609,"log_dups_size":0,"ondisk_log_size":7609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41458334,"num_objects":5062,"num_object_clones":0,"num_object_copies":20248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5062,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7608,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2839}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7622","reported_seq":7693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.641067+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:20.641067+0000","last_peered":"2024-09-16T04:46:20.641067+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:20.641067+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7622,"log_dups_size":0,"ondisk_log_size":7622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41350412,"num_objects":5048,"num_object_clones":0,"num_object_copies":20192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7620,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2821}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7726","reported_seq":7819,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.961325+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:20.961325+0000","last_peered":"2024-09-16T04:46:20.961325+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:20.961325+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7726,"log_dups_size":0,"ondisk_log_size":7726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41080076,"num_objects":5015,"num_object_clones":0,"num_object_copies":20060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7724,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7443","reported_seq":7522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.967230+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:20.967230+0000","last_peered":"2024-09-16T04:46:20.967230+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:20.967230+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7443,"log_dups_size":0,"ondisk_log_size":7443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39886848,"num_objects":4869,"num_object_clones":0,"num_object_copies":19476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7443,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2678}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7580","reported_seq":7650,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.707319+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:20.707319+0000","last_peered":"2024-09-16T04:46:20.707319+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:20.707319+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7580,"log_dups_size":0,"ondisk_log_size":7580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41121036,"num_objects":5020,"num_object_clones":0,"num_object_copies":20080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7578,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2796}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7747","reported_seq":12304,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:22.599068+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:22.599068+0000","last_peered":"2024-09-16T04:46:22.599068+0000","last_clean":"2024-09-16T04:46:22.599068+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:22.599068+0000","last_undegraded":"2024-09-16T04:46:22.599068+0000","last_fullsized":"2024-09-16T04:46:22.599068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7747,"log_dups_size":0,"ondisk_log_size":7747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:26.328077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41268492,"num_objects":5038,"num_object_clones":0,"num_object_copies":20152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7745,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7783","reported_seq":10370,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.994393+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:20.994393+0000","last_peered":"2024-09-16T04:46:20.994393+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:20.994393+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7783,"log_dups_size":0,"ondisk_log_size":7783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40733208,"num_objects":4973,"num_object_clones":0,"num_object_copies":19892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7780,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1267,"num_bytes_recovered":10264576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3894}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7685","reported_seq":7782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.519648+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:20.519648+0000","last_peered":"2024-09-16T04:46:20.519648+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:20.519648+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7685,"log_dups_size":0,"ondisk_log_size":7685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41555212,"num_objects":5073,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7683,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2882}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7553","reported_seq":7656,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.894738+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:20.894738+0000","last_peered":"2024-09-16T04:46:20.894738+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:20.894738+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7553,"log_dups_size":0,"ondisk_log_size":7553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40447890,"num_objects":4938,"num_object_clones":0,"num_object_copies":19752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7550,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2771}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7403","reported_seq":7470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.556851+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:20.556851+0000","last_peered":"2024-09-16T04:46:20.556851+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:20.556851+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7403,"log_dups_size":0,"ondisk_log_size":7403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40368774,"num_objects":4928,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7402,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2813}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7554","reported_seq":7635,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.450095+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:20.450095+0000","last_peered":"2024-09-16T04:46:20.450095+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:20.450095+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7554,"log_dups_size":0,"ondisk_log_size":7554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40786566,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7553,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7679","reported_seq":7780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.990200+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:20.990200+0000","last_peered":"2024-09-16T04:46:20.990200+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:20.990200+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7679,"log_dups_size":0,"ondisk_log_size":7679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41800972,"num_objects":5103,"num_object_clones":0,"num_object_copies":20412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7677,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2893}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7513","reported_seq":7603,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.415842+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:20.415842+0000","last_peered":"2024-09-16T04:46:20.415842+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:20.415842+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7513,"log_dups_size":0,"ondisk_log_size":7513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40432908,"num_objects":4936,"num_object_clones":0,"num_object_copies":19744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7511,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2809}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7430","reported_seq":7503,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:20.640004+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:20.640004+0000","last_peered":"2024-09-16T04:46:20.640004+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:20.640004+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7430,"log_dups_size":0,"ondisk_log_size":7430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39949580,"num_objects":4877,"num_object_clones":0,"num_object_copies":19508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7428,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":654755520,"num_objects":79932,"num_object_clones":0,"num_object_copies":319728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79932,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121555,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3673,"num_bytes_recovered":28598272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1253310464,"data_stored":1253310464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121582,"ondisk_log_size":121582,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749592,"kb_used_data":320828,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93622248,"statfs":{"total":96636764160,"available":95869181952,"internally_reserved":0,"allocated":328527872,"data_stored":328347903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":439021263},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,4,16,1,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761532,"kb_used_data":320812,"kb_used_omap":1,"kb_used_meta":440702,"kb_avail":93610308,"statfs":{"total":96636764160,"available":95856955392,"internally_reserved":0,"allocated":328511488,"data_stored":328331519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":451279301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051644,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563056,"kb_used_data":264800,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93808784,"statfs":{"total":96636764160,"available":96060194816,"internally_reserved":0,"allocated":271155200,"data_stored":270975231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305393359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753360,"kb_used_data":320504,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93618480,"statfs":{"total":96636764160,"available":95865323520,"internally_reserved":0,"allocated":328196096,"data_stored":328016127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":443215566},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,0,1,2,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327483392,"data_stored":327483392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270843904,"data_stored":270843904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":327737344,"data_stored":327737344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327245824,"data_stored":327245824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:28.770 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:29.367 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:29.368 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:29.389 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-16T04:46:28.663839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":645146304,"num_objects":78759,"num_object_clones":0,"num_object_copies":315036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78759,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122728,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3725,"num_bytes_recovered":28983296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122755,"ondisk_log_size":122755,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2812804,"kb_used_data":1212220,"kb_used_omap":14,"kb_used_meta":1600497,"kb_avail":374674556,"statfs":{"total":386547056640,"available":383666745344,"internally_reserved":0,"allocated":1241313280,"data_stored":1240593404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1638909489},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,10,5,10,5,4,0,0,5,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19857408,"num_objects":-2424,"num_object_clones":0,"num_object_copies":-9696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2424,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003130"},"pg_stats":[{"pgid":"1.f","version":"33'7697","reported_seq":7778,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.966728+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:25.966728+0000","last_peered":"2024-09-16T04:46:25.966728+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:25.966728+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40607744,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7697,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2756}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7703","reported_seq":7776,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.638518+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:25.638518+0000","last_peered":"2024-09-16T04:46:25.638518+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:25.638518+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7703,"log_dups_size":0,"ondisk_log_size":7703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40719628,"num_objects":4971,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7701,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7680","reported_seq":7765,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.574513+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:25.574513+0000","last_peered":"2024-09-16T04:46:25.574513+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:25.574513+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7680,"log_dups_size":0,"ondisk_log_size":7680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40876702,"num_objects":4991,"num_object_clones":0,"num_object_copies":19964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4991,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7679,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2768}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7713","reported_seq":7784,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.806097+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:25.806097+0000","last_peered":"2024-09-16T04:46:25.806097+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:25.806097+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40604940,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7711,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2730}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7784","reported_seq":7879,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.758965+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:25.758965+0000","last_peered":"2024-09-16T04:46:25.758965+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:25.758965+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7784,"log_dups_size":0,"ondisk_log_size":7784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40604940,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7782,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7515","reported_seq":7598,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.946885+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:25.946885+0000","last_peered":"2024-09-16T04:46:25.946885+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:25.946885+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7515,"log_dups_size":0,"ondisk_log_size":7515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39297024,"num_objects":4797,"num_object_clones":0,"num_object_copies":19188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7515,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7652","reported_seq":7722,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.753909+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:25.753909+0000","last_peered":"2024-09-16T04:46:25.753909+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:25.753909+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7652,"log_dups_size":0,"ondisk_log_size":7652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40531212,"num_objects":4948,"num_object_clones":0,"num_object_copies":19792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7650,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2724}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7826","reported_seq":12383,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:27.738395+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:27.738395+0000","last_peered":"2024-09-16T04:46:27.738395+0000","last_clean":"2024-09-16T04:46:27.738395+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:27.738395+0000","last_undegraded":"2024-09-16T04:46:27.738395+0000","last_fullsized":"2024-09-16T04:46:27.738395+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7826,"log_dups_size":0,"ondisk_log_size":7826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:31.394739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40621324,"num_objects":4959,"num_object_clones":0,"num_object_copies":19836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7824,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7872","reported_seq":10553,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:26.047096+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:26.047096+0000","last_peered":"2024-09-16T04:46:26.047096+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:26.047096+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7872,"log_dups_size":0,"ondisk_log_size":7872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40004120,"num_objects":4884,"num_object_clones":0,"num_object_copies":19536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1315,"num_bytes_recovered":10649600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7757","reported_seq":7854,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.570698+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:25.570698+0000","last_peered":"2024-09-16T04:46:25.570698+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:25.570698+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7757,"log_dups_size":0,"ondisk_log_size":7757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40965388,"num_objects":5001,"num_object_clones":0,"num_object_copies":20004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7755,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2810}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7618","reported_seq":7721,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.813865+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:25.813865+0000","last_peered":"2024-09-16T04:46:25.813865+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:25.813865+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7618,"log_dups_size":0,"ondisk_log_size":7618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39915410,"num_objects":4873,"num_object_clones":0,"num_object_copies":19492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7615,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7475","reported_seq":7542,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:26.014758+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:26.014758+0000","last_peered":"2024-09-16T04:46:26.014758+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:26.014758+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7475,"log_dups_size":0,"ondisk_log_size":7475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39778950,"num_objects":4856,"num_object_clones":0,"num_object_copies":19424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7474,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2741}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7632","reported_seq":7713,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.405072+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:25.405072+0000","last_peered":"2024-09-16T04:46:25.405072+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:25.405072+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7632,"log_dups_size":0,"ondisk_log_size":7632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40147590,"num_objects":4901,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7631,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2679}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7759","reported_seq":7862,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.939360+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:25.939360+0000","last_peered":"2024-09-16T04:46:25.939360+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:25.939360+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7759,"log_dups_size":0,"ondisk_log_size":7759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41145612,"num_objects":5023,"num_object_clones":0,"num_object_copies":20092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7757,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2813}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7568","reported_seq":7658,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.557712+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:25.557712+0000","last_peered":"2024-09-16T04:46:25.557712+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:25.557712+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7568,"log_dups_size":0,"ondisk_log_size":7568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39982348,"num_objects":4881,"num_object_clones":0,"num_object_copies":19524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7566,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2754}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7504","reported_seq":7577,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.858872+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:25.858872+0000","last_peered":"2024-09-16T04:46:25.858872+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:25.858872+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7504,"log_dups_size":0,"ondisk_log_size":7504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39343372,"num_objects":4803,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7502,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":645146304,"num_objects":78759,"num_object_clones":0,"num_object_copies":315036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78759,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122728,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3725,"num_bytes_recovered":28983296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1238528000,"data_stored":1238528000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122755,"ondisk_log_size":122755,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748940,"kb_used_data":316028,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93622900,"statfs":{"total":96636764160,"available":95869849600,"internally_reserved":0,"allocated":323612672,"data_stored":323432703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443281103},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,0,4,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744264,"kb_used_data":316016,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93627576,"statfs":{"total":96636764160,"available":95874637824,"internally_reserved":0,"allocated":323600384,"data_stored":323420415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,4,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051645,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566956,"kb_used_data":264540,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93804884,"statfs":{"total":96636764160,"available":96056201216,"internally_reserved":0,"allocated":270888960,"data_stored":270708991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309653199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752644,"kb_used_data":315636,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93619196,"statfs":{"total":96636764160,"available":95866056704,"internally_reserved":0,"allocated":323211264,"data_stored":323031295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":447475406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,3,5,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":322621440,"data_stored":322621440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270585856,"data_stored":270585856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":322895872,"data_stored":322895872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":322424832,"data_stored":322424832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:29.390 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:30.012 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:30.012 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:30.038 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-16T04:46:28.663839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":645146304,"num_objects":78759,"num_object_clones":0,"num_object_copies":315036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78759,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122728,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3725,"num_bytes_recovered":28983296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122755,"ondisk_log_size":122755,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2812804,"kb_used_data":1212220,"kb_used_omap":14,"kb_used_meta":1600497,"kb_avail":374674556,"statfs":{"total":386547056640,"available":383666745344,"internally_reserved":0,"allocated":1241313280,"data_stored":1240593404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1638909489},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,10,5,10,5,4,0,0,5,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19857408,"num_objects":-2424,"num_object_clones":0,"num_object_copies":-9696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2424,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003130"},"pg_stats":[{"pgid":"1.f","version":"33'7697","reported_seq":7778,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.966728+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:25.966728+0000","last_peered":"2024-09-16T04:46:25.966728+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:25.966728+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40607744,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7697,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2756}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7703","reported_seq":7776,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.638518+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:25.638518+0000","last_peered":"2024-09-16T04:46:25.638518+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:25.638518+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7703,"log_dups_size":0,"ondisk_log_size":7703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40719628,"num_objects":4971,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7701,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7680","reported_seq":7765,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.574513+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:25.574513+0000","last_peered":"2024-09-16T04:46:25.574513+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:25.574513+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7680,"log_dups_size":0,"ondisk_log_size":7680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40876702,"num_objects":4991,"num_object_clones":0,"num_object_copies":19964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4991,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7679,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2768}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7713","reported_seq":7784,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.806097+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:25.806097+0000","last_peered":"2024-09-16T04:46:25.806097+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:25.806097+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40604940,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7711,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2730}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7784","reported_seq":7879,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.758965+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:25.758965+0000","last_peered":"2024-09-16T04:46:25.758965+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:25.758965+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7784,"log_dups_size":0,"ondisk_log_size":7784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40604940,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7782,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7515","reported_seq":7598,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.946885+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:25.946885+0000","last_peered":"2024-09-16T04:46:25.946885+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:25.946885+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7515,"log_dups_size":0,"ondisk_log_size":7515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39297024,"num_objects":4797,"num_object_clones":0,"num_object_copies":19188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7515,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7652","reported_seq":7722,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.753909+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:25.753909+0000","last_peered":"2024-09-16T04:46:25.753909+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:25.753909+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7652,"log_dups_size":0,"ondisk_log_size":7652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40531212,"num_objects":4948,"num_object_clones":0,"num_object_copies":19792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7650,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2724}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7826","reported_seq":12383,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:27.738395+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:27.738395+0000","last_peered":"2024-09-16T04:46:27.738395+0000","last_clean":"2024-09-16T04:46:27.738395+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:27.738395+0000","last_undegraded":"2024-09-16T04:46:27.738395+0000","last_fullsized":"2024-09-16T04:46:27.738395+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7826,"log_dups_size":0,"ondisk_log_size":7826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:31.394739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40621324,"num_objects":4959,"num_object_clones":0,"num_object_copies":19836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7824,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7872","reported_seq":10553,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:26.047096+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:26.047096+0000","last_peered":"2024-09-16T04:46:26.047096+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:26.047096+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7872,"log_dups_size":0,"ondisk_log_size":7872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40004120,"num_objects":4884,"num_object_clones":0,"num_object_copies":19536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1315,"num_bytes_recovered":10649600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7757","reported_seq":7854,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.570698+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:25.570698+0000","last_peered":"2024-09-16T04:46:25.570698+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:25.570698+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7757,"log_dups_size":0,"ondisk_log_size":7757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40965388,"num_objects":5001,"num_object_clones":0,"num_object_copies":20004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7755,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2810}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7618","reported_seq":7721,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.813865+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:25.813865+0000","last_peered":"2024-09-16T04:46:25.813865+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:25.813865+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7618,"log_dups_size":0,"ondisk_log_size":7618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39915410,"num_objects":4873,"num_object_clones":0,"num_object_copies":19492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7615,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7475","reported_seq":7542,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:26.014758+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:26.014758+0000","last_peered":"2024-09-16T04:46:26.014758+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:26.014758+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7475,"log_dups_size":0,"ondisk_log_size":7475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39778950,"num_objects":4856,"num_object_clones":0,"num_object_copies":19424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7474,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2741}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7632","reported_seq":7713,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.405072+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:25.405072+0000","last_peered":"2024-09-16T04:46:25.405072+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:25.405072+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7632,"log_dups_size":0,"ondisk_log_size":7632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40147590,"num_objects":4901,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7631,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2679}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7759","reported_seq":7862,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.939360+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:25.939360+0000","last_peered":"2024-09-16T04:46:25.939360+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:25.939360+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7759,"log_dups_size":0,"ondisk_log_size":7759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41145612,"num_objects":5023,"num_object_clones":0,"num_object_copies":20092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7757,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2813}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7568","reported_seq":7658,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.557712+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:25.557712+0000","last_peered":"2024-09-16T04:46:25.557712+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:25.557712+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7568,"log_dups_size":0,"ondisk_log_size":7568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39982348,"num_objects":4881,"num_object_clones":0,"num_object_copies":19524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7566,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2754}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7504","reported_seq":7577,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.858872+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:25.858872+0000","last_peered":"2024-09-16T04:46:25.858872+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:25.858872+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7504,"log_dups_size":0,"ondisk_log_size":7504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39343372,"num_objects":4803,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7502,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":645146304,"num_objects":78759,"num_object_clones":0,"num_object_copies":315036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78759,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122728,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3725,"num_bytes_recovered":28983296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1238528000,"data_stored":1238528000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122755,"ondisk_log_size":122755,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748940,"kb_used_data":316028,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93622900,"statfs":{"total":96636764160,"available":95869849600,"internally_reserved":0,"allocated":323612672,"data_stored":323432703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443281103},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,0,4,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744264,"kb_used_data":316016,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93627576,"statfs":{"total":96636764160,"available":95874637824,"internally_reserved":0,"allocated":323600384,"data_stored":323420415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,4,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051645,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566956,"kb_used_data":264540,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93804884,"statfs":{"total":96636764160,"available":96056201216,"internally_reserved":0,"allocated":270888960,"data_stored":270708991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309653199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752644,"kb_used_data":315636,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93619196,"statfs":{"total":96636764160,"available":95866056704,"internally_reserved":0,"allocated":323211264,"data_stored":323031295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":447475406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,3,5,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":322621440,"data_stored":322621440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270585856,"data_stored":270585856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":322895872,"data_stored":322895872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":322424832,"data_stored":322424832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:30.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:46:30.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7697", 'reported_seq': 7778, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:25.966728+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:46:25.966728+0000', 'last_peered': '2024-09-16T04:46:25.966728+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:46:25.966728+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7697, 'log_dups_size': 0, 'ondisk_log_size': 7697, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40607744, 'num_objects': 4957, 'num_object_clones': 0, 'num_object_copies': 19828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2201, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7697, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2756}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:30.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:46:30.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7703", 'reported_seq': 7776, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:25.638518+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:46:25.638518+0000', 'last_peered': '2024-09-16T04:46:25.638518+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:46:25.638518+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7703, 'log_dups_size': 0, 'ondisk_log_size': 7703, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40719628, 'num_objects': 4971, 'num_object_clones': 0, 'num_object_copies': 19884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2210, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7701, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2763}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:30.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:46:30.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7680", 'reported_seq': 7765, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:25.574513+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:46:25.574513+0000', 'last_peered': '2024-09-16T04:46:25.574513+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:46:25.574513+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7680, 'log_dups_size': 0, 'ondisk_log_size': 7680, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40876702, 'num_objects': 4991, 'num_object_clones': 0, 'num_object_copies': 19964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4991, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7679, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2768}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:30.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:46:30.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7713", 'reported_seq': 7784, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:25.806097+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:46:25.806097+0000', 'last_peered': '2024-09-16T04:46:25.806097+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:46:25.806097+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7713, 'log_dups_size': 0, 'ondisk_log_size': 7713, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40604940, 'num_objects': 4957, 'num_object_clones': 0, 'num_object_copies': 19828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7711, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2730}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:30.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:46:30.040 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7784", 'reported_seq': 7879, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:25.758965+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:46:25.758965+0000', 'last_peered': '2024-09-16T04:46:25.758965+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:46:25.758965+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7784, 'log_dups_size': 0, 'ondisk_log_size': 7784, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40604940, 'num_objects': 4957, 'num_object_clones': 0, 'num_object_copies': 19828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2266, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7782, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2693}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:30.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:46:30.040 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7515", 'reported_seq': 7598, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:25.946885+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:46:25.946885+0000', 'last_peered': '2024-09-16T04:46:25.946885+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:46:25.946885+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7515, 'log_dups_size': 0, 'ondisk_log_size': 7515, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39297024, 'num_objects': 4797, 'num_object_clones': 0, 'num_object_copies': 19188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2189, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4797, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7515, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2608}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:30.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:46:30.040 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7652", 'reported_seq': 7722, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:25.753909+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:46:25.753909+0000', 'last_peered': '2024-09-16T04:46:25.753909+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:46:25.753909+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7652, 'log_dups_size': 0, 'ondisk_log_size': 7652, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40531212, 'num_objects': 4948, 'num_object_clones': 0, 'num_object_copies': 19792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7650, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2724}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:30.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:46:30.040 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7872", 'reported_seq': 10553, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:26.047096+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:46:26.047096+0000', 'last_peered': '2024-09-16T04:46:26.047096+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:46:26.047096+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7872, 'log_dups_size': 0, 'ondisk_log_size': 7872, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40004120, 'num_objects': 4884, 'num_object_clones': 0, 'num_object_copies': 19536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1031, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4884, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7869, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1315, 'num_bytes_recovered': 10649600, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3853}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:30.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:46:30.040 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7757", 'reported_seq': 7854, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:25.570698+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:46:25.570698+0000', 'last_peered': '2024-09-16T04:46:25.570698+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:46:25.570698+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7757, 'log_dups_size': 0, 'ondisk_log_size': 7757, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40965388, 'num_objects': 5001, 'num_object_clones': 0, 'num_object_copies': 20004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2193, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7755, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2810}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:30.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:46:30.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7618", 'reported_seq': 7721, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:25.813865+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:46:25.813865+0000', 'last_peered': '2024-09-16T04:46:25.813865+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:46:25.813865+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7618, 'log_dups_size': 0, 'ondisk_log_size': 7618, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39915410, 'num_objects': 4873, 'num_object_clones': 0, 'num_object_copies': 19492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2170, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7615, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2706}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:30.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:46:30.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7475", 'reported_seq': 7542, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:26.014758+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:46:26.014758+0000', 'last_peered': '2024-09-16T04:46:26.014758+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:46:26.014758+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7475, 'log_dups_size': 0, 'ondisk_log_size': 7475, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39778950, 'num_objects': 4856, 'num_object_clones': 0, 'num_object_copies': 19424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2116, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4856, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7474, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2741}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:30.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:46:30.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7632", 'reported_seq': 7713, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:25.405072+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:46:25.405072+0000', 'last_peered': '2024-09-16T04:46:25.405072+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:46:25.405072+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7632, 'log_dups_size': 0, 'ondisk_log_size': 7632, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40147590, 'num_objects': 4901, 'num_object_clones': 0, 'num_object_copies': 19604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4901, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7631, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2679}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:30.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:46:30.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7759", 'reported_seq': 7862, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:25.939360+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:46:25.939360+0000', 'last_peered': '2024-09-16T04:46:25.939360+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:46:25.939360+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7759, 'log_dups_size': 0, 'ondisk_log_size': 7759, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41145612, 'num_objects': 5023, 'num_object_clones': 0, 'num_object_copies': 20092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5023, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7757, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2813}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:30.042 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:46:30.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7568", 'reported_seq': 7658, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:25.557712+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:46:25.557712+0000', 'last_peered': '2024-09-16T04:46:25.557712+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:46:25.557712+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7568, 'log_dups_size': 0, 'ondisk_log_size': 7568, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39982348, 'num_objects': 4881, 'num_object_clones': 0, 'num_object_copies': 19524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4881, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7566, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2754}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:30.042 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:46:30.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7504", 'reported_seq': 7577, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:25.858872+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:46:25.858872+0000', 'last_peered': '2024-09-16T04:46:25.858872+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:46:25.858872+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7504, 'log_dups_size': 0, 'ondisk_log_size': 7504, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39343372, 'num_objects': 4803, 'num_object_clones': 0, 'num_object_copies': 19212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7502, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2567}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:30.042 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:46:30.744 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:30.745 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31857/313952 objects degraded (10.147%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":403,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":78488,"data_bytes":642926272,"bytes_used":2879627264,"bytes_avail":383667429376,"bytes_total":386547056640,"degraded_objects":31857,"degraded_total":313952,"degraded_ratio":0.10147092549179494,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":66884,"recovering_keys_per_sec":0,"num_objects_recovered":119,"num_bytes_recovered":802816,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":218},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:46:30.771 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:46:30.771 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:31.364 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:31.365 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:31.387 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-09-16T04:46:30.664306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":642926272,"num_objects":78488,"num_object_clones":0,"num_object_copies":313952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122999,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3725,"num_bytes_recovered":28983296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123026,"ondisk_log_size":123026,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2812136,"kb_used_data":1207400,"kb_used_omap":14,"kb_used_meta":1604657,"kb_avail":374675224,"statfs":{"total":386547056640,"available":383667429376,"internally_reserved":0,"allocated":1236377600,"data_stored":1235657724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1643169329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,8,9,8,4,5,0,0,5,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21512192,"num_objects":-2626,"num_object_clones":0,"num_object_copies":-10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2626,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002998"},"pg_stats":[{"pgid":"1.f","version":"33'7697","reported_seq":7778,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.966728+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:25.966728+0000","last_peered":"2024-09-16T04:46:25.966728+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:25.966728+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40607744,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7697,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2756}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7703","reported_seq":7776,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.638518+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:25.638518+0000","last_peered":"2024-09-16T04:46:25.638518+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:25.638518+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7703,"log_dups_size":0,"ondisk_log_size":7703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40719628,"num_objects":4971,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7701,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7745","reported_seq":7830,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.580489+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:30.580489+0000","last_peered":"2024-09-16T04:46:30.580489+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:30.580489+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7745,"log_dups_size":0,"ondisk_log_size":7745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40344222,"num_objects":4926,"num_object_clones":0,"num_object_copies":19704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4926,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7744,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2703}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7713","reported_seq":7784,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.806097+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:25.806097+0000","last_peered":"2024-09-16T04:46:25.806097+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:25.806097+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40604940,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7711,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2730}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7784","reported_seq":7879,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.758965+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:25.758965+0000","last_peered":"2024-09-16T04:46:25.758965+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:25.758965+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7784,"log_dups_size":0,"ondisk_log_size":7784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40604940,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7782,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7515","reported_seq":7598,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.946885+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:25.946885+0000","last_peered":"2024-09-16T04:46:25.946885+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:25.946885+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7515,"log_dups_size":0,"ondisk_log_size":7515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39297024,"num_objects":4797,"num_object_clones":0,"num_object_copies":19188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7515,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7652","reported_seq":7722,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.753909+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:25.753909+0000","last_peered":"2024-09-16T04:46:25.753909+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:25.753909+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7652,"log_dups_size":0,"ondisk_log_size":7652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40531212,"num_objects":4948,"num_object_clones":0,"num_object_copies":19792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7650,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2724}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7826","reported_seq":12383,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:27.738395+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:27.738395+0000","last_peered":"2024-09-16T04:46:27.738395+0000","last_clean":"2024-09-16T04:46:27.738395+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:27.738395+0000","last_undegraded":"2024-09-16T04:46:27.738395+0000","last_fullsized":"2024-09-16T04:46:27.738395+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7826,"log_dups_size":0,"ondisk_log_size":7826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:31.394739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40621324,"num_objects":4959,"num_object_clones":0,"num_object_copies":19836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7824,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7872","reported_seq":10553,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:26.047096+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:26.047096+0000","last_peered":"2024-09-16T04:46:26.047096+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:26.047096+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7872,"log_dups_size":0,"ondisk_log_size":7872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40004120,"num_objects":4884,"num_object_clones":0,"num_object_copies":19536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1315,"num_bytes_recovered":10649600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7809","reported_seq":7907,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.634391+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:30.634391+0000","last_peered":"2024-09-16T04:46:30.634391+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:30.634391+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7809,"log_dups_size":0,"ondisk_log_size":7809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40539404,"num_objects":4949,"num_object_clones":0,"num_object_copies":19796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7807,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7618","reported_seq":7721,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.813865+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:25.813865+0000","last_peered":"2024-09-16T04:46:25.813865+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:25.813865+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7618,"log_dups_size":0,"ondisk_log_size":7618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39915410,"num_objects":4873,"num_object_clones":0,"num_object_copies":19492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7615,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7475","reported_seq":7542,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:26.014758+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:26.014758+0000","last_peered":"2024-09-16T04:46:26.014758+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:26.014758+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7475,"log_dups_size":0,"ondisk_log_size":7475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39778950,"num_objects":4856,"num_object_clones":0,"num_object_copies":19424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7474,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2741}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7707","reported_seq":7788,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.595849+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:30.595849+0000","last_peered":"2024-09-16T04:46:30.595849+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:30.595849+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7707,"log_dups_size":0,"ondisk_log_size":7707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39533190,"num_objects":4826,"num_object_clones":0,"num_object_copies":19304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7706,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2604}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7759","reported_seq":7862,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.939360+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:25.939360+0000","last_peered":"2024-09-16T04:46:25.939360+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:25.939360+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7759,"log_dups_size":0,"ondisk_log_size":7759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41145612,"num_objects":5023,"num_object_clones":0,"num_object_copies":20092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7757,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2813}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7647","reported_seq":7737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.542170+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:30.542170+0000","last_peered":"2024-09-16T04:46:30.542170+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:30.542170+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7647,"log_dups_size":0,"ondisk_log_size":7647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39335180,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2675}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7504","reported_seq":7577,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:25.858872+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:25.858872+0000","last_peered":"2024-09-16T04:46:25.858872+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:25.858872+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7504,"log_dups_size":0,"ondisk_log_size":7504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39343372,"num_objects":4803,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7502,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":642926272,"num_objects":78488,"num_object_clones":0,"num_object_copies":313952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122999,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3725,"num_bytes_recovered":28983296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1233674240,"data_stored":1233674240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123026,"ondisk_log_size":123026,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748940,"kb_used_data":316028,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93622900,"statfs":{"total":96636764160,"available":95869849600,"internally_reserved":0,"allocated":323612672,"data_stored":323432703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443281103},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,0,4,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743596,"kb_used_data":311196,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93628244,"statfs":{"total":96636764160,"available":95875321856,"internally_reserved":0,"allocated":318664704,"data_stored":318484735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,5,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051645,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566956,"kb_used_data":264540,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93804884,"statfs":{"total":96636764160,"available":96056201216,"internally_reserved":0,"allocated":270888960,"data_stored":270708991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309653199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752644,"kb_used_data":315636,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93619196,"statfs":{"total":96636764160,"available":95866056704,"internally_reserved":0,"allocated":323211264,"data_stored":323031295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":447475406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,3,5,2,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":322621440,"data_stored":322621440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270585856,"data_stored":270585856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318042112,"data_stored":318042112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":322424832,"data_stored":322424832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:34.389 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:34.968 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:34.969 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:34.994 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-16T04:46:34.665287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":635422400,"num_objects":77572,"num_object_clones":0,"num_object_copies":310288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123915,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3772,"num_bytes_recovered":29302784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123942,"ondisk_log_size":123942,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2802476,"kb_used_data":1197724,"kb_used_omap":14,"kb_used_meta":1604657,"kb_avail":374684884,"statfs":{"total":386547056640,"available":383677321216,"internally_reserved":0,"allocated":1226469376,"data_stored":1225749500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1643169329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,8,13,15,6,5,2,0,0,0,1,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19906560,"num_objects":-2430,"num_object_clones":0,"num_object_copies":-9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2430,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":704512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003024"},"pg_stats":[{"pgid":"1.f","version":"33'7771","reported_seq":7854,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.975880+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:30.975880+0000","last_peered":"2024-09-16T04:46:30.975880+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:30.975880+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7771,"log_dups_size":0,"ondisk_log_size":7771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40001536,"num_objects":4883,"num_object_clones":0,"num_object_copies":19532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7771,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7776","reported_seq":7851,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.863273+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:30.863273+0000","last_peered":"2024-09-16T04:46:30.863273+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:30.863273+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7776,"log_dups_size":0,"ondisk_log_size":7776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40121612,"num_objects":4898,"num_object_clones":0,"num_object_copies":19592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7745","reported_seq":7830,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.580489+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:30.580489+0000","last_peered":"2024-09-16T04:46:30.580489+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:30.580489+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7745,"log_dups_size":0,"ondisk_log_size":7745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40344222,"num_objects":4926,"num_object_clones":0,"num_object_copies":19704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4926,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7744,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2703}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7792","reported_seq":7863,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.834811+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:30.834811+0000","last_peered":"2024-09-16T04:46:30.834811+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:30.834811+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7792,"log_dups_size":0,"ondisk_log_size":7792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39957772,"num_objects":4878,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7790,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7853","reported_seq":7950,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.975897+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:30.975897+0000","last_peered":"2024-09-16T04:46:30.975897+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:30.975897+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7853,"log_dups_size":0,"ondisk_log_size":7853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40039692,"num_objects":4888,"num_object_clones":0,"num_object_copies":19552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7851,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2625}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7592","reported_seq":7677,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:31.111526+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:31.111526+0000","last_peered":"2024-09-16T04:46:31.111526+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:31.111526+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7592,"log_dups_size":0,"ondisk_log_size":7592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38666240,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7592,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7738","reported_seq":7808,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.741355+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:30.741355+0000","last_peered":"2024-09-16T04:46:30.741355+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:30.741355+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7738,"log_dups_size":0,"ondisk_log_size":7738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39826700,"num_objects":4862,"num_object_clones":0,"num_object_copies":19448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7736,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2638}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7910","reported_seq":12467,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:32.640071+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:32.640071+0000","last_peered":"2024-09-16T04:46:32.640071+0000","last_clean":"2024-09-16T04:46:32.640071+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:32.640071+0000","last_undegraded":"2024-09-16T04:46:32.640071+0000","last_fullsized":"2024-09-16T04:46:32.640071+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:37.264253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39933196,"num_objects":4875,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7958","reported_seq":10719,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:31.070240+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:31.070240+0000","last_peered":"2024-09-16T04:46:31.070240+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:31.070240+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7958,"log_dups_size":0,"ondisk_log_size":7958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39299608,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":992,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7955,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":10969088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7809","reported_seq":7907,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.634391+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:30.634391+0000","last_peered":"2024-09-16T04:46:30.634391+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:30.634391+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7809,"log_dups_size":0,"ondisk_log_size":7809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40539404,"num_objects":4949,"num_object_clones":0,"num_object_copies":19796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7807,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7698","reported_seq":7805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:31.006534+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:31.006534+0000","last_peered":"2024-09-16T04:46:31.006534+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:31.006534+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7698,"log_dups_size":0,"ondisk_log_size":7698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39260050,"num_objects":4793,"num_object_clones":0,"num_object_copies":19172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7695,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7537","reported_seq":7606,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:31.024872+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:31.024872+0000","last_peered":"2024-09-16T04:46:31.024872+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:31.024872+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39271046,"num_objects":4794,"num_object_clones":0,"num_object_copies":19176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7536,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2680}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7707","reported_seq":7788,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.595849+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:30.595849+0000","last_peered":"2024-09-16T04:46:30.595849+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:30.595849+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7707,"log_dups_size":0,"ondisk_log_size":7707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39533190,"num_objects":4826,"num_object_clones":0,"num_object_copies":19304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7706,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2604}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7830","reported_seq":7935,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.965481+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:30.965481+0000","last_peered":"2024-09-16T04:46:30.965481+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:30.965481+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7830,"log_dups_size":0,"ondisk_log_size":7830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40563980,"num_objects":4952,"num_object_clones":0,"num_object_copies":19808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7828,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2743}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7647","reported_seq":7737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.542170+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:30.542170+0000","last_peered":"2024-09-16T04:46:30.542170+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:30.542170+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7647,"log_dups_size":0,"ondisk_log_size":7647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39335180,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2675}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7579","reported_seq":7652,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.798713+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:30.798713+0000","last_peered":"2024-09-16T04:46:30.798713+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:30.798713+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7579,"log_dups_size":0,"ondisk_log_size":7579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38728972,"num_objects":4728,"num_object_clones":0,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7577,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2492}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":635422400,"num_objects":77572,"num_object_clones":0,"num_object_copies":310288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123915,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3772,"num_bytes_recovered":29302784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1223733248,"data_stored":1223733248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123942,"ondisk_log_size":123942,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748328,"kb_used_data":311256,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93623512,"statfs":{"total":96636764160,"available":95870476288,"internally_reserved":0,"allocated":318726144,"data_stored":318546175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447540943},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,13,3,3,2,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743596,"kb_used_data":311196,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93628244,"statfs":{"total":96636764160,"available":95875321856,"internally_reserved":0,"allocated":318664704,"data_stored":318484735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,5,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570840,"kb_used_data":264264,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93801000,"statfs":{"total":96636764160,"available":96052224000,"internally_reserved":0,"allocated":270606336,"data_stored":270426367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313913039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739712,"kb_used_data":311008,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93632128,"statfs":{"total":96636764160,"available":95879299072,"internally_reserved":0,"allocated":318472192,"data_stored":318292223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":438955726},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,2,1,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317820928,"data_stored":317820928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270295040,"data_stored":270295040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318042112,"data_stored":318042112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":317575168,"data_stored":317575168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:34.995 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:35.550 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:35.551 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:35.583 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-16T04:46:34.665287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":635422400,"num_objects":77572,"num_object_clones":0,"num_object_copies":310288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123915,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3772,"num_bytes_recovered":29302784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123942,"ondisk_log_size":123942,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2802476,"kb_used_data":1197724,"kb_used_omap":14,"kb_used_meta":1604657,"kb_avail":374684884,"statfs":{"total":386547056640,"available":383677321216,"internally_reserved":0,"allocated":1226469376,"data_stored":1225749500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1643169329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,8,13,15,6,5,2,0,0,0,1,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19906560,"num_objects":-2430,"num_object_clones":0,"num_object_copies":-9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2430,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":704512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003024"},"pg_stats":[{"pgid":"1.f","version":"33'7771","reported_seq":7854,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.975880+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:30.975880+0000","last_peered":"2024-09-16T04:46:30.975880+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:30.975880+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7771,"log_dups_size":0,"ondisk_log_size":7771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40001536,"num_objects":4883,"num_object_clones":0,"num_object_copies":19532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7771,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7776","reported_seq":7851,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.863273+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:30.863273+0000","last_peered":"2024-09-16T04:46:30.863273+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:30.863273+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7776,"log_dups_size":0,"ondisk_log_size":7776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40121612,"num_objects":4898,"num_object_clones":0,"num_object_copies":19592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7745","reported_seq":7830,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.580489+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:30.580489+0000","last_peered":"2024-09-16T04:46:30.580489+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:30.580489+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7745,"log_dups_size":0,"ondisk_log_size":7745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40344222,"num_objects":4926,"num_object_clones":0,"num_object_copies":19704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4926,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7744,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2703}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7792","reported_seq":7863,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.834811+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:30.834811+0000","last_peered":"2024-09-16T04:46:30.834811+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:30.834811+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7792,"log_dups_size":0,"ondisk_log_size":7792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39957772,"num_objects":4878,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7790,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7853","reported_seq":7950,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.975897+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:30.975897+0000","last_peered":"2024-09-16T04:46:30.975897+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:30.975897+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7853,"log_dups_size":0,"ondisk_log_size":7853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40039692,"num_objects":4888,"num_object_clones":0,"num_object_copies":19552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7851,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2625}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7592","reported_seq":7677,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:31.111526+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:31.111526+0000","last_peered":"2024-09-16T04:46:31.111526+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:31.111526+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7592,"log_dups_size":0,"ondisk_log_size":7592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38666240,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7592,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7738","reported_seq":7808,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.741355+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:30.741355+0000","last_peered":"2024-09-16T04:46:30.741355+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:30.741355+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7738,"log_dups_size":0,"ondisk_log_size":7738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39826700,"num_objects":4862,"num_object_clones":0,"num_object_copies":19448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7736,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2638}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7910","reported_seq":12467,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:32.640071+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:32.640071+0000","last_peered":"2024-09-16T04:46:32.640071+0000","last_clean":"2024-09-16T04:46:32.640071+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:32.640071+0000","last_undegraded":"2024-09-16T04:46:32.640071+0000","last_fullsized":"2024-09-16T04:46:32.640071+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:37.264253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39933196,"num_objects":4875,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'7958","reported_seq":10719,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:31.070240+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:31.070240+0000","last_peered":"2024-09-16T04:46:31.070240+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:31.070240+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7958,"log_dups_size":0,"ondisk_log_size":7958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39299608,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":992,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7955,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":10969088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7809","reported_seq":7907,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.634391+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:30.634391+0000","last_peered":"2024-09-16T04:46:30.634391+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:30.634391+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7809,"log_dups_size":0,"ondisk_log_size":7809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40539404,"num_objects":4949,"num_object_clones":0,"num_object_copies":19796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7807,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7698","reported_seq":7805,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:31.006534+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:31.006534+0000","last_peered":"2024-09-16T04:46:31.006534+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:31.006534+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7698,"log_dups_size":0,"ondisk_log_size":7698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39260050,"num_objects":4793,"num_object_clones":0,"num_object_copies":19172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7695,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7537","reported_seq":7606,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:31.024872+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:31.024872+0000","last_peered":"2024-09-16T04:46:31.024872+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:31.024872+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7537,"log_dups_size":0,"ondisk_log_size":7537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39271046,"num_objects":4794,"num_object_clones":0,"num_object_copies":19176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7536,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2680}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7707","reported_seq":7788,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.595849+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:30.595849+0000","last_peered":"2024-09-16T04:46:30.595849+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:30.595849+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7707,"log_dups_size":0,"ondisk_log_size":7707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39533190,"num_objects":4826,"num_object_clones":0,"num_object_copies":19304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7706,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2604}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7830","reported_seq":7935,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.965481+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:30.965481+0000","last_peered":"2024-09-16T04:46:30.965481+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:30.965481+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7830,"log_dups_size":0,"ondisk_log_size":7830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40563980,"num_objects":4952,"num_object_clones":0,"num_object_copies":19808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7828,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2743}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7647","reported_seq":7737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.542170+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:30.542170+0000","last_peered":"2024-09-16T04:46:30.542170+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:30.542170+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7647,"log_dups_size":0,"ondisk_log_size":7647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39335180,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2675}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7579","reported_seq":7652,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:30.798713+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:30.798713+0000","last_peered":"2024-09-16T04:46:30.798713+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:30.798713+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7579,"log_dups_size":0,"ondisk_log_size":7579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38728972,"num_objects":4728,"num_object_clones":0,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7577,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2492}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":635422400,"num_objects":77572,"num_object_clones":0,"num_object_copies":310288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123915,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3772,"num_bytes_recovered":29302784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1223733248,"data_stored":1223733248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123942,"ondisk_log_size":123942,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748328,"kb_used_data":311256,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93623512,"statfs":{"total":96636764160,"available":95870476288,"internally_reserved":0,"allocated":318726144,"data_stored":318546175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447540943},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,13,3,3,2,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743596,"kb_used_data":311196,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93628244,"statfs":{"total":96636764160,"available":95875321856,"internally_reserved":0,"allocated":318664704,"data_stored":318484735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,5,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570840,"kb_used_data":264264,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93801000,"statfs":{"total":96636764160,"available":96052224000,"internally_reserved":0,"allocated":270606336,"data_stored":270426367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313913039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739712,"kb_used_data":311008,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93632128,"statfs":{"total":96636764160,"available":95879299072,"internally_reserved":0,"allocated":318472192,"data_stored":318292223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":438955726},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,2,1,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317820928,"data_stored":317820928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270295040,"data_stored":270295040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318042112,"data_stored":318042112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":317575168,"data_stored":317575168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:35.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:46:35.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7771", 'reported_seq': 7854, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:30.975880+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:46:30.975880+0000', 'last_peered': '2024-09-16T04:46:30.975880+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:46:30.975880+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7771, 'log_dups_size': 0, 'ondisk_log_size': 7771, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40001536, 'num_objects': 4883, 'num_object_clones': 0, 'num_object_copies': 19532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7771, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2683}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:35.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:46:35.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7776", 'reported_seq': 7851, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:30.863273+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:46:30.863273+0000', 'last_peered': '2024-09-16T04:46:30.863273+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:46:30.863273+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7776, 'log_dups_size': 0, 'ondisk_log_size': 7776, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40121612, 'num_objects': 4898, 'num_object_clones': 0, 'num_object_copies': 19592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2209, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4898, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7774, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2691}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:35.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:46:35.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7745", 'reported_seq': 7830, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:30.580489+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:46:30.580489+0000', 'last_peered': '2024-09-16T04:46:30.580489+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:46:30.580489+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7745, 'log_dups_size': 0, 'ondisk_log_size': 7745, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40344222, 'num_objects': 4926, 'num_object_clones': 0, 'num_object_copies': 19704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4926, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7744, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2703}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:35.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:46:35.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7792", 'reported_seq': 7863, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:30.834811+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:46:30.834811+0000', 'last_peered': '2024-09-16T04:46:30.834811+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:46:30.834811+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7792, 'log_dups_size': 0, 'ondisk_log_size': 7792, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39957772, 'num_objects': 4878, 'num_object_clones': 0, 'num_object_copies': 19512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4878, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7790, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2651}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:35.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:46:35.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7853", 'reported_seq': 7950, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:30.975897+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:46:30.975897+0000', 'last_peered': '2024-09-16T04:46:30.975897+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:46:30.975897+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7853, 'log_dups_size': 0, 'ondisk_log_size': 7853, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40039692, 'num_objects': 4888, 'num_object_clones': 0, 'num_object_copies': 19552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4888, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7851, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2625}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:35.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:46:35.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7592", 'reported_seq': 7677, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:31.111526+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:46:31.111526+0000', 'last_peered': '2024-09-16T04:46:31.111526+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:46:31.111526+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7592, 'log_dups_size': 0, 'ondisk_log_size': 7592, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38666240, 'num_objects': 4720, 'num_object_clones': 0, 'num_object_copies': 18880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2188, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7592, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2532}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:35.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:46:35.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7738", 'reported_seq': 7808, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:30.741355+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:46:30.741355+0000', 'last_peered': '2024-09-16T04:46:30.741355+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:46:30.741355+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7738, 'log_dups_size': 0, 'ondisk_log_size': 7738, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39826700, 'num_objects': 4862, 'num_object_clones': 0, 'num_object_copies': 19448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4862, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7736, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2638}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:35.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:46:35.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7958", 'reported_seq': 10719, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:31.070240+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:46:31.070240+0000', 'last_peered': '2024-09-16T04:46:31.070240+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:46:31.070240+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7958, 'log_dups_size': 0, 'ondisk_log_size': 7958, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39299608, 'num_objects': 4798, 'num_object_clones': 0, 'num_object_copies': 19192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 992, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7955, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1354, 'num_bytes_recovered': 10969088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3806}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:35.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:46:35.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7809", 'reported_seq': 7907, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:30.634391+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:46:30.634391+0000', 'last_peered': '2024-09-16T04:46:30.634391+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:46:30.634391+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7809, 'log_dups_size': 0, 'ondisk_log_size': 7809, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40539404, 'num_objects': 4949, 'num_object_clones': 0, 'num_object_copies': 19796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2193, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4949, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7807, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2758}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:35.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:46:35.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7698", 'reported_seq': 7805, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:31.006534+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:46:31.006534+0000', 'last_peered': '2024-09-16T04:46:31.006534+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:46:31.006534+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7698, 'log_dups_size': 0, 'ondisk_log_size': 7698, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39260050, 'num_objects': 4793, 'num_object_clones': 0, 'num_object_copies': 19172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2168, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7695, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2628}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:35.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:46:35.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7537", 'reported_seq': 7606, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:31.024872+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:46:31.024872+0000', 'last_peered': '2024-09-16T04:46:31.024872+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:46:31.024872+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7537, 'log_dups_size': 0, 'ondisk_log_size': 7537, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39271046, 'num_objects': 4794, 'num_object_clones': 0, 'num_object_copies': 19176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2115, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4794, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7536, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2680}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:35.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:46:35.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7707", 'reported_seq': 7788, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:30.595849+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:46:30.595849+0000', 'last_peered': '2024-09-16T04:46:30.595849+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:46:30.595849+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7707, 'log_dups_size': 0, 'ondisk_log_size': 7707, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39533190, 'num_objects': 4826, 'num_object_clones': 0, 'num_object_copies': 19304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7706, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2604}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:35.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:46:35.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7830", 'reported_seq': 7935, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:30.965481+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:46:30.965481+0000', 'last_peered': '2024-09-16T04:46:30.965481+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:46:30.965481+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7830, 'log_dups_size': 0, 'ondisk_log_size': 7830, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40563980, 'num_objects': 4952, 'num_object_clones': 0, 'num_object_copies': 19808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2211, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4952, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7828, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2743}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:35.587 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:46:35.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7647", 'reported_seq': 7737, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:30.542170+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:46:30.542170+0000', 'last_peered': '2024-09-16T04:46:30.542170+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:46:30.542170+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7647, 'log_dups_size': 0, 'ondisk_log_size': 7647, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39335180, 'num_objects': 4802, 'num_object_clones': 0, 'num_object_copies': 19208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7645, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2675}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:35.587 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:46:35.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7579", 'reported_seq': 7652, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:30.798713+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:46:30.798713+0000', 'last_peered': '2024-09-16T04:46:30.798713+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:46:30.798713+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7579, 'log_dups_size': 0, 'ondisk_log_size': 7579, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38728972, 'num_objects': 4728, 'num_object_clones': 0, 'num_object_copies': 18912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4728, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7577, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2492}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:35.587 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:46:36.283 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:36.283 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31810/310288 objects degraded (10.252%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":408,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":77572,"data_bytes":635422400,"bytes_used":2869735424,"bytes_avail":383677321216,"bytes_total":386547056640,"degraded_objects":31810,"degraded_total":310288,"degraded_ratio":0.10251766101170526,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":58694,"recovering_keys_per_sec":0,"num_objects_recovered":99,"num_bytes_recovered":704512,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":202},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:46:36.308 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:46:36.308 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:36.896 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:36.897 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:36.919 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-16T04:46:36.665771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":626616000,"num_objects":76497,"num_object_clones":0,"num_object_copies":305988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76497,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124990,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3847,"num_bytes_recovered":29786112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125017,"ondisk_log_size":125017,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788196,"kb_used_data":1183456,"kb_used_omap":14,"kb_used_meta":1604657,"kb_avail":374699164,"statfs":{"total":386547056640,"available":383691943936,"internally_reserved":0,"allocated":1211858944,"data_stored":1211139068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1643169329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,10,6,14,2,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28139520,"num_objects":-3435,"num_object_clones":0,"num_object_copies":-13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3435,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002995"},"pg_stats":[{"pgid":"1.f","version":"33'7834","reported_seq":7917,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.860902+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:35.860902+0000","last_peered":"2024-09-16T04:46:35.860902+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:35.860902+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7834,"log_dups_size":0,"ondisk_log_size":7834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39485440,"num_objects":4820,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7834,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7857","reported_seq":7932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.796457+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:35.796457+0000","last_peered":"2024-09-16T04:46:35.796457+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:35.796457+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7857,"log_dups_size":0,"ondisk_log_size":7857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39458060,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7855,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2610}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7811","reported_seq":7896,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.602879+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:35.602879+0000","last_peered":"2024-09-16T04:46:35.602879+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:35.602879+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7811,"log_dups_size":0,"ondisk_log_size":7811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39803550,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7810,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2637}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7851","reported_seq":7924,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.783251+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:35.783251+0000","last_peered":"2024-09-16T04:46:35.783251+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:35.783251+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7851,"log_dups_size":0,"ondisk_log_size":7851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39474444,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7849,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7935","reported_seq":8038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.148784+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:36.148784+0000","last_peered":"2024-09-16T04:46:36.148784+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:36.148784+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39367948,"num_objects":4806,"num_object_clones":0,"num_object_copies":19224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2546}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7667","reported_seq":7764,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.961930+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:35.961930+0000","last_peered":"2024-09-16T04:46:35.961930+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:35.961930+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7667,"log_dups_size":0,"ondisk_log_size":7667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38051840,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7667,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7801","reported_seq":7871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.795057+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:35.795057+0000","last_peered":"2024-09-16T04:46:35.795057+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:35.795057+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7801,"log_dups_size":0,"ondisk_log_size":7801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39310604,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7799,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2575}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7910","reported_seq":12467,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:32.640071+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:32.640071+0000","last_peered":"2024-09-16T04:46:32.640071+0000","last_clean":"2024-09-16T04:46:32.640071+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:32.640071+0000","last_undegraded":"2024-09-16T04:46:32.640071+0000","last_fullsized":"2024-09-16T04:46:32.640071+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:37.264253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39933196,"num_objects":4875,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8033","reported_seq":10912,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.125669+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:36.125669+0000","last_peered":"2024-09-16T04:46:36.125669+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:36.125669+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8033,"log_dups_size":0,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38685208,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8030,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1413,"num_bytes_recovered":11452416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3790}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7878","reported_seq":7976,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.581037+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:35.581037+0000","last_peered":"2024-09-16T04:46:35.581037+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:35.581037+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7878,"log_dups_size":0,"ondisk_log_size":7878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39974156,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7876,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7777","reported_seq":7883,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.205805+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:36.205805+0000","last_peered":"2024-09-16T04:46:36.205805+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:36.205805+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38612882,"num_objects":4714,"num_object_clones":0,"num_object_copies":18856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7602","reported_seq":7672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.189827+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:36.189827+0000","last_peered":"2024-09-16T04:46:36.189827+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:36.189827+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7602,"log_dups_size":0,"ondisk_log_size":7602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38738566,"num_objects":4729,"num_object_clones":0,"num_object_copies":18916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7601,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2616}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7783","reported_seq":7864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.480532+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:35.480532+0000","last_peered":"2024-09-16T04:46:35.480532+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:35.480532+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7783,"log_dups_size":0,"ondisk_log_size":7783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38910598,"num_objects":4750,"num_object_clones":0,"num_object_copies":19000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7782,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2528}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7903","reported_seq":8012,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.077324+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:36.077324+0000","last_peered":"2024-09-16T04:46:36.077324+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:36.077324+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7903,"log_dups_size":0,"ondisk_log_size":7903,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39965964,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7901,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2673}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7717","reported_seq":7807,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.593350+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:35.593350+0000","last_peered":"2024-09-16T04:46:35.593350+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:35.593350+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7717,"log_dups_size":0,"ondisk_log_size":7717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38761740,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7715,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2605}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7658","reported_seq":7734,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.900037+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:35.900037+0000","last_peered":"2024-09-16T04:46:35.900037+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:35.900037+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7658,"log_dups_size":0,"ondisk_log_size":7658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38081804,"num_objects":4649,"num_object_clones":0,"num_object_copies":18596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7656,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":626616000,"num_objects":76497,"num_object_clones":0,"num_object_copies":305988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76497,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124990,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3847,"num_bytes_recovered":29786112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1209802752,"data_stored":1209802752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125017,"ondisk_log_size":125017,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734644,"kb_used_data":305892,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93637196,"statfs":{"total":96636764160,"available":95884488704,"internally_reserved":0,"allocated":313233408,"data_stored":313053439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":439021263},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743280,"kb_used_data":306720,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93628560,"statfs":{"total":96636764160,"available":95875645440,"internally_reserved":0,"allocated":314081280,"data_stored":313901311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570840,"kb_used_data":264264,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93801000,"statfs":{"total":96636764160,"available":96052224000,"internally_reserved":0,"allocated":270606336,"data_stored":270426367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313913039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739432,"kb_used_data":306580,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93632408,"statfs":{"total":96636764160,"available":95879585792,"internally_reserved":0,"allocated":313937920,"data_stored":313757951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":443215566},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,4,3,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313159680,"data_stored":313159680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270295040,"data_stored":270295040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313438208,"data_stored":313438208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":312909824,"data_stored":312909824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:39.921 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:40.550 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:40.551 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:40.575 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-09-16T04:46:38.666333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":626034368,"num_objects":76426,"num_object_clones":0,"num_object_copies":305704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76426,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125061,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3847,"num_bytes_recovered":29786112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125088,"ondisk_log_size":125088,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2792076,"kb_used_data":1183176,"kb_used_omap":14,"kb_used_meta":1608817,"kb_avail":374695284,"statfs":{"total":386547056640,"available":383687970816,"internally_reserved":0,"allocated":1211572224,"data_stored":1210852348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1647429169},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,6,5,10,19,3,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19759104,"num_objects":-2412,"num_object_clones":0,"num_object_copies":-9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2412,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003068"},"pg_stats":[{"pgid":"1.f","version":"33'7834","reported_seq":7917,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.860902+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:35.860902+0000","last_peered":"2024-09-16T04:46:35.860902+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:35.860902+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7834,"log_dups_size":0,"ondisk_log_size":7834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39485440,"num_objects":4820,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7834,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7857","reported_seq":7932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.796457+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:35.796457+0000","last_peered":"2024-09-16T04:46:35.796457+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:35.796457+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7857,"log_dups_size":0,"ondisk_log_size":7857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39458060,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7855,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2610}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7811","reported_seq":7896,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.602879+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:35.602879+0000","last_peered":"2024-09-16T04:46:35.602879+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:35.602879+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7811,"log_dups_size":0,"ondisk_log_size":7811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39803550,"num_objects":4860,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7810,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2637}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7851","reported_seq":7924,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.783251+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:35.783251+0000","last_peered":"2024-09-16T04:46:35.783251+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:35.783251+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7851,"log_dups_size":0,"ondisk_log_size":7851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39474444,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7849,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7935","reported_seq":8038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.148784+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:36.148784+0000","last_peered":"2024-09-16T04:46:36.148784+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:36.148784+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39367948,"num_objects":4806,"num_object_clones":0,"num_object_copies":19224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2546}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7667","reported_seq":7764,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.961930+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:35.961930+0000","last_peered":"2024-09-16T04:46:35.961930+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:35.961930+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7667,"log_dups_size":0,"ondisk_log_size":7667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38051840,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7667,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7801","reported_seq":7871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.795057+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:35.795057+0000","last_peered":"2024-09-16T04:46:35.795057+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:35.795057+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7801,"log_dups_size":0,"ondisk_log_size":7801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39310604,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7799,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2575}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7981","reported_seq":12538,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:37.754398+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:37.754398+0000","last_peered":"2024-09-16T04:46:37.754398+0000","last_clean":"2024-09-16T04:46:37.754398+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:37.754398+0000","last_undegraded":"2024-09-16T04:46:37.754398+0000","last_fullsized":"2024-09-16T04:46:37.754398+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7981,"log_dups_size":0,"ondisk_log_size":7981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:42.341393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39351564,"num_objects":4804,"num_object_clones":0,"num_object_copies":19216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7979,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8033","reported_seq":10912,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.125669+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:36.125669+0000","last_peered":"2024-09-16T04:46:36.125669+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:36.125669+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8033,"log_dups_size":0,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38685208,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8030,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1413,"num_bytes_recovered":11452416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3790}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7878","reported_seq":7976,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.581037+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:35.581037+0000","last_peered":"2024-09-16T04:46:35.581037+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:35.581037+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7878,"log_dups_size":0,"ondisk_log_size":7878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39974156,"num_objects":4880,"num_object_clones":0,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7876,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7777","reported_seq":7883,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.205805+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:36.205805+0000","last_peered":"2024-09-16T04:46:36.205805+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:36.205805+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38612882,"num_objects":4714,"num_object_clones":0,"num_object_copies":18856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7602","reported_seq":7672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.189827+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:36.189827+0000","last_peered":"2024-09-16T04:46:36.189827+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:36.189827+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7602,"log_dups_size":0,"ondisk_log_size":7602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38738566,"num_objects":4729,"num_object_clones":0,"num_object_copies":18916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7601,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2616}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7783","reported_seq":7864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.480532+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:35.480532+0000","last_peered":"2024-09-16T04:46:35.480532+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:35.480532+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7783,"log_dups_size":0,"ondisk_log_size":7783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38910598,"num_objects":4750,"num_object_clones":0,"num_object_copies":19000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7782,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2528}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7903","reported_seq":8012,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.077324+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:36.077324+0000","last_peered":"2024-09-16T04:46:36.077324+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:36.077324+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7903,"log_dups_size":0,"ondisk_log_size":7903,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39965964,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7901,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2673}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7717","reported_seq":7807,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.593350+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:35.593350+0000","last_peered":"2024-09-16T04:46:35.593350+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:35.593350+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7717,"log_dups_size":0,"ondisk_log_size":7717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38761740,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7715,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2605}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7658","reported_seq":7734,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.900037+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:35.900037+0000","last_peered":"2024-09-16T04:46:35.900037+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:35.900037+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7658,"log_dups_size":0,"ondisk_log_size":7658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38081804,"num_objects":4649,"num_object_clones":0,"num_object_copies":18596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7656,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":626034368,"num_objects":76426,"num_object_clones":0,"num_object_copies":305704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76426,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125061,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3847,"num_bytes_recovered":29786112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1209528320,"data_stored":1209528320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125088,"ondisk_log_size":125088,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734644,"kb_used_data":305892,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93637196,"statfs":{"total":96636764160,"available":95884488704,"internally_reserved":0,"allocated":313233408,"data_stored":313053439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":439021263},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743280,"kb_used_data":306720,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93628560,"statfs":{"total":96636764160,"available":95875645440,"internally_reserved":0,"allocated":314081280,"data_stored":313901311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051647,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574720,"kb_used_data":263984,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93797120,"statfs":{"total":96636764160,"available":96048250880,"internally_reserved":0,"allocated":270319616,"data_stored":270139647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318172879},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739432,"kb_used_data":306580,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93632408,"statfs":{"total":96636764160,"available":95879585792,"internally_reserved":0,"allocated":313937920,"data_stored":313757951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":443215566},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,4,3,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313159680,"data_stored":313159680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270020608,"data_stored":270020608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313438208,"data_stored":313438208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":312909824,"data_stored":312909824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:40.576 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:41.141 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:41.141 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:41.173 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-16T04:46:40.666776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":623724224,"num_objects":76144,"num_object_clones":0,"num_object_copies":304576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76144,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125343,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3847,"num_bytes_recovered":29786112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125370,"ondisk_log_size":125370,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2774760,"kb_used_data":1178336,"kb_used_omap":14,"kb_used_meta":1596337,"kb_avail":374712600,"statfs":{"total":386547056640,"available":383705702400,"internally_reserved":0,"allocated":1206616064,"data_stored":1205896188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1634649649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,7,6,5,10,15,2,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21422080,"num_objects":-2615,"num_object_clones":0,"num_object_copies":-10460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2615,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002937"},"pg_stats":[{"pgid":"1.f","version":"33'7834","reported_seq":7917,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.860902+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:35.860902+0000","last_peered":"2024-09-16T04:46:35.860902+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:35.860902+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7834,"log_dups_size":0,"ondisk_log_size":7834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39485440,"num_objects":4820,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7834,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7857","reported_seq":7932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.796457+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:35.796457+0000","last_peered":"2024-09-16T04:46:35.796457+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:35.796457+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7857,"log_dups_size":0,"ondisk_log_size":7857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39458060,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7855,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2610}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7889","reported_seq":7974,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:40.548138+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:40.548138+0000","last_peered":"2024-09-16T04:46:40.548138+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:40.548138+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7889,"log_dups_size":0,"ondisk_log_size":7889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39164574,"num_objects":4782,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4782,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7888,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2559}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7851","reported_seq":7924,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.783251+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:35.783251+0000","last_peered":"2024-09-16T04:46:35.783251+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:35.783251+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7851,"log_dups_size":0,"ondisk_log_size":7851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39474444,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7849,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7935","reported_seq":8038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.148784+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:36.148784+0000","last_peered":"2024-09-16T04:46:36.148784+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:36.148784+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39367948,"num_objects":4806,"num_object_clones":0,"num_object_copies":19224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2546}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7667","reported_seq":7764,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.961930+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:35.961930+0000","last_peered":"2024-09-16T04:46:35.961930+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:35.961930+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7667,"log_dups_size":0,"ondisk_log_size":7667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38051840,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7667,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7801","reported_seq":7871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.795057+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:35.795057+0000","last_peered":"2024-09-16T04:46:35.795057+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:35.795057+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7801,"log_dups_size":0,"ondisk_log_size":7801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39310604,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7799,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2575}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7981","reported_seq":12538,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:37.754398+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:37.754398+0000","last_peered":"2024-09-16T04:46:37.754398+0000","last_clean":"2024-09-16T04:46:37.754398+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:37.754398+0000","last_undegraded":"2024-09-16T04:46:37.754398+0000","last_fullsized":"2024-09-16T04:46:37.754398+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7981,"log_dups_size":0,"ondisk_log_size":7981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:42.341393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39351564,"num_objects":4804,"num_object_clones":0,"num_object_copies":19216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7979,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8033","reported_seq":10912,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.125669+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:36.125669+0000","last_peered":"2024-09-16T04:46:36.125669+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:36.125669+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8033,"log_dups_size":0,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38685208,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8030,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1413,"num_bytes_recovered":11452416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3790}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7962","reported_seq":8060,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:40.560259+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:40.560259+0000","last_peered":"2024-09-16T04:46:40.560259+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:40.560259+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7962,"log_dups_size":0,"ondisk_log_size":7962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39286028,"num_objects":4796,"num_object_clones":0,"num_object_copies":19184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7960,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2605}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7777","reported_seq":7883,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.205805+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:36.205805+0000","last_peered":"2024-09-16T04:46:36.205805+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:36.205805+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38612882,"num_objects":4714,"num_object_clones":0,"num_object_copies":18856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7602","reported_seq":7672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.189827+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:36.189827+0000","last_peered":"2024-09-16T04:46:36.189827+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:36.189827+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7602,"log_dups_size":0,"ondisk_log_size":7602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38738566,"num_objects":4729,"num_object_clones":0,"num_object_copies":18916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7601,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2616}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7849","reported_seq":7930,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:40.540708+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:40.540708+0000","last_peered":"2024-09-16T04:46:40.540708+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:40.540708+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7849,"log_dups_size":0,"ondisk_log_size":7849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38369926,"num_objects":4684,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7848,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2462}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7903","reported_seq":8012,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.077324+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:36.077324+0000","last_peered":"2024-09-16T04:46:36.077324+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:36.077324+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7903,"log_dups_size":0,"ondisk_log_size":7903,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39965964,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7901,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2673}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7771","reported_seq":7861,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:40.643729+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:40.643729+0000","last_peered":"2024-09-16T04:46:40.643729+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:40.643729+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7771,"log_dups_size":0,"ondisk_log_size":7771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38319372,"num_objects":4678,"num_object_clones":0,"num_object_copies":18712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7769,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2551}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7658","reported_seq":7734,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.900037+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:35.900037+0000","last_peered":"2024-09-16T04:46:35.900037+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:35.900037+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7658,"log_dups_size":0,"ondisk_log_size":7658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38081804,"num_objects":4649,"num_object_clones":0,"num_object_copies":18596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7656,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":623724224,"num_objects":76144,"num_object_clones":0,"num_object_copies":304576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76144,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125343,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3847,"num_bytes_recovered":29786112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1204633600,"data_stored":1204633600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125370,"ondisk_log_size":125370,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734644,"kb_used_data":305892,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93637196,"statfs":{"total":96636764160,"available":95884488704,"internally_reserved":0,"allocated":313233408,"data_stored":313053439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":439021263},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725964,"kb_used_data":301880,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93645876,"statfs":{"total":96636764160,"available":95893377024,"internally_reserved":0,"allocated":309125120,"data_stored":308945151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051647,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574720,"kb_used_data":263984,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93797120,"statfs":{"total":96636764160,"available":96048250880,"internally_reserved":0,"allocated":270319616,"data_stored":270139647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318172879},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739432,"kb_used_data":306580,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93632408,"statfs":{"total":96636764160,"available":95879585792,"internally_reserved":0,"allocated":313937920,"data_stored":313757951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":443215566},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,4,3,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313159680,"data_stored":313159680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270020608,"data_stored":270020608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308543488,"data_stored":308543488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":312909824,"data_stored":312909824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:41.174 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:46:41.174 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7834", 'reported_seq': 7917, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:35.860902+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:46:35.860902+0000', 'last_peered': '2024-09-16T04:46:35.860902+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:46:35.860902+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7834, 'log_dups_size': 0, 'ondisk_log_size': 7834, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39485440, 'num_objects': 4820, 'num_object_clones': 0, 'num_object_copies': 19280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4820, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7834, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2620}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:41.175 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:46:41.175 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7857", 'reported_seq': 7932, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:35.796457+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:46:35.796457+0000', 'last_peered': '2024-09-16T04:46:35.796457+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:46:35.796457+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7857, 'log_dups_size': 0, 'ondisk_log_size': 7857, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39458060, 'num_objects': 4817, 'num_object_clones': 0, 'num_object_copies': 19268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2209, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7855, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2610}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:41.175 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:46:41.175 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7889", 'reported_seq': 7974, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:40.548138+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:46:40.548138+0000', 'last_peered': '2024-09-16T04:46:40.548138+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:46:40.548138+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7889, 'log_dups_size': 0, 'ondisk_log_size': 7889, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39164574, 'num_objects': 4782, 'num_object_clones': 0, 'num_object_copies': 19128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4782, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7888, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2559}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:41.175 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:46:41.175 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7851", 'reported_seq': 7924, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:35.783251+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:46:35.783251+0000', 'last_peered': '2024-09-16T04:46:35.783251+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:46:35.783251+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7851, 'log_dups_size': 0, 'ondisk_log_size': 7851, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39474444, 'num_objects': 4819, 'num_object_clones': 0, 'num_object_copies': 19276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7849, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:41.175 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:46:41.175 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7935", 'reported_seq': 8038, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:36.148784+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:46:36.148784+0000', 'last_peered': '2024-09-16T04:46:36.148784+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:46:36.148784+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7935, 'log_dups_size': 0, 'ondisk_log_size': 7935, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39367948, 'num_objects': 4806, 'num_object_clones': 0, 'num_object_copies': 19224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4806, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7933, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2546}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:41.175 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:46:41.175 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7667", 'reported_seq': 7764, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:35.961930+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:46:35.961930+0000', 'last_peered': '2024-09-16T04:46:35.961930+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:46:35.961930+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7667, 'log_dups_size': 0, 'ondisk_log_size': 7667, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38051840, 'num_objects': 4645, 'num_object_clones': 0, 'num_object_copies': 18580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2182, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4645, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7667, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2463}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:41.176 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:46:41.176 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7801", 'reported_seq': 7871, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:35.795057+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:46:35.795057+0000', 'last_peered': '2024-09-16T04:46:35.795057+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:46:35.795057+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7801, 'log_dups_size': 0, 'ondisk_log_size': 7801, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39310604, 'num_objects': 4799, 'num_object_clones': 0, 'num_object_copies': 19196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4799, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7799, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2575}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:41.176 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:46:41.176 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8033", 'reported_seq': 10912, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:36.125669+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:46:36.125669+0000', 'last_peered': '2024-09-16T04:46:36.125669+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:46:36.125669+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8033, 'log_dups_size': 0, 'ondisk_log_size': 8033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38685208, 'num_objects': 4723, 'num_object_clones': 0, 'num_object_copies': 18892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 933, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4723, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8030, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1413, 'num_bytes_recovered': 11452416, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3790}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:41.176 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:46:41.176 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7962", 'reported_seq': 8060, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:40.560259+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:46:40.560259+0000', 'last_peered': '2024-09-16T04:46:40.560259+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:46:40.560259+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7962, 'log_dups_size': 0, 'ondisk_log_size': 7962, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39286028, 'num_objects': 4796, 'num_object_clones': 0, 'num_object_copies': 19184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2193, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4796, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7960, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2605}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:41.176 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:46:41.176 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7777", 'reported_seq': 7883, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:36.205805+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:46:36.205805+0000', 'last_peered': '2024-09-16T04:46:36.205805+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:46:36.205805+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7777, 'log_dups_size': 0, 'ondisk_log_size': 7777, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38612882, 'num_objects': 4714, 'num_object_clones': 0, 'num_object_copies': 18856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2168, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4714, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7774, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2549}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:41.176 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:46:41.176 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7602", 'reported_seq': 7672, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:36.189827+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:46:36.189827+0000', 'last_peered': '2024-09-16T04:46:36.189827+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:46:36.189827+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7602, 'log_dups_size': 0, 'ondisk_log_size': 7602, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38738566, 'num_objects': 4729, 'num_object_clones': 0, 'num_object_copies': 18916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2114, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4729, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7601, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2616}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:41.177 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:46:41.177 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7849", 'reported_seq': 7930, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:40.540708+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:46:40.540708+0000', 'last_peered': '2024-09-16T04:46:40.540708+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:46:40.540708+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7849, 'log_dups_size': 0, 'ondisk_log_size': 7849, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38369926, 'num_objects': 4684, 'num_object_clones': 0, 'num_object_copies': 18736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7848, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2462}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:41.177 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:46:41.177 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7903", 'reported_seq': 8012, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:36.077324+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:46:36.077324+0000', 'last_peered': '2024-09-16T04:46:36.077324+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:46:36.077324+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7903, 'log_dups_size': 0, 'ondisk_log_size': 7903, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39965964, 'num_objects': 4879, 'num_object_clones': 0, 'num_object_copies': 19516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7901, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2673}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:41.177 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:46:41.177 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7771", 'reported_seq': 7861, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:40.643729+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:46:40.643729+0000', 'last_peered': '2024-09-16T04:46:40.643729+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:46:40.643729+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7771, 'log_dups_size': 0, 'ondisk_log_size': 7771, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38319372, 'num_objects': 4678, 'num_object_clones': 0, 'num_object_copies': 18712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7769, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2551}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:41.177 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:46:41.177 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7658", 'reported_seq': 7734, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:35.900037+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:46:35.900037+0000', 'last_peered': '2024-09-16T04:46:35.900037+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:46:35.900037+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7658, 'log_dups_size': 0, 'ondisk_log_size': 7658, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38081804, 'num_objects': 4649, 'num_object_clones': 0, 'num_object_copies': 18596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2236, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4649, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7656, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2415}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:41.178 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:46:41.900 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:41.900 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31735/304576 objects degraded (10.419%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":414,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":76144,"data_bytes":623724224,"bytes_used":2841354240,"bytes_avail":383705702400,"bytes_total":386547056640,"degraded_objects":31735,"degraded_total":304576,"degraded_ratio":0.10419402710653498,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":66884,"recovering_keys_per_sec":0,"num_objects_recovered":122,"num_bytes_recovered":802816,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":217},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:46:41.923 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:46:41.923 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:42.565 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:42.565 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:42.587 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-16T04:46:40.666776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":623724224,"num_objects":76144,"num_object_clones":0,"num_object_copies":304576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76144,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125343,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3847,"num_bytes_recovered":29786112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125370,"ondisk_log_size":125370,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2774760,"kb_used_data":1178336,"kb_used_omap":14,"kb_used_meta":1596337,"kb_avail":374712600,"statfs":{"total":386547056640,"available":383705702400,"internally_reserved":0,"allocated":1206616064,"data_stored":1205896188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1634649649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,7,6,5,10,15,2,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21422080,"num_objects":-2615,"num_object_clones":0,"num_object_copies":-10460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2615,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002937"},"pg_stats":[{"pgid":"1.f","version":"33'7834","reported_seq":7917,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.860902+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:35.860902+0000","last_peered":"2024-09-16T04:46:35.860902+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:35.860902+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7834,"log_dups_size":0,"ondisk_log_size":7834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39485440,"num_objects":4820,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7834,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7857","reported_seq":7932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.796457+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:35.796457+0000","last_peered":"2024-09-16T04:46:35.796457+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:35.796457+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7857,"log_dups_size":0,"ondisk_log_size":7857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39458060,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7855,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2610}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7889","reported_seq":7974,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:40.548138+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:40.548138+0000","last_peered":"2024-09-16T04:46:40.548138+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:40.548138+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7889,"log_dups_size":0,"ondisk_log_size":7889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39164574,"num_objects":4782,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4782,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7888,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2559}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7851","reported_seq":7924,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.783251+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:35.783251+0000","last_peered":"2024-09-16T04:46:35.783251+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:35.783251+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7851,"log_dups_size":0,"ondisk_log_size":7851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39474444,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7849,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7935","reported_seq":8038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.148784+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:36.148784+0000","last_peered":"2024-09-16T04:46:36.148784+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:36.148784+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39367948,"num_objects":4806,"num_object_clones":0,"num_object_copies":19224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2546}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7667","reported_seq":7764,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.961930+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:35.961930+0000","last_peered":"2024-09-16T04:46:35.961930+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:35.961930+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7667,"log_dups_size":0,"ondisk_log_size":7667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38051840,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7667,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7801","reported_seq":7871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.795057+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:35.795057+0000","last_peered":"2024-09-16T04:46:35.795057+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:35.795057+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7801,"log_dups_size":0,"ondisk_log_size":7801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39310604,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7799,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2575}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7981","reported_seq":12538,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:37.754398+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:37.754398+0000","last_peered":"2024-09-16T04:46:37.754398+0000","last_clean":"2024-09-16T04:46:37.754398+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:37.754398+0000","last_undegraded":"2024-09-16T04:46:37.754398+0000","last_fullsized":"2024-09-16T04:46:37.754398+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7981,"log_dups_size":0,"ondisk_log_size":7981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:42.341393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39351564,"num_objects":4804,"num_object_clones":0,"num_object_copies":19216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7979,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8033","reported_seq":10912,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.125669+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:36.125669+0000","last_peered":"2024-09-16T04:46:36.125669+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:36.125669+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8033,"log_dups_size":0,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38685208,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8030,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1413,"num_bytes_recovered":11452416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3790}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7962","reported_seq":8060,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:40.560259+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:40.560259+0000","last_peered":"2024-09-16T04:46:40.560259+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:40.560259+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7962,"log_dups_size":0,"ondisk_log_size":7962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39286028,"num_objects":4796,"num_object_clones":0,"num_object_copies":19184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7960,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2605}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7777","reported_seq":7883,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.205805+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:36.205805+0000","last_peered":"2024-09-16T04:46:36.205805+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:36.205805+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38612882,"num_objects":4714,"num_object_clones":0,"num_object_copies":18856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7602","reported_seq":7672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.189827+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:36.189827+0000","last_peered":"2024-09-16T04:46:36.189827+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:36.189827+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7602,"log_dups_size":0,"ondisk_log_size":7602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38738566,"num_objects":4729,"num_object_clones":0,"num_object_copies":18916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7601,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2616}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7849","reported_seq":7930,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:40.540708+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:40.540708+0000","last_peered":"2024-09-16T04:46:40.540708+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:40.540708+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7849,"log_dups_size":0,"ondisk_log_size":7849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38369926,"num_objects":4684,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7848,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2462}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7903","reported_seq":8012,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:36.077324+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:36.077324+0000","last_peered":"2024-09-16T04:46:36.077324+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:36.077324+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7903,"log_dups_size":0,"ondisk_log_size":7903,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39965964,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7901,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2673}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7771","reported_seq":7861,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:40.643729+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:40.643729+0000","last_peered":"2024-09-16T04:46:40.643729+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:40.643729+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7771,"log_dups_size":0,"ondisk_log_size":7771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38319372,"num_objects":4678,"num_object_clones":0,"num_object_copies":18712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7769,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2551}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7658","reported_seq":7734,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:35.900037+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:35.900037+0000","last_peered":"2024-09-16T04:46:35.900037+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:35.900037+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7658,"log_dups_size":0,"ondisk_log_size":7658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38081804,"num_objects":4649,"num_object_clones":0,"num_object_copies":18596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7656,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":623724224,"num_objects":76144,"num_object_clones":0,"num_object_copies":304576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76144,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125343,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3847,"num_bytes_recovered":29786112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1204633600,"data_stored":1204633600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125370,"ondisk_log_size":125370,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734644,"kb_used_data":305892,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93637196,"statfs":{"total":96636764160,"available":95884488704,"internally_reserved":0,"allocated":313233408,"data_stored":313053439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":439021263},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725964,"kb_used_data":301880,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93645876,"statfs":{"total":96636764160,"available":95893377024,"internally_reserved":0,"allocated":309125120,"data_stored":308945151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051647,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574720,"kb_used_data":263984,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93797120,"statfs":{"total":96636764160,"available":96048250880,"internally_reserved":0,"allocated":270319616,"data_stored":270139647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318172879},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739432,"kb_used_data":306580,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93632408,"statfs":{"total":96636764160,"available":95879585792,"internally_reserved":0,"allocated":313937920,"data_stored":313757951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":443215566},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,4,3,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313159680,"data_stored":313159680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270020608,"data_stored":270020608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308543488,"data_stored":308543488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":312909824,"data_stored":312909824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:45.590 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:46.221 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:46.224 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:46.256 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-16T04:46:44.668014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":616195776,"num_objects":75225,"num_object_clones":0,"num_object_copies":300900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31683,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75225,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126262,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3899,"num_bytes_recovered":30146560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126289,"ondisk_log_size":126289,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2761540,"kb_used_data":1169256,"kb_used_omap":14,"kb_used_meta":1592177,"kb_avail":374725820,"statfs":{"total":386547056640,"available":383719239680,"internally_reserved":0,"allocated":1197318144,"data_stored":1196598268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1630389809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,2,6,5,11,3,9,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19914752,"num_objects":-2431,"num_object_clones":0,"num_object_copies":-9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003239"},"pg_stats":[{"pgid":"1.f","version":"33'7915","reported_seq":7998,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:41.200177+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:41.200177+0000","last_peered":"2024-09-16T04:46:41.200177+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:41.200177+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7915,"log_dups_size":0,"ondisk_log_size":7915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38821888,"num_objects":4739,"num_object_clones":0,"num_object_copies":18956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7915,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2539}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7920","reported_seq":7999,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:40.977058+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:40.977058+0000","last_peered":"2024-09-16T04:46:40.977058+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:40.977058+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7920,"log_dups_size":0,"ondisk_log_size":7920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38941964,"num_objects":4754,"num_object_clones":0,"num_object_copies":19016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7918,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2549}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7889","reported_seq":7974,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:40.548138+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:40.548138+0000","last_peered":"2024-09-16T04:46:40.548138+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:40.548138+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7889,"log_dups_size":0,"ondisk_log_size":7889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39164574,"num_objects":4782,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4782,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7888,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2559}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7925","reported_seq":8001,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:40.994605+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:40.994605+0000","last_peered":"2024-09-16T04:46:40.994605+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:40.994605+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7925,"log_dups_size":0,"ondisk_log_size":7925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38868236,"num_objects":4745,"num_object_clones":0,"num_object_copies":18980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7923,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8010","reported_seq":8113,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:41.055462+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:41.055462+0000","last_peered":"2024-09-16T04:46:41.055462+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:41.055462+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8010,"log_dups_size":0,"ondisk_log_size":8010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38753548,"num_objects":4731,"num_object_clones":0,"num_object_copies":18924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8008,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2471}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7736","reported_seq":7833,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:41.094114+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:41.094114+0000","last_peered":"2024-09-16T04:46:41.094114+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:41.094114+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7736,"log_dups_size":0,"ondisk_log_size":7736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37486592,"num_objects":4576,"num_object_clones":0,"num_object_copies":18304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7736,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7877","reported_seq":7949,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:40.924561+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:40.924561+0000","last_peered":"2024-09-16T04:46:40.924561+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:40.924561+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7877,"log_dups_size":0,"ondisk_log_size":7877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38688012,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7875,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2500}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8059","reported_seq":12616,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:42.860751+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:42.860751+0000","last_peered":"2024-09-16T04:46:42.860751+0000","last_clean":"2024-09-16T04:46:42.860751+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:42.860751+0000","last_undegraded":"2024-09-16T04:46:42.860751+0000","last_fullsized":"2024-09-16T04:46:42.860751+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8059,"log_dups_size":0,"ondisk_log_size":8059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:47.406648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38712588,"num_objects":4726,"num_object_clones":0,"num_object_copies":18904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8057,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8120","reported_seq":11086,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:41.204343+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:41.204343+0000","last_peered":"2024-09-16T04:46:41.204343+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:41.204343+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8120,"log_dups_size":0,"ondisk_log_size":8120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37972504,"num_objects":4636,"num_object_clones":0,"num_object_copies":18544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8117,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1457,"num_bytes_recovered":11812864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7962","reported_seq":8060,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:40.560259+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:40.560259+0000","last_peered":"2024-09-16T04:46:40.560259+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:40.560259+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7962,"log_dups_size":0,"ondisk_log_size":7962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39286028,"num_objects":4796,"num_object_clones":0,"num_object_copies":19184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7960,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2605}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7851","reported_seq":7955,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:41.032348+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:41.032348+0000","last_peered":"2024-09-16T04:46:41.032348+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:41.032348+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7851,"log_dups_size":0,"ondisk_log_size":7851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38006674,"num_objects":4640,"num_object_clones":0,"num_object_copies":18560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7848,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2475}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7702","reported_seq":7773,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:41.134379+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:41.134379+0000","last_peered":"2024-09-16T04:46:41.134379+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:41.134379+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7702,"log_dups_size":0,"ondisk_log_size":7702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37919366,"num_objects":4629,"num_object_clones":0,"num_object_copies":18516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7701,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2516}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7849","reported_seq":7930,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:40.540708+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:40.540708+0000","last_peered":"2024-09-16T04:46:40.540708+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:40.540708+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7849,"log_dups_size":0,"ondisk_log_size":7849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38369926,"num_objects":4684,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7848,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2462}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7984","reported_seq":8095,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:41.140387+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:41.140387+0000","last_peered":"2024-09-16T04:46:41.140387+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:41.140387+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7984,"log_dups_size":0,"ondisk_log_size":7984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39302412,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7982,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2592}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7771","reported_seq":7861,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:40.643729+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:40.643729+0000","last_peered":"2024-09-16T04:46:40.643729+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:40.643729+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7771,"log_dups_size":0,"ondisk_log_size":7771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38319372,"num_objects":4678,"num_object_clones":0,"num_object_copies":18712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7769,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2551}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7719","reported_seq":7802,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:41.033379+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:41.033379+0000","last_peered":"2024-09-16T04:46:41.033379+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:41.033379+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37582092,"num_objects":4588,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2357}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":616195776,"num_objects":75225,"num_object_clones":0,"num_object_copies":300900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31683,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75225,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126262,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3899,"num_bytes_recovered":30146560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1194471424,"data_stored":1194471424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126289,"ondisk_log_size":126289,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734860,"kb_used_data":301924,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93636980,"statfs":{"total":96636764160,"available":95884267520,"internally_reserved":0,"allocated":309170176,"data_stored":308990207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443281103},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,2,5,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725964,"kb_used_data":301880,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93645876,"statfs":{"total":96636764160,"available":95893377024,"internally_reserved":0,"allocated":309125120,"data_stored":308945151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562000,"kb_used_data":263744,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93809840,"statfs":{"total":96636764160,"available":96061276160,"internally_reserved":0,"allocated":270073856,"data_stored":269893887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305393359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738716,"kb_used_data":301708,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93633124,"statfs":{"total":96636764160,"available":95880318976,"internally_reserved":0,"allocated":308948992,"data_stored":308769023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":447475406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,2,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":308166656,"data_stored":308166656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269762560,"data_stored":269762560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308543488,"data_stored":308543488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":307998720,"data_stored":307998720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:46.258 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:46.861 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:46.861 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:46.894 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-16T04:46:46.668697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":606987968,"num_objects":74101,"num_object_clones":0,"num_object_copies":296404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31629,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127386,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3953,"num_bytes_recovered":30515200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127413,"ondisk_log_size":127413,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2759012,"kb_used_data":1154256,"kb_used_omap":14,"kb_used_meta":1604657,"kb_avail":374728348,"statfs":{"total":386547056640,"available":383721828352,"internally_reserved":0,"allocated":1181958144,"data_stored":1181238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1643169329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,13,4,10,5,12,1,0,0,0,1,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28434432,"num_objects":-3471,"num_object_clones":0,"num_object_copies":-13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003410"},"pg_stats":[{"pgid":"1.f","version":"33'7998","reported_seq":8083,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.290238+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:46.290238+0000","last_peered":"2024-09-16T04:46:46.290238+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:46.290238+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7998,"log_dups_size":0,"ondisk_log_size":7998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38141952,"num_objects":4656,"num_object_clones":0,"num_object_copies":18624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2457}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8013","reported_seq":8088,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.027312+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:46.027312+0000","last_peered":"2024-09-16T04:46:46.027312+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:46.027312+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38180108,"num_objects":4661,"num_object_clones":0,"num_object_copies":18644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8011,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2456}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7968","reported_seq":8053,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.603499+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:45.603499+0000","last_peered":"2024-09-16T04:46:45.603499+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:45.603499+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7968,"log_dups_size":0,"ondisk_log_size":7968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38517406,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7967,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2480}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7987","reported_seq":8065,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.082154+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:46.082154+0000","last_peered":"2024-09-16T04:46:46.082154+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:46.082154+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7987,"log_dups_size":0,"ondisk_log_size":7987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38360332,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7985,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2459}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8078","reported_seq":8187,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.831374+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:45.831374+0000","last_peered":"2024-09-16T04:46:45.831374+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:45.831374+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8078,"log_dups_size":0,"ondisk_log_size":8078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38196492,"num_objects":4663,"num_object_clones":0,"num_object_copies":18652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8076,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7800","reported_seq":7897,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.211358+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:46.211358+0000","last_peered":"2024-09-16T04:46:46.211358+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:46.211358+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7800,"log_dups_size":0,"ondisk_log_size":7800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36962304,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7941","reported_seq":8013,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.961969+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:45.961969+0000","last_peered":"2024-09-16T04:46:45.961969+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:45.961969+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7941,"log_dups_size":0,"ondisk_log_size":7941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38163724,"num_objects":4659,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7939,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2436}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8059","reported_seq":12616,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:42.860751+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:42.860751+0000","last_peered":"2024-09-16T04:46:42.860751+0000","last_clean":"2024-09-16T04:46:42.860751+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:42.860751+0000","last_undegraded":"2024-09-16T04:46:42.860751+0000","last_fullsized":"2024-09-16T04:46:42.860751+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8059,"log_dups_size":0,"ondisk_log_size":8059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:47.406648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38712588,"num_objects":4726,"num_object_clones":0,"num_object_copies":18904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8057,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8200","reported_seq":11257,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.292332+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:46.292332+0000","last_peered":"2024-09-16T04:46:46.292332+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:46.292332+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8200,"log_dups_size":0,"ondisk_log_size":8200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37317144,"num_objects":4556,"num_object_clones":0,"num_object_copies":18224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8197,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":12181504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3712}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8033","reported_seq":8131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.499894+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:45.499894+0000","last_peered":"2024-09-16T04:46:45.499894+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:45.499894+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8033,"log_dups_size":0,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38704396,"num_objects":4725,"num_object_clones":0,"num_object_copies":18900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8031,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2534}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7929","reported_seq":8038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.177704+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:46.177704+0000","last_peered":"2024-09-16T04:46:46.177704+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:46.177704+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7929,"log_dups_size":0,"ondisk_log_size":7929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37367698,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7926,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7789","reported_seq":7862,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.055662+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:46.055662+0000","last_peered":"2024-09-16T04:46:46.055662+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:46.055662+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7789,"log_dups_size":0,"ondisk_log_size":7789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37206662,"num_objects":4542,"num_object_clones":0,"num_object_copies":18168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7788,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2430}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7936","reported_seq":8017,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.543343+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:45.543343+0000","last_peered":"2024-09-16T04:46:45.543343+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:45.543343+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7936,"log_dups_size":0,"ondisk_log_size":7936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37657222,"num_objects":4597,"num_object_clones":0,"num_object_copies":18388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2375}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8060","reported_seq":8175,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.244686+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:46.244686+0000","last_peered":"2024-09-16T04:46:46.244686+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:46.244686+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8060,"log_dups_size":0,"ondisk_log_size":8060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38679820,"num_objects":4722,"num_object_clones":0,"num_object_copies":18888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8058,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2518}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7824","reported_seq":7914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.571212+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:45.571212+0000","last_peered":"2024-09-16T04:46:45.571212+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:45.571212+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7824,"log_dups_size":0,"ondisk_log_size":7824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37885196,"num_objects":4625,"num_object_clones":0,"num_object_copies":18500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7822,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2498}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7798","reported_seq":7882,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.096148+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:46.096148+0000","last_peered":"2024-09-16T04:46:46.096148+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:46.096148+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7798,"log_dups_size":0,"ondisk_log_size":7798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36934924,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7796,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2279}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":606987968,"num_objects":74101,"num_object_clones":0,"num_object_copies":296404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31629,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127386,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3953,"num_bytes_recovered":30515200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1179684864,"data_stored":1179684864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127413,"ondisk_log_size":127413,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733336,"kb_used_data":296260,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93638504,"statfs":{"total":96636764160,"available":95885828096,"internally_reserved":0,"allocated":303370240,"data_stored":303190271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447540943},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,5,0,2,1,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725420,"kb_used_data":297168,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93646420,"statfs":{"total":96636764160,"available":95893934080,"internally_reserved":0,"allocated":304300032,"data_stored":304120063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,5,2,5,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562000,"kb_used_data":263744,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93809840,"statfs":{"total":96636764160,"available":96061276160,"internally_reserved":0,"allocated":270073856,"data_stored":269893887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305393359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247887,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738256,"kb_used_data":297084,"kb_used_omap":4,"kb_used_meta":441147,"kb_avail":93633584,"statfs":{"total":96636764160,"available":95880790016,"internally_reserved":0,"allocated":304214016,"data_stored":304034047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":451735246},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,1,0,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":303218688,"data_stored":303218688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269762560,"data_stored":269762560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303644672,"data_stored":303644672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":303058944,"data_stored":303058944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:46.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:46:46.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7998", 'reported_seq': 8083, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.290238+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:46:46.290238+0000', 'last_peered': '2024-09-16T04:46:46.290238+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:46:46.290238+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7998, 'log_dups_size': 0, 'ondisk_log_size': 7998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38141952, 'num_objects': 4656, 'num_object_clones': 0, 'num_object_copies': 18624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2199, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7998, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2457}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:46.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:46:46.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8013", 'reported_seq': 8088, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.027312+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:46:46.027312+0000', 'last_peered': '2024-09-16T04:46:46.027312+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:46:46.027312+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8013, 'log_dups_size': 0, 'ondisk_log_size': 8013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38180108, 'num_objects': 4661, 'num_object_clones': 0, 'num_object_copies': 18644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2207, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4661, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8011, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2456}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:46.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:46:46.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7968", 'reported_seq': 8053, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:45.603499+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:46:45.603499+0000', 'last_peered': '2024-09-16T04:46:45.603499+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:46:45.603499+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7968, 'log_dups_size': 0, 'ondisk_log_size': 7968, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38517406, 'num_objects': 4703, 'num_object_clones': 0, 'num_object_copies': 18812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4703, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7967, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2480}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:46.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:46:46.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7987", 'reported_seq': 8065, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.082154+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:46:46.082154+0000', 'last_peered': '2024-09-16T04:46:46.082154+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:46:46.082154+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7987, 'log_dups_size': 0, 'ondisk_log_size': 7987, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38360332, 'num_objects': 4683, 'num_object_clones': 0, 'num_object_copies': 18732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4683, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7985, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2459}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:46.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:46:46.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8078", 'reported_seq': 8187, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:45.831374+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:46:45.831374+0000', 'last_peered': '2024-09-16T04:46:45.831374+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:46:45.831374+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8078, 'log_dups_size': 0, 'ondisk_log_size': 8078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38196492, 'num_objects': 4663, 'num_object_clones': 0, 'num_object_copies': 18652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8076, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2406}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:46.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:46:46.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7800", 'reported_seq': 7897, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.211358+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:46:46.211358+0000', 'last_peered': '2024-09-16T04:46:46.211358+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:46:46.211358+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7800, 'log_dups_size': 0, 'ondisk_log_size': 7800, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36962304, 'num_objects': 4512, 'num_object_clones': 0, 'num_object_copies': 18048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2182, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7800, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2330}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:46.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:46:46.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7941", 'reported_seq': 8013, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:45.961969+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:46:45.961969+0000', 'last_peered': '2024-09-16T04:46:45.961969+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:46:45.961969+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7941, 'log_dups_size': 0, 'ondisk_log_size': 7941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38163724, 'num_objects': 4659, 'num_object_clones': 0, 'num_object_copies': 18636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2225, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4659, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7939, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2436}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:46.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:46:46.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8200", 'reported_seq': 11257, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.292332+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:46:46.292332+0000', 'last_peered': '2024-09-16T04:46:46.292332+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:46:46.292332+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8200, 'log_dups_size': 0, 'ondisk_log_size': 8200, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37317144, 'num_objects': 4556, 'num_object_clones': 0, 'num_object_copies': 18224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 844, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4556, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8197, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1502, 'num_bytes_recovered': 12181504, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3712}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:46.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:46:46.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8033", 'reported_seq': 8131, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:45.499894+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:46:45.499894+0000', 'last_peered': '2024-09-16T04:46:45.499894+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:46:45.499894+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8033, 'log_dups_size': 0, 'ondisk_log_size': 8033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38704396, 'num_objects': 4725, 'num_object_clones': 0, 'num_object_copies': 18900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2193, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8031, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2534}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:46.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:46:46.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7929", 'reported_seq': 8038, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.177704+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:46:46.177704+0000', 'last_peered': '2024-09-16T04:46:46.177704+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:46:46.177704+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7929, 'log_dups_size': 0, 'ondisk_log_size': 7929, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37367698, 'num_objects': 4562, 'num_object_clones': 0, 'num_object_copies': 18248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2167, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4562, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7926, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2398}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:46.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:46:46.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7789", 'reported_seq': 7862, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.055662+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:46:46.055662+0000', 'last_peered': '2024-09-16T04:46:46.055662+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:46:46.055662+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7789, 'log_dups_size': 0, 'ondisk_log_size': 7789, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37206662, 'num_objects': 4542, 'num_object_clones': 0, 'num_object_copies': 18168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2113, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4542, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7788, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2430}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:46.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:46:46.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7936", 'reported_seq': 8017, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:45.543343+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:46:45.543343+0000', 'last_peered': '2024-09-16T04:46:45.543343+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:46:45.543343+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7936, 'log_dups_size': 0, 'ondisk_log_size': 7936, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37657222, 'num_objects': 4597, 'num_object_clones': 0, 'num_object_copies': 18388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7935, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2375}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:46.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:46:46.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8060", 'reported_seq': 8175, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.244686+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:46:46.244686+0000', 'last_peered': '2024-09-16T04:46:46.244686+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:46:46.244686+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8060, 'log_dups_size': 0, 'ondisk_log_size': 8060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38679820, 'num_objects': 4722, 'num_object_clones': 0, 'num_object_copies': 18888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2206, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8058, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2518}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:46.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:46:46.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7824", 'reported_seq': 7914, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:45.571212+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:46:45.571212+0000', 'last_peered': '2024-09-16T04:46:45.571212+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:46:45.571212+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7824, 'log_dups_size': 0, 'ondisk_log_size': 7824, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37885196, 'num_objects': 4625, 'num_object_clones': 0, 'num_object_copies': 18500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4625, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7822, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2498}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:46.899 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:46:46.899 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7798", 'reported_seq': 7882, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.096148+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:46:46.096148+0000', 'last_peered': '2024-09-16T04:46:46.096148+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:46:46.096148+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7798, 'log_dups_size': 0, 'ondisk_log_size': 7798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36934924, 'num_objects': 4509, 'num_object_clones': 0, 'num_object_copies': 18036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2232, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7796, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2279}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:46.900 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:46:47.662 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:47.662 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31629/296404 objects degraded (10.671%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":419,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":74101,"data_bytes":606987968,"bytes_used":2825228288,"bytes_avail":383721828352,"bytes_total":386547056640,"degraded_objects":31629,"degraded_total":296404,"degraded_ratio":0.10670908624714916,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":101005,"recovering_keys_per_sec":0,"num_objects_recovered":181,"num_bytes_recovered":1212416,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":289},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:46:47.693 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:46:47.693 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:48.347 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:48.348 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:48.373 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-16T04:46:46.668697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":606987968,"num_objects":74101,"num_object_clones":0,"num_object_copies":296404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31629,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127386,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3953,"num_bytes_recovered":30515200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127413,"ondisk_log_size":127413,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2759012,"kb_used_data":1154256,"kb_used_omap":14,"kb_used_meta":1604657,"kb_avail":374728348,"statfs":{"total":386547056640,"available":383721828352,"internally_reserved":0,"allocated":1181958144,"data_stored":1181238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1643169329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,13,4,10,5,12,1,0,0,0,1,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28434432,"num_objects":-3471,"num_object_clones":0,"num_object_copies":-13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003410"},"pg_stats":[{"pgid":"1.f","version":"33'7998","reported_seq":8083,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.290238+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:46.290238+0000","last_peered":"2024-09-16T04:46:46.290238+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:46.290238+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7998,"log_dups_size":0,"ondisk_log_size":7998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38141952,"num_objects":4656,"num_object_clones":0,"num_object_copies":18624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2457}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8013","reported_seq":8088,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.027312+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:46.027312+0000","last_peered":"2024-09-16T04:46:46.027312+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:46.027312+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38180108,"num_objects":4661,"num_object_clones":0,"num_object_copies":18644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8011,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2456}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7968","reported_seq":8053,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.603499+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:45.603499+0000","last_peered":"2024-09-16T04:46:45.603499+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:45.603499+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7968,"log_dups_size":0,"ondisk_log_size":7968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38517406,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7967,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2480}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7987","reported_seq":8065,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.082154+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:46.082154+0000","last_peered":"2024-09-16T04:46:46.082154+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:46.082154+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7987,"log_dups_size":0,"ondisk_log_size":7987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38360332,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7985,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2459}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8078","reported_seq":8187,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.831374+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:45.831374+0000","last_peered":"2024-09-16T04:46:45.831374+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:45.831374+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8078,"log_dups_size":0,"ondisk_log_size":8078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38196492,"num_objects":4663,"num_object_clones":0,"num_object_copies":18652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8076,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7800","reported_seq":7897,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.211358+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:46.211358+0000","last_peered":"2024-09-16T04:46:46.211358+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:46.211358+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7800,"log_dups_size":0,"ondisk_log_size":7800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36962304,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7941","reported_seq":8013,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.961969+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:45.961969+0000","last_peered":"2024-09-16T04:46:45.961969+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:45.961969+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7941,"log_dups_size":0,"ondisk_log_size":7941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38163724,"num_objects":4659,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7939,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2436}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8059","reported_seq":12616,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:42.860751+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:42.860751+0000","last_peered":"2024-09-16T04:46:42.860751+0000","last_clean":"2024-09-16T04:46:42.860751+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:42.860751+0000","last_undegraded":"2024-09-16T04:46:42.860751+0000","last_fullsized":"2024-09-16T04:46:42.860751+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8059,"log_dups_size":0,"ondisk_log_size":8059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:47.406648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38712588,"num_objects":4726,"num_object_clones":0,"num_object_copies":18904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8057,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8200","reported_seq":11257,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.292332+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:46.292332+0000","last_peered":"2024-09-16T04:46:46.292332+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:46.292332+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8200,"log_dups_size":0,"ondisk_log_size":8200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37317144,"num_objects":4556,"num_object_clones":0,"num_object_copies":18224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8197,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":12181504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3712}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8033","reported_seq":8131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.499894+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:45.499894+0000","last_peered":"2024-09-16T04:46:45.499894+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:45.499894+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8033,"log_dups_size":0,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38704396,"num_objects":4725,"num_object_clones":0,"num_object_copies":18900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8031,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2534}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7929","reported_seq":8038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.177704+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:46.177704+0000","last_peered":"2024-09-16T04:46:46.177704+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:46.177704+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7929,"log_dups_size":0,"ondisk_log_size":7929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37367698,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7926,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7789","reported_seq":7862,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.055662+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:46.055662+0000","last_peered":"2024-09-16T04:46:46.055662+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:46.055662+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7789,"log_dups_size":0,"ondisk_log_size":7789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37206662,"num_objects":4542,"num_object_clones":0,"num_object_copies":18168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7788,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2430}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7936","reported_seq":8017,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.543343+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:45.543343+0000","last_peered":"2024-09-16T04:46:45.543343+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:45.543343+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7936,"log_dups_size":0,"ondisk_log_size":7936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37657222,"num_objects":4597,"num_object_clones":0,"num_object_copies":18388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2375}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8060","reported_seq":8175,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.244686+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:46.244686+0000","last_peered":"2024-09-16T04:46:46.244686+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:46.244686+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8060,"log_dups_size":0,"ondisk_log_size":8060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38679820,"num_objects":4722,"num_object_clones":0,"num_object_copies":18888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8058,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2518}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7824","reported_seq":7914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.571212+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:45.571212+0000","last_peered":"2024-09-16T04:46:45.571212+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:45.571212+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7824,"log_dups_size":0,"ondisk_log_size":7824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37885196,"num_objects":4625,"num_object_clones":0,"num_object_copies":18500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7822,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2498}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7798","reported_seq":7882,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.096148+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:46.096148+0000","last_peered":"2024-09-16T04:46:46.096148+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:46.096148+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7798,"log_dups_size":0,"ondisk_log_size":7798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36934924,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7796,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2279}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":606987968,"num_objects":74101,"num_object_clones":0,"num_object_copies":296404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31629,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127386,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3953,"num_bytes_recovered":30515200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1179684864,"data_stored":1179684864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127413,"ondisk_log_size":127413,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733336,"kb_used_data":296260,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93638504,"statfs":{"total":96636764160,"available":95885828096,"internally_reserved":0,"allocated":303370240,"data_stored":303190271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447540943},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,5,0,2,1,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725420,"kb_used_data":297168,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93646420,"statfs":{"total":96636764160,"available":95893934080,"internally_reserved":0,"allocated":304300032,"data_stored":304120063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,5,2,5,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562000,"kb_used_data":263744,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93809840,"statfs":{"total":96636764160,"available":96061276160,"internally_reserved":0,"allocated":270073856,"data_stored":269893887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305393359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247887,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738256,"kb_used_data":297084,"kb_used_omap":4,"kb_used_meta":441147,"kb_avail":93633584,"statfs":{"total":96636764160,"available":95880790016,"internally_reserved":0,"allocated":304214016,"data_stored":304034047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":451735246},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,1,0,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":303218688,"data_stored":303218688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269762560,"data_stored":269762560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303644672,"data_stored":303644672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":303058944,"data_stored":303058944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:51.375 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:51.995 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:51.995 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:52.017 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-16T04:46:50.670441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":606430912,"num_objects":74033,"num_object_clones":0,"num_object_copies":296132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31629,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127454,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3953,"num_bytes_recovered":30515200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127481,"ondisk_log_size":127481,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767104,"kb_used_data":1154028,"kb_used_omap":14,"kb_used_meta":1612977,"kb_avail":374720256,"statfs":{"total":386547056640,"available":383713542144,"internally_reserved":0,"allocated":1181724672,"data_stored":1181004796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1651689009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,8,3,9,9,14,1,0,0,0,1,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19603456,"num_objects":-2393,"num_object_clones":0,"num_object_copies":-9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2393,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":729088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004108"},"pg_stats":[{"pgid":"1.f","version":"33'7998","reported_seq":8083,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.290238+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:46.290238+0000","last_peered":"2024-09-16T04:46:46.290238+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:46.290238+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7998,"log_dups_size":0,"ondisk_log_size":7998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38141952,"num_objects":4656,"num_object_clones":0,"num_object_copies":18624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2457}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8013","reported_seq":8088,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.027312+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:46.027312+0000","last_peered":"2024-09-16T04:46:46.027312+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:46.027312+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38180108,"num_objects":4661,"num_object_clones":0,"num_object_copies":18644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8011,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2456}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7968","reported_seq":8053,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.603499+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:45.603499+0000","last_peered":"2024-09-16T04:46:45.603499+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:45.603499+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7968,"log_dups_size":0,"ondisk_log_size":7968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38517406,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7967,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2480}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7987","reported_seq":8065,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.082154+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:46.082154+0000","last_peered":"2024-09-16T04:46:46.082154+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:46.082154+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7987,"log_dups_size":0,"ondisk_log_size":7987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38360332,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7985,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2459}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8078","reported_seq":8187,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.831374+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:45.831374+0000","last_peered":"2024-09-16T04:46:45.831374+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:45.831374+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8078,"log_dups_size":0,"ondisk_log_size":8078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38196492,"num_objects":4663,"num_object_clones":0,"num_object_copies":18652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8076,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7800","reported_seq":7897,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.211358+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:46.211358+0000","last_peered":"2024-09-16T04:46:46.211358+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:46.211358+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7800,"log_dups_size":0,"ondisk_log_size":7800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36962304,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7941","reported_seq":8013,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.961969+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:45.961969+0000","last_peered":"2024-09-16T04:46:45.961969+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:45.961969+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7941,"log_dups_size":0,"ondisk_log_size":7941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38163724,"num_objects":4659,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7939,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2436}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8127","reported_seq":12684,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:47.852940+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:47.852940+0000","last_peered":"2024-09-16T04:46:47.852940+0000","last_clean":"2024-09-16T04:46:47.852940+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:47.852940+0000","last_undegraded":"2024-09-16T04:46:47.852940+0000","last_fullsized":"2024-09-16T04:46:47.852940+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:52.409049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38155532,"num_objects":4658,"num_object_clones":0,"num_object_copies":18632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8125,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8200","reported_seq":11257,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.292332+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:46.292332+0000","last_peered":"2024-09-16T04:46:46.292332+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:46.292332+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8200,"log_dups_size":0,"ondisk_log_size":8200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37317144,"num_objects":4556,"num_object_clones":0,"num_object_copies":18224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8197,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":12181504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3712}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8033","reported_seq":8131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.499894+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:45.499894+0000","last_peered":"2024-09-16T04:46:45.499894+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:45.499894+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8033,"log_dups_size":0,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38704396,"num_objects":4725,"num_object_clones":0,"num_object_copies":18900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8031,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2534}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7929","reported_seq":8038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.177704+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:46.177704+0000","last_peered":"2024-09-16T04:46:46.177704+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:46.177704+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7929,"log_dups_size":0,"ondisk_log_size":7929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37367698,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7926,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7789","reported_seq":7862,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.055662+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:46.055662+0000","last_peered":"2024-09-16T04:46:46.055662+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:46.055662+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7789,"log_dups_size":0,"ondisk_log_size":7789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37206662,"num_objects":4542,"num_object_clones":0,"num_object_copies":18168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7788,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2430}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7936","reported_seq":8017,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.543343+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:45.543343+0000","last_peered":"2024-09-16T04:46:45.543343+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:45.543343+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7936,"log_dups_size":0,"ondisk_log_size":7936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37657222,"num_objects":4597,"num_object_clones":0,"num_object_copies":18388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2375}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8060","reported_seq":8175,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.244686+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:46.244686+0000","last_peered":"2024-09-16T04:46:46.244686+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:46.244686+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8060,"log_dups_size":0,"ondisk_log_size":8060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38679820,"num_objects":4722,"num_object_clones":0,"num_object_copies":18888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8058,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2518}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7824","reported_seq":7914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.571212+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:45.571212+0000","last_peered":"2024-09-16T04:46:45.571212+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:45.571212+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7824,"log_dups_size":0,"ondisk_log_size":7824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37885196,"num_objects":4625,"num_object_clones":0,"num_object_copies":18500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7822,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2498}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7798","reported_seq":7882,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.096148+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:46.096148+0000","last_peered":"2024-09-16T04:46:46.096148+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:46.096148+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7798,"log_dups_size":0,"ondisk_log_size":7798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36934924,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7796,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2279}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":606430912,"num_objects":74033,"num_object_clones":0,"num_object_copies":296132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31629,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127454,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3953,"num_bytes_recovered":30515200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1179443200,"data_stored":1179443200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127481,"ondisk_log_size":127481,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733336,"kb_used_data":296260,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93638504,"statfs":{"total":96636764160,"available":95885828096,"internally_reserved":0,"allocated":303370240,"data_stored":303190271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447540943},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,5,0,2,1,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725420,"kb_used_data":297168,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93646420,"statfs":{"total":96636764160,"available":95893934080,"internally_reserved":0,"allocated":304300032,"data_stored":304120063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,5,2,5,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051649,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570092,"kb_used_data":263516,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93801748,"statfs":{"total":96636764160,"available":96052989952,"internally_reserved":0,"allocated":269840384,"data_stored":269660415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313913039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247887,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738256,"kb_used_data":297084,"kb_used_omap":4,"kb_used_meta":441147,"kb_avail":93633584,"statfs":{"total":96636764160,"available":95880790016,"internally_reserved":0,"allocated":304214016,"data_stored":304034047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":451735246},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,1,0,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":303218688,"data_stored":303218688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269520896,"data_stored":269520896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303644672,"data_stored":303644672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":303058944,"data_stored":303058944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:52.020 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:52.624 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:52.624 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:52.653 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-16T04:46:50.670441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":606430912,"num_objects":74033,"num_object_clones":0,"num_object_copies":296132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31629,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127454,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3953,"num_bytes_recovered":30515200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127481,"ondisk_log_size":127481,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767104,"kb_used_data":1154028,"kb_used_omap":14,"kb_used_meta":1612977,"kb_avail":374720256,"statfs":{"total":386547056640,"available":383713542144,"internally_reserved":0,"allocated":1181724672,"data_stored":1181004796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1651689009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,8,3,9,9,14,1,0,0,0,1,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19603456,"num_objects":-2393,"num_object_clones":0,"num_object_copies":-9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2393,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":729088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004108"},"pg_stats":[{"pgid":"1.f","version":"33'7998","reported_seq":8083,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.290238+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:46.290238+0000","last_peered":"2024-09-16T04:46:46.290238+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:46.290238+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7998,"log_dups_size":0,"ondisk_log_size":7998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38141952,"num_objects":4656,"num_object_clones":0,"num_object_copies":18624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2457}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8013","reported_seq":8088,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.027312+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:46.027312+0000","last_peered":"2024-09-16T04:46:46.027312+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:46.027312+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38180108,"num_objects":4661,"num_object_clones":0,"num_object_copies":18644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8011,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2456}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7968","reported_seq":8053,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.603499+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:45.603499+0000","last_peered":"2024-09-16T04:46:45.603499+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:45.603499+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7968,"log_dups_size":0,"ondisk_log_size":7968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38517406,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7967,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2480}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7987","reported_seq":8065,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.082154+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:46.082154+0000","last_peered":"2024-09-16T04:46:46.082154+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:46.082154+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7987,"log_dups_size":0,"ondisk_log_size":7987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38360332,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7985,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2459}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8078","reported_seq":8187,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.831374+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:45.831374+0000","last_peered":"2024-09-16T04:46:45.831374+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:45.831374+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8078,"log_dups_size":0,"ondisk_log_size":8078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38196492,"num_objects":4663,"num_object_clones":0,"num_object_copies":18652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8076,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7800","reported_seq":7897,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.211358+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:46.211358+0000","last_peered":"2024-09-16T04:46:46.211358+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:46.211358+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7800,"log_dups_size":0,"ondisk_log_size":7800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36962304,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7941","reported_seq":8013,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.961969+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:45.961969+0000","last_peered":"2024-09-16T04:46:45.961969+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:45.961969+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7941,"log_dups_size":0,"ondisk_log_size":7941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38163724,"num_objects":4659,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7939,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2436}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8127","reported_seq":12684,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:47.852940+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:47.852940+0000","last_peered":"2024-09-16T04:46:47.852940+0000","last_clean":"2024-09-16T04:46:47.852940+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:47.852940+0000","last_undegraded":"2024-09-16T04:46:47.852940+0000","last_fullsized":"2024-09-16T04:46:47.852940+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:52.409049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38155532,"num_objects":4658,"num_object_clones":0,"num_object_copies":18632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8125,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8200","reported_seq":11257,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.292332+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:46.292332+0000","last_peered":"2024-09-16T04:46:46.292332+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:46.292332+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8200,"log_dups_size":0,"ondisk_log_size":8200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37317144,"num_objects":4556,"num_object_clones":0,"num_object_copies":18224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8197,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":12181504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3712}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8033","reported_seq":8131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.499894+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:45.499894+0000","last_peered":"2024-09-16T04:46:45.499894+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:45.499894+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8033,"log_dups_size":0,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38704396,"num_objects":4725,"num_object_clones":0,"num_object_copies":18900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8031,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2534}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7929","reported_seq":8038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.177704+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:46.177704+0000","last_peered":"2024-09-16T04:46:46.177704+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:46.177704+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7929,"log_dups_size":0,"ondisk_log_size":7929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37367698,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7926,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7789","reported_seq":7862,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.055662+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:46.055662+0000","last_peered":"2024-09-16T04:46:46.055662+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:46.055662+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7789,"log_dups_size":0,"ondisk_log_size":7789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37206662,"num_objects":4542,"num_object_clones":0,"num_object_copies":18168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7788,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2430}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7936","reported_seq":8017,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.543343+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:45.543343+0000","last_peered":"2024-09-16T04:46:45.543343+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:45.543343+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7936,"log_dups_size":0,"ondisk_log_size":7936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37657222,"num_objects":4597,"num_object_clones":0,"num_object_copies":18388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2375}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8060","reported_seq":8175,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.244686+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:46.244686+0000","last_peered":"2024-09-16T04:46:46.244686+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:46.244686+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8060,"log_dups_size":0,"ondisk_log_size":8060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38679820,"num_objects":4722,"num_object_clones":0,"num_object_copies":18888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8058,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2518}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7824","reported_seq":7914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:45.571212+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:45.571212+0000","last_peered":"2024-09-16T04:46:45.571212+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:45.571212+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7824,"log_dups_size":0,"ondisk_log_size":7824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37885196,"num_objects":4625,"num_object_clones":0,"num_object_copies":18500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7822,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2498}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7798","reported_seq":7882,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:46.096148+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:46.096148+0000","last_peered":"2024-09-16T04:46:46.096148+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:46.096148+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7798,"log_dups_size":0,"ondisk_log_size":7798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36934924,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7796,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2279}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":606430912,"num_objects":74033,"num_object_clones":0,"num_object_copies":296132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31629,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127454,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3953,"num_bytes_recovered":30515200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1179443200,"data_stored":1179443200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127481,"ondisk_log_size":127481,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733336,"kb_used_data":296260,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93638504,"statfs":{"total":96636764160,"available":95885828096,"internally_reserved":0,"allocated":303370240,"data_stored":303190271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447540943},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,5,0,2,1,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725420,"kb_used_data":297168,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93646420,"statfs":{"total":96636764160,"available":95893934080,"internally_reserved":0,"allocated":304300032,"data_stored":304120063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,5,2,5,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051649,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570092,"kb_used_data":263516,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93801748,"statfs":{"total":96636764160,"available":96052989952,"internally_reserved":0,"allocated":269840384,"data_stored":269660415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313913039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247887,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738256,"kb_used_data":297084,"kb_used_omap":4,"kb_used_meta":441147,"kb_avail":93633584,"statfs":{"total":96636764160,"available":95880790016,"internally_reserved":0,"allocated":304214016,"data_stored":304034047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":451735246},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,1,0,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":303218688,"data_stored":303218688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269520896,"data_stored":269520896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303644672,"data_stored":303644672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":303058944,"data_stored":303058944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:52.654 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:46:52.655 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7998", 'reported_seq': 8083, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.290238+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:46:46.290238+0000', 'last_peered': '2024-09-16T04:46:46.290238+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:46:46.290238+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7998, 'log_dups_size': 0, 'ondisk_log_size': 7998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38141952, 'num_objects': 4656, 'num_object_clones': 0, 'num_object_copies': 18624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2199, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7998, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2457}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:52.655 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:46:52.655 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8013", 'reported_seq': 8088, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.027312+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:46:46.027312+0000', 'last_peered': '2024-09-16T04:46:46.027312+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:46:46.027312+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8013, 'log_dups_size': 0, 'ondisk_log_size': 8013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38180108, 'num_objects': 4661, 'num_object_clones': 0, 'num_object_copies': 18644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2207, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4661, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8011, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2456}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:52.655 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:46:52.655 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7968", 'reported_seq': 8053, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:45.603499+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:46:45.603499+0000', 'last_peered': '2024-09-16T04:46:45.603499+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:46:45.603499+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7968, 'log_dups_size': 0, 'ondisk_log_size': 7968, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38517406, 'num_objects': 4703, 'num_object_clones': 0, 'num_object_copies': 18812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4703, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7967, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2480}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:52.655 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:46:52.655 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7987", 'reported_seq': 8065, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.082154+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:46:46.082154+0000', 'last_peered': '2024-09-16T04:46:46.082154+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:46:46.082154+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7987, 'log_dups_size': 0, 'ondisk_log_size': 7987, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38360332, 'num_objects': 4683, 'num_object_clones': 0, 'num_object_copies': 18732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4683, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7985, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2459}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:52.655 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:46:52.655 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8078", 'reported_seq': 8187, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:45.831374+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:46:45.831374+0000', 'last_peered': '2024-09-16T04:46:45.831374+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:46:45.831374+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8078, 'log_dups_size': 0, 'ondisk_log_size': 8078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38196492, 'num_objects': 4663, 'num_object_clones': 0, 'num_object_copies': 18652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8076, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2406}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:52.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:46:52.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7800", 'reported_seq': 7897, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.211358+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:46:46.211358+0000', 'last_peered': '2024-09-16T04:46:46.211358+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:46:46.211358+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7800, 'log_dups_size': 0, 'ondisk_log_size': 7800, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36962304, 'num_objects': 4512, 'num_object_clones': 0, 'num_object_copies': 18048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2182, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7800, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2330}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:52.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:46:52.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7941", 'reported_seq': 8013, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:45.961969+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:46:45.961969+0000', 'last_peered': '2024-09-16T04:46:45.961969+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:46:45.961969+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7941, 'log_dups_size': 0, 'ondisk_log_size': 7941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38163724, 'num_objects': 4659, 'num_object_clones': 0, 'num_object_copies': 18636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2225, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4659, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7939, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2436}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:52.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:46:52.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8200", 'reported_seq': 11257, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.292332+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:46:46.292332+0000', 'last_peered': '2024-09-16T04:46:46.292332+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:46:46.292332+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8200, 'log_dups_size': 0, 'ondisk_log_size': 8200, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37317144, 'num_objects': 4556, 'num_object_clones': 0, 'num_object_copies': 18224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 844, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4556, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8197, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1502, 'num_bytes_recovered': 12181504, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3712}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:52.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:46:52.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8033", 'reported_seq': 8131, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:45.499894+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:46:45.499894+0000', 'last_peered': '2024-09-16T04:46:45.499894+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:46:45.499894+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8033, 'log_dups_size': 0, 'ondisk_log_size': 8033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38704396, 'num_objects': 4725, 'num_object_clones': 0, 'num_object_copies': 18900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2193, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8031, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2534}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:52.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:46:52.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7929", 'reported_seq': 8038, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.177704+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:46:46.177704+0000', 'last_peered': '2024-09-16T04:46:46.177704+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:46:46.177704+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7929, 'log_dups_size': 0, 'ondisk_log_size': 7929, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37367698, 'num_objects': 4562, 'num_object_clones': 0, 'num_object_copies': 18248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2167, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4562, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7926, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2398}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:52.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:46:52.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7789", 'reported_seq': 7862, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.055662+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:46:46.055662+0000', 'last_peered': '2024-09-16T04:46:46.055662+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:46:46.055662+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7789, 'log_dups_size': 0, 'ondisk_log_size': 7789, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37206662, 'num_objects': 4542, 'num_object_clones': 0, 'num_object_copies': 18168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2113, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4542, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7788, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2430}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:52.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:46:52.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7936", 'reported_seq': 8017, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:45.543343+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:46:45.543343+0000', 'last_peered': '2024-09-16T04:46:45.543343+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:46:45.543343+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7936, 'log_dups_size': 0, 'ondisk_log_size': 7936, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37657222, 'num_objects': 4597, 'num_object_clones': 0, 'num_object_copies': 18388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7935, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2375}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:52.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:46:52.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8060", 'reported_seq': 8175, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.244686+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:46:46.244686+0000', 'last_peered': '2024-09-16T04:46:46.244686+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:46:46.244686+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8060, 'log_dups_size': 0, 'ondisk_log_size': 8060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38679820, 'num_objects': 4722, 'num_object_clones': 0, 'num_object_copies': 18888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2206, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8058, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2518}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:52.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:46:52.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7824", 'reported_seq': 7914, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:45.571212+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:46:45.571212+0000', 'last_peered': '2024-09-16T04:46:45.571212+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:46:45.571212+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7824, 'log_dups_size': 0, 'ondisk_log_size': 7824, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37885196, 'num_objects': 4625, 'num_object_clones': 0, 'num_object_copies': 18500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4625, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7822, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2498}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:52.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:46:52.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7798", 'reported_seq': 7882, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:46.096148+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:46:46.096148+0000', 'last_peered': '2024-09-16T04:46:46.096148+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:46:46.096148+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7798, 'log_dups_size': 0, 'ondisk_log_size': 7798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36934924, 'num_objects': 4509, 'num_object_clones': 0, 'num_object_copies': 18036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2232, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7796, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2279}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:52.658 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:46:53.422 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:53.422 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31568/291736 objects degraded (10.821%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":425,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":72934,"data_bytes":597427904,"bytes_used":2817445888,"bytes_avail":383729610752,"bytes_total":386547056640,"degraded_objects":31568,"degraded_total":291736,"degraded_ratio":0.10820742040749171,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":94826,"recovering_keys_per_sec":0,"num_objects_recovered":167,"num_bytes_recovered":1138688,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":267},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:46:53.442 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:46:53.442 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:54.048 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:54.050 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:54.084 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-09-16T04:46:52.674950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597427904,"num_objects":72934,"num_object_clones":0,"num_object_copies":291736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128553,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4014,"num_bytes_recovered":30924800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128580,"ondisk_log_size":128580,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2751412,"kb_used_data":1141108,"kb_used_omap":14,"kb_used_meta":1610225,"kb_avail":374735948,"statfs":{"total":386547056640,"available":383729610752,"internally_reserved":0,"allocated":1168494592,"data_stored":1167774716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1648870961},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,4,3,17,6,1,0,0,4,1,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26296320,"num_objects":-3210,"num_object_clones":0,"num_object_copies":-12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3210,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008173"},"pg_stats":[{"pgid":"1.f","version":"33'8067","reported_seq":8158,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:51.280136+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:51.280136+0000","last_peered":"2024-09-16T04:46:51.280136+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:51.280136+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8067,"log_dups_size":0,"ondisk_log_size":8067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37576704,"num_objects":4587,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8067,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2391}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8075","reported_seq":8154,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:50.990095+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:50.990095+0000","last_peered":"2024-09-16T04:46:50.990095+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:50.990095+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8075,"log_dups_size":0,"ondisk_log_size":8075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37672204,"num_objects":4599,"num_object_clones":0,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8073,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2394}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8038","reported_seq":8123,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:50.644156+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:50.644156+0000","last_peered":"2024-09-16T04:46:50.644156+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:50.644156+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8038,"log_dups_size":0,"ondisk_log_size":8038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37943966,"num_objects":4633,"num_object_clones":0,"num_object_copies":18532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8037,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2410}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8058","reported_seq":8136,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:51.115307+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:51.115307+0000","last_peered":"2024-09-16T04:46:51.115307+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:51.115307+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8058,"log_dups_size":0,"ondisk_log_size":8058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37778700,"num_objects":4612,"num_object_clones":0,"num_object_copies":18448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8056,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2388}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8147","reported_seq":8258,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:51.302264+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:51.302264+0000","last_peered":"2024-09-16T04:46:51.302264+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:51.302264+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8147,"log_dups_size":0,"ondisk_log_size":8147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37631244,"num_objects":4594,"num_object_clones":0,"num_object_copies":18376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8145,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7870","reported_seq":7967,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:51.223960+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:51.223960+0000","last_peered":"2024-09-16T04:46:51.223960+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:51.223960+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7870,"log_dups_size":0,"ondisk_log_size":7870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36388864,"num_objects":4442,"num_object_clones":0,"num_object_copies":17768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7870,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8025","reported_seq":8098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:51.098060+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:51.098060+0000","last_peered":"2024-09-16T04:46:51.098060+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:51.098060+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8025,"log_dups_size":0,"ondisk_log_size":8025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37475596,"num_objects":4575,"num_object_clones":0,"num_object_copies":18300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8023,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2352}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8127","reported_seq":12684,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:47.852940+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:47.852940+0000","last_peered":"2024-09-16T04:46:47.852940+0000","last_clean":"2024-09-16T04:46:47.852940+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:47.852940+0000","last_undegraded":"2024-09-16T04:46:47.852940+0000","last_fullsized":"2024-09-16T04:46:47.852940+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:52.409049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38155532,"num_objects":4658,"num_object_clones":0,"num_object_copies":18632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8125,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8287","reported_seq":11443,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:51.325427+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:51.325427+0000","last_peered":"2024-09-16T04:46:51.325427+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:51.325427+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8287,"log_dups_size":0,"ondisk_log_size":8287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36604440,"num_objects":4469,"num_object_clones":0,"num_object_copies":17876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8284,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1552,"num_bytes_recovered":12591104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3675}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8103","reported_seq":8201,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:50.454056+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:50.454056+0000","last_peered":"2024-09-16T04:46:50.454056+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:50.454056+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8103,"log_dups_size":0,"ondisk_log_size":8103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38130956,"num_objects":4655,"num_object_clones":0,"num_object_copies":18620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8101,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2464}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8013","reported_seq":8124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:51.224113+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:51.224113+0000","last_peered":"2024-09-16T04:46:51.224113+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:51.224113+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36679570,"num_objects":4478,"num_object_clones":0,"num_object_copies":17912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8010,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7858","reported_seq":7935,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:51.224093+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:51.224093+0000","last_peered":"2024-09-16T04:46:51.224093+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:51.224093+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7858,"log_dups_size":0,"ondisk_log_size":7858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36641414,"num_objects":4473,"num_object_clones":0,"num_object_copies":17892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7857,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2364}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8001","reported_seq":8080,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:50.610466+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:50.610466+0000","last_peered":"2024-09-16T04:46:50.610466+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:50.610466+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8001,"log_dups_size":0,"ondisk_log_size":8001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37124742,"num_objects":4532,"num_object_clones":0,"num_object_copies":18128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8000,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2310}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8140","reported_seq":8258,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:51.267010+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:51.267010+0000","last_peered":"2024-09-16T04:46:51.267010+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:51.267010+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8140,"log_dups_size":0,"ondisk_log_size":8140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38024460,"num_objects":4642,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8138,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2440}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7890","reported_seq":7982,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:50.546018+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:50.546018+0000","last_peered":"2024-09-16T04:46:50.546018+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:50.546018+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7890,"log_dups_size":0,"ondisk_log_size":7890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37344524,"num_objects":4559,"num_object_clones":0,"num_object_copies":18236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7888,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2433}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7881","reported_seq":7966,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:51.138059+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:51.138059+0000","last_peered":"2024-09-16T04:46:51.138059+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:51.138059+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7881,"log_dups_size":0,"ondisk_log_size":7881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36254988,"num_objects":4426,"num_object_clones":0,"num_object_copies":17704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7879,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2196}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":597427904,"num_objects":72934,"num_object_clones":0,"num_object_copies":291736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128553,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4014,"num_bytes_recovered":30924800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1165201408,"data_stored":1165201408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128580,"ondisk_log_size":128580,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717080,"kb_used_data":292488,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93654760,"statfs":{"total":96636764160,"available":95902474240,"internally_reserved":0,"allocated":299507712,"data_stored":299327743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434761423},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,1,5,3,0,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743152,"kb_used_data":292704,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93628688,"statfs":{"total":96636764160,"available":95875776512,"internally_reserved":0,"allocated":299728896,"data_stored":299548927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,6,1,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051649,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570092,"kb_used_data":263516,"kb_used_omap":4,"kb_used_meta":306555,"kb_avail":93801748,"statfs":{"total":96636764160,"available":96052989952,"internally_reserved":0,"allocated":269840384,"data_stored":269660415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313913039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721088,"kb_used_data":292400,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93650752,"statfs":{"total":96636764160,"available":95898370048,"internally_reserved":0,"allocated":299417600,"data_stored":299237631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":438955726},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,1,0,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":298459136,"data_stored":298459136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269520896,"data_stored":269520896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298926080,"data_stored":298926080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":298295296,"data_stored":298295296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:57.086 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:57.682 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:57.683 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:57.710 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-09-16T04:46:56.676288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":588039872,"num_objects":71788,"num_object_clones":0,"num_object_copies":287152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71788,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129699,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4069,"num_bytes_recovered":31350784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129726,"ondisk_log_size":129726,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2790212,"kb_used_data":1127132,"kb_used_omap":14,"kb_used_meta":1662961,"kb_avail":374697148,"statfs":{"total":386547056640,"available":383689879552,"internally_reserved":0,"allocated":1154183168,"data_stored":1153463292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1702872625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,7,5,9,8,10,8,0,0,4,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28155904,"num_objects":-3437,"num_object_clones":0,"num_object_copies":-13748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3437,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008274"},"pg_stats":[{"pgid":"1.f","version":"33'8145","reported_seq":8238,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.279977+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:56.279977+0000","last_peered":"2024-09-16T04:46:56.279977+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:56.279977+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36937728,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8145,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2314}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8149","reported_seq":8228,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:55.997661+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:55.997661+0000","last_peered":"2024-09-16T04:46:55.997661+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:55.997661+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8149,"log_dups_size":0,"ondisk_log_size":8149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37065996,"num_objects":4525,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8147,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8117","reported_seq":8202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:55.697661+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:55.697661+0000","last_peered":"2024-09-16T04:46:55.697661+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:55.697661+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8117,"log_dups_size":0,"ondisk_log_size":8117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37296798,"num_objects":4554,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8116,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8124","reported_seq":8204,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.011425+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:56.011425+0000","last_peered":"2024-09-16T04:46:56.011425+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:56.011425+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8124,"log_dups_size":0,"ondisk_log_size":8124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37238028,"num_objects":4546,"num_object_clones":0,"num_object_copies":18184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8122,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8226","reported_seq":8338,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.123754+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:56.123754+0000","last_peered":"2024-09-16T04:46:56.123754+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:56.123754+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8226,"log_dups_size":0,"ondisk_log_size":8226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36984076,"num_objects":4515,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8224,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7942","reported_seq":8039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.374751+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:56.374751+0000","last_peered":"2024-09-16T04:46:56.374751+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:56.374751+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7942,"log_dups_size":0,"ondisk_log_size":7942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35799040,"num_objects":4370,"num_object_clones":0,"num_object_copies":17480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7942,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2188}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8083","reported_seq":8156,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.142735+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:56.142735+0000","last_peered":"2024-09-16T04:46:56.142735+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:56.142735+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8083,"log_dups_size":0,"ondisk_log_size":8083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37000460,"num_objects":4517,"num_object_clones":0,"num_object_copies":18068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8081,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2294}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8198","reported_seq":12755,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:52.960275+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:52.960275+0000","last_peered":"2024-09-16T04:46:52.960275+0000","last_clean":"2024-09-16T04:46:52.960275+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:52.960275+0000","last_undegraded":"2024-09-16T04:46:52.960275+0000","last_fullsized":"2024-09-16T04:46:52.960275+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8198,"log_dups_size":0,"ondisk_log_size":8198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:57.456952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37573900,"num_objects":4587,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8196,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8348","reported_seq":11609,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.347123+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:56.347123+0000","last_peered":"2024-09-16T04:46:56.347123+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:56.347123+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8348,"log_dups_size":0,"ondisk_log_size":8348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36104728,"num_objects":4408,"num_object_clones":0,"num_object_copies":17632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8345,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":13017088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3666}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8170","reported_seq":8268,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:55.456341+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:55.456341+0000","last_peered":"2024-09-16T04:46:55.456341+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:55.456341+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8170,"log_dups_size":0,"ondisk_log_size":8170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37582092,"num_objects":4588,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8168,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2397}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8083","reported_seq":8196,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.122934+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:56.122934+0000","last_peered":"2024-09-16T04:46:56.122934+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:56.122934+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8083,"log_dups_size":0,"ondisk_log_size":8083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36106130,"num_objects":4408,"num_object_clones":0,"num_object_copies":17632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8080,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7933","reported_seq":8012,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.273606+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:56.273606+0000","last_peered":"2024-09-16T04:46:56.273606+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:56.273606+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7933,"log_dups_size":0,"ondisk_log_size":7933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36027014,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2289}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8061","reported_seq":8142,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:55.518406+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:55.518406+0000","last_peered":"2024-09-16T04:46:55.518406+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:55.518406+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8061,"log_dups_size":0,"ondisk_log_size":8061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36633222,"num_objects":4472,"num_object_clones":0,"num_object_copies":17888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8060,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8217","reported_seq":8336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.270455+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:56.270455+0000","last_peered":"2024-09-16T04:46:56.270455+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:56.270455+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8217,"log_dups_size":0,"ondisk_log_size":8217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37393676,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8215,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2363}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7969","reported_seq":8061,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:55.553749+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:55.553749+0000","last_peered":"2024-09-16T04:46:55.553749+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:55.553749+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7969,"log_dups_size":0,"ondisk_log_size":7969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36697356,"num_objects":4480,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7967,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2354}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7961","reported_seq":8046,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.115073+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:56.115073+0000","last_peered":"2024-09-16T04:46:56.115073+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:56.115073+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7961,"log_dups_size":0,"ondisk_log_size":7961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35599628,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7959,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2116}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":588039872,"num_objects":71788,"num_object_clones":0,"num_object_copies":287152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71788,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129699,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4069,"num_bytes_recovered":31350784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1150844928,"data_stored":1150844928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129726,"ondisk_log_size":129726,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734736,"kb_used_data":287936,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93637104,"statfs":{"total":96636764160,"available":95884394496,"internally_reserved":0,"allocated":294846464,"data_stored":294666495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":457502415},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,2,3,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742760,"kb_used_data":288112,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93629080,"statfs":{"total":96636764160,"available":95876177920,"internally_reserved":0,"allocated":295026688,"data_stored":294846719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,0,3,2,1,5,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051650,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574020,"kb_used_data":263284,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93797820,"statfs":{"total":96636764160,"available":96048967680,"internally_reserved":0,"allocated":269602816,"data_stored":269422847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318172879},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,0,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738696,"kb_used_data":287800,"kb_used_omap":4,"kb_used_meta":450875,"kb_avail":93633144,"statfs":{"total":96636764160,"available":95880339456,"internally_reserved":0,"allocated":294707200,"data_stored":294527231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":461696718},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,1,3,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":293761024,"data_stored":293761024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269299712,"data_stored":269299712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294215680,"data_stored":294215680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":293568512,"data_stored":293568512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:57.712 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:58.297 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:58.297 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:58.322 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-09-16T04:46:56.676288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":588039872,"num_objects":71788,"num_object_clones":0,"num_object_copies":287152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71788,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129699,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4069,"num_bytes_recovered":31350784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129726,"ondisk_log_size":129726,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2790212,"kb_used_data":1127132,"kb_used_omap":14,"kb_used_meta":1662961,"kb_avail":374697148,"statfs":{"total":386547056640,"available":383689879552,"internally_reserved":0,"allocated":1154183168,"data_stored":1153463292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1702872625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,7,5,9,8,10,8,0,0,4,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28155904,"num_objects":-3437,"num_object_clones":0,"num_object_copies":-13748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3437,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008274"},"pg_stats":[{"pgid":"1.f","version":"33'8145","reported_seq":8238,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.279977+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:56.279977+0000","last_peered":"2024-09-16T04:46:56.279977+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:56.279977+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36937728,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8145,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2314}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8149","reported_seq":8228,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:55.997661+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:55.997661+0000","last_peered":"2024-09-16T04:46:55.997661+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:55.997661+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8149,"log_dups_size":0,"ondisk_log_size":8149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37065996,"num_objects":4525,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8147,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8117","reported_seq":8202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:55.697661+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:55.697661+0000","last_peered":"2024-09-16T04:46:55.697661+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:55.697661+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8117,"log_dups_size":0,"ondisk_log_size":8117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37296798,"num_objects":4554,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8116,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8124","reported_seq":8204,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.011425+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:56.011425+0000","last_peered":"2024-09-16T04:46:56.011425+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:56.011425+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8124,"log_dups_size":0,"ondisk_log_size":8124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37238028,"num_objects":4546,"num_object_clones":0,"num_object_copies":18184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8122,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8226","reported_seq":8338,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.123754+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:56.123754+0000","last_peered":"2024-09-16T04:46:56.123754+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:56.123754+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8226,"log_dups_size":0,"ondisk_log_size":8226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36984076,"num_objects":4515,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8224,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7942","reported_seq":8039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.374751+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:56.374751+0000","last_peered":"2024-09-16T04:46:56.374751+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:56.374751+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7942,"log_dups_size":0,"ondisk_log_size":7942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35799040,"num_objects":4370,"num_object_clones":0,"num_object_copies":17480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7942,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2188}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8083","reported_seq":8156,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.142735+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:56.142735+0000","last_peered":"2024-09-16T04:46:56.142735+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:56.142735+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8083,"log_dups_size":0,"ondisk_log_size":8083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37000460,"num_objects":4517,"num_object_clones":0,"num_object_copies":18068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8081,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2294}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8198","reported_seq":12755,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:52.960275+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:52.960275+0000","last_peered":"2024-09-16T04:46:52.960275+0000","last_clean":"2024-09-16T04:46:52.960275+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:52.960275+0000","last_undegraded":"2024-09-16T04:46:52.960275+0000","last_fullsized":"2024-09-16T04:46:52.960275+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8198,"log_dups_size":0,"ondisk_log_size":8198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:46:57.456952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37573900,"num_objects":4587,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8196,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8348","reported_seq":11609,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.347123+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:56.347123+0000","last_peered":"2024-09-16T04:46:56.347123+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:56.347123+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8348,"log_dups_size":0,"ondisk_log_size":8348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36104728,"num_objects":4408,"num_object_clones":0,"num_object_copies":17632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8345,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":13017088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3666}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8170","reported_seq":8268,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:55.456341+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:55.456341+0000","last_peered":"2024-09-16T04:46:55.456341+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:55.456341+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8170,"log_dups_size":0,"ondisk_log_size":8170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37582092,"num_objects":4588,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8168,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2397}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8083","reported_seq":8196,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.122934+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:56.122934+0000","last_peered":"2024-09-16T04:46:56.122934+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:56.122934+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8083,"log_dups_size":0,"ondisk_log_size":8083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36106130,"num_objects":4408,"num_object_clones":0,"num_object_copies":17632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8080,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7933","reported_seq":8012,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.273606+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:56.273606+0000","last_peered":"2024-09-16T04:46:56.273606+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:56.273606+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7933,"log_dups_size":0,"ondisk_log_size":7933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36027014,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2289}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8061","reported_seq":8142,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:55.518406+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:55.518406+0000","last_peered":"2024-09-16T04:46:55.518406+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:55.518406+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8061,"log_dups_size":0,"ondisk_log_size":8061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36633222,"num_objects":4472,"num_object_clones":0,"num_object_copies":17888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8060,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8217","reported_seq":8336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.270455+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:56.270455+0000","last_peered":"2024-09-16T04:46:56.270455+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:56.270455+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8217,"log_dups_size":0,"ondisk_log_size":8217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37393676,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8215,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2363}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7969","reported_seq":8061,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:55.553749+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:55.553749+0000","last_peered":"2024-09-16T04:46:55.553749+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:55.553749+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7969,"log_dups_size":0,"ondisk_log_size":7969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36697356,"num_objects":4480,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7967,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2354}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7961","reported_seq":8046,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.115073+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:56.115073+0000","last_peered":"2024-09-16T04:46:56.115073+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:56.115073+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7961,"log_dups_size":0,"ondisk_log_size":7961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35599628,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7959,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2116}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":588039872,"num_objects":71788,"num_object_clones":0,"num_object_copies":287152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71788,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129699,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4069,"num_bytes_recovered":31350784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1150844928,"data_stored":1150844928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129726,"ondisk_log_size":129726,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734736,"kb_used_data":287936,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93637104,"statfs":{"total":96636764160,"available":95884394496,"internally_reserved":0,"allocated":294846464,"data_stored":294666495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":457502415},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,2,3,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742760,"kb_used_data":288112,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93629080,"statfs":{"total":96636764160,"available":95876177920,"internally_reserved":0,"allocated":295026688,"data_stored":294846719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,0,3,2,1,5,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051650,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574020,"kb_used_data":263284,"kb_used_omap":4,"kb_used_meta":310715,"kb_avail":93797820,"statfs":{"total":96636764160,"available":96048967680,"internally_reserved":0,"allocated":269602816,"data_stored":269422847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318172879},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,0,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738696,"kb_used_data":287800,"kb_used_omap":4,"kb_used_meta":450875,"kb_avail":93633144,"statfs":{"total":96636764160,"available":95880339456,"internally_reserved":0,"allocated":294707200,"data_stored":294527231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":461696718},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,1,3,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":293761024,"data_stored":293761024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269299712,"data_stored":269299712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294215680,"data_stored":294215680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":293568512,"data_stored":293568512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:46:58.323 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:46:58.323 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8145", 'reported_seq': 8238, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:56.279977+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:46:56.279977+0000', 'last_peered': '2024-09-16T04:46:56.279977+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:46:56.279977+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8145, 'log_dups_size': 0, 'ondisk_log_size': 8145, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36937728, 'num_objects': 4509, 'num_object_clones': 0, 'num_object_copies': 18036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2195, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8145, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2314}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:58.323 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:46:58.323 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8149", 'reported_seq': 8228, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:55.997661+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:46:55.997661+0000', 'last_peered': '2024-09-16T04:46:55.997661+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:46:55.997661+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8149, 'log_dups_size': 0, 'ondisk_log_size': 8149, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37065996, 'num_objects': 4525, 'num_object_clones': 0, 'num_object_copies': 18100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2207, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8147, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2320}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:58.323 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:46:58.323 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8117", 'reported_seq': 8202, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:55.697661+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:46:55.697661+0000', 'last_peered': '2024-09-16T04:46:55.697661+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:46:55.697661+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8117, 'log_dups_size': 0, 'ondisk_log_size': 8117, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37296798, 'num_objects': 4554, 'num_object_clones': 0, 'num_object_copies': 18216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4554, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8116, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2331}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:58.323 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:46:58.323 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8124", 'reported_seq': 8204, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:56.011425+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:46:56.011425+0000', 'last_peered': '2024-09-16T04:46:56.011425+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:46:56.011425+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8124, 'log_dups_size': 0, 'ondisk_log_size': 8124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37238028, 'num_objects': 4546, 'num_object_clones': 0, 'num_object_copies': 18184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2225, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4546, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8122, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2323}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:58.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:46:58.324 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8226", 'reported_seq': 8338, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:56.123754+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:46:56.123754+0000', 'last_peered': '2024-09-16T04:46:56.123754+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:46:56.123754+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8226, 'log_dups_size': 0, 'ondisk_log_size': 8226, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36984076, 'num_objects': 4515, 'num_object_clones': 0, 'num_object_copies': 18060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2258, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8224, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 29, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2259}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:58.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:46:58.324 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7942", 'reported_seq': 8039, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:56.374751+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:46:56.374751+0000', 'last_peered': '2024-09-16T04:46:56.374751+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:46:56.374751+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7942, 'log_dups_size': 0, 'ondisk_log_size': 7942, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35799040, 'num_objects': 4370, 'num_object_clones': 0, 'num_object_copies': 17480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2182, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4370, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7942, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2188}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:58.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:46:58.324 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8083", 'reported_seq': 8156, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:56.142735+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:46:56.142735+0000', 'last_peered': '2024-09-16T04:46:56.142735+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:46:56.142735+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8083, 'log_dups_size': 0, 'ondisk_log_size': 8083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37000460, 'num_objects': 4517, 'num_object_clones': 0, 'num_object_copies': 18068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2225, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8081, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2294}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:58.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:46:58.324 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8348", 'reported_seq': 11609, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:56.347123+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:46:56.347123+0000', 'last_peered': '2024-09-16T04:46:56.347123+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:46:56.347123+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8348, 'log_dups_size': 0, 'ondisk_log_size': 8348, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36104728, 'num_objects': 4408, 'num_object_clones': 0, 'num_object_copies': 17632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 742, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8345, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1604, 'num_bytes_recovered': 13017088, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3666}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:58.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:46:58.324 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8170", 'reported_seq': 8268, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:55.456341+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:46:55.456341+0000', 'last_peered': '2024-09-16T04:46:55.456341+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:46:55.456341+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8170, 'log_dups_size': 0, 'ondisk_log_size': 8170, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37582092, 'num_objects': 4588, 'num_object_clones': 0, 'num_object_copies': 18352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2193, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8168, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2397}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:58.324 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:46:58.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8083", 'reported_seq': 8196, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:56.122934+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:46:56.122934+0000', 'last_peered': '2024-09-16T04:46:56.122934+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:46:56.122934+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8083, 'log_dups_size': 0, 'ondisk_log_size': 8083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36106130, 'num_objects': 4408, 'num_object_clones': 0, 'num_object_copies': 17632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2165, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8080, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:58.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:46:58.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7933", 'reported_seq': 8012, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:56.273606+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:46:56.273606+0000', 'last_peered': '2024-09-16T04:46:56.273606+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:46:56.273606+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7933, 'log_dups_size': 0, 'ondisk_log_size': 7933, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36027014, 'num_objects': 4398, 'num_object_clones': 0, 'num_object_copies': 17592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7932, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2289}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:58.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:46:58.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8061", 'reported_seq': 8142, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:55.518406+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:46:55.518406+0000', 'last_peered': '2024-09-16T04:46:55.518406+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:46:55.518406+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8061, 'log_dups_size': 0, 'ondisk_log_size': 8061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36633222, 'num_objects': 4472, 'num_object_clones': 0, 'num_object_copies': 17888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8060, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2250}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:58.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:46:58.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8217", 'reported_seq': 8336, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:56.270455+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:46:56.270455+0000', 'last_peered': '2024-09-16T04:46:56.270455+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:46:56.270455+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8217, 'log_dups_size': 0, 'ondisk_log_size': 8217, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37393676, 'num_objects': 4565, 'num_object_clones': 0, 'num_object_copies': 18260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2204, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4565, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8215, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2363}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:46:58.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:46:58.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7969", 'reported_seq': 8061, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:55.553749+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:46:55.553749+0000', 'last_peered': '2024-09-16T04:46:55.553749+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:46:55.553749+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7969, 'log_dups_size': 0, 'ondisk_log_size': 7969, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36697356, 'num_objects': 4480, 'num_object_clones': 0, 'num_object_copies': 17920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2128, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7967, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2354}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:46:58.325 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:46:58.325 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7961", 'reported_seq': 8046, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:46:56.115073+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:46:56.115073+0000', 'last_peered': '2024-09-16T04:46:56.115073+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:46:56.115073+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7961, 'log_dups_size': 0, 'ondisk_log_size': 7961, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35599628, 'num_objects': 4346, 'num_object_clones': 0, 'num_object_copies': 17384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2232, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7959, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2116}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:46:58.326 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:46:59.094 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:59.094 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31513/286900 objects degraded (10.984%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":431,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":71725,"data_bytes":587523776,"bytes_used":2861182976,"bytes_avail":383685873664,"bytes_total":386547056640,"degraded_objects":31513,"degraded_total":286900,"degraded_ratio":0.10983966538863715,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":69583,"recovering_keys_per_sec":0,"num_objects_recovered":116,"num_bytes_recovered":835584,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":197},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:46:59.120 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:46:59.120 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:46:59.704 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:46:59.705 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:46:59.723 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-09-16T04:46:58.677128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":587523776,"num_objects":71725,"num_object_clones":0,"num_object_copies":286900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129762,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4069,"num_bytes_recovered":31350784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129789,"ondisk_log_size":129789,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2794124,"kb_used_data":1126884,"kb_used_omap":14,"kb_used_meta":1667121,"kb_avail":374693236,"statfs":{"total":386547056640,"available":383685873664,"internally_reserved":0,"allocated":1153929216,"data_stored":1153209340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1707132465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,6,10,6,6,8,0,0,4,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19464192,"num_objects":-2376,"num_object_clones":0,"num_object_copies":-9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2376,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":835584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008431"},"pg_stats":[{"pgid":"1.f","version":"33'8145","reported_seq":8238,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.279977+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:46:56.279977+0000","last_peered":"2024-09-16T04:46:56.279977+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:46:56.279977+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36937728,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8145,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2314}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8149","reported_seq":8228,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:55.997661+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:46:55.997661+0000","last_peered":"2024-09-16T04:46:55.997661+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:46:55.997661+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8149,"log_dups_size":0,"ondisk_log_size":8149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37065996,"num_objects":4525,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8147,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8117","reported_seq":8202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:55.697661+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:46:55.697661+0000","last_peered":"2024-09-16T04:46:55.697661+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:46:55.697661+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8117,"log_dups_size":0,"ondisk_log_size":8117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37296798,"num_objects":4554,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8116,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2331}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8124","reported_seq":8204,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.011425+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:46:56.011425+0000","last_peered":"2024-09-16T04:46:56.011425+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:46:56.011425+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8124,"log_dups_size":0,"ondisk_log_size":8124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37238028,"num_objects":4546,"num_object_clones":0,"num_object_copies":18184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8122,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8226","reported_seq":8338,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.123754+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:46:56.123754+0000","last_peered":"2024-09-16T04:46:56.123754+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:46:56.123754+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8226,"log_dups_size":0,"ondisk_log_size":8226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36984076,"num_objects":4515,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8224,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7942","reported_seq":8039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.374751+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:46:56.374751+0000","last_peered":"2024-09-16T04:46:56.374751+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:46:56.374751+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7942,"log_dups_size":0,"ondisk_log_size":7942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35799040,"num_objects":4370,"num_object_clones":0,"num_object_copies":17480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7942,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2188}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8083","reported_seq":8156,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.142735+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:46:56.142735+0000","last_peered":"2024-09-16T04:46:56.142735+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:46:56.142735+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8083,"log_dups_size":0,"ondisk_log_size":8083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37000460,"num_objects":4517,"num_object_clones":0,"num_object_copies":18068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8081,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2294}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8261","reported_seq":12818,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:57.818506+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:57.818506+0000","last_peered":"2024-09-16T04:46:57.818506+0000","last_clean":"2024-09-16T04:46:57.818506+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:57.818506+0000","last_undegraded":"2024-09-16T04:46:57.818506+0000","last_fullsized":"2024-09-16T04:46:57.818506+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37057804,"num_objects":4524,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8348","reported_seq":11609,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.347123+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:46:56.347123+0000","last_peered":"2024-09-16T04:46:56.347123+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:46:56.347123+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8348,"log_dups_size":0,"ondisk_log_size":8348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36104728,"num_objects":4408,"num_object_clones":0,"num_object_copies":17632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8345,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":13017088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3666}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8170","reported_seq":8268,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:55.456341+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:46:55.456341+0000","last_peered":"2024-09-16T04:46:55.456341+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:46:55.456341+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8170,"log_dups_size":0,"ondisk_log_size":8170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37582092,"num_objects":4588,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8168,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2397}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8083","reported_seq":8196,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.122934+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:46:56.122934+0000","last_peered":"2024-09-16T04:46:56.122934+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:46:56.122934+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8083,"log_dups_size":0,"ondisk_log_size":8083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36106130,"num_objects":4408,"num_object_clones":0,"num_object_copies":17632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8080,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7933","reported_seq":8012,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.273606+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:46:56.273606+0000","last_peered":"2024-09-16T04:46:56.273606+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:46:56.273606+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7933,"log_dups_size":0,"ondisk_log_size":7933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36027014,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2289}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8061","reported_seq":8142,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:55.518406+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:46:55.518406+0000","last_peered":"2024-09-16T04:46:55.518406+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:46:55.518406+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8061,"log_dups_size":0,"ondisk_log_size":8061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36633222,"num_objects":4472,"num_object_clones":0,"num_object_copies":17888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8060,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8217","reported_seq":8336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.270455+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:46:56.270455+0000","last_peered":"2024-09-16T04:46:56.270455+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:46:56.270455+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8217,"log_dups_size":0,"ondisk_log_size":8217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37393676,"num_objects":4565,"num_object_clones":0,"num_object_copies":18260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8215,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2363}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7969","reported_seq":8061,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:55.553749+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:46:55.553749+0000","last_peered":"2024-09-16T04:46:55.553749+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:46:55.553749+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7969,"log_dups_size":0,"ondisk_log_size":7969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36697356,"num_objects":4480,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7967,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2354}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7961","reported_seq":8046,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:46:56.115073+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:46:56.115073+0000","last_peered":"2024-09-16T04:46:56.115073+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:46:56.115073+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7961,"log_dups_size":0,"ondisk_log_size":7961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35599628,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7959,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2116}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":587523776,"num_objects":71725,"num_object_clones":0,"num_object_copies":286900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129762,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4069,"num_bytes_recovered":31350784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1150590976,"data_stored":1150590976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129789,"ondisk_log_size":129789,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734736,"kb_used_data":287936,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93637104,"statfs":{"total":96636764160,"available":95884394496,"internally_reserved":0,"allocated":294846464,"data_stored":294666495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":457502415},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,2,3,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742760,"kb_used_data":288112,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93629080,"statfs":{"total":96636764160,"available":95876177920,"internally_reserved":0,"allocated":295026688,"data_stored":294846719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,0,3,2,1,5,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577932,"kb_used_data":263036,"kb_used_omap":4,"kb_used_meta":314875,"kb_avail":93793908,"statfs":{"total":96636764160,"available":96044961792,"internally_reserved":0,"allocated":269348864,"data_stored":269168895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":322432719},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738696,"kb_used_data":287800,"kb_used_omap":4,"kb_used_meta":450875,"kb_avail":93633144,"statfs":{"total":96636764160,"available":95880339456,"internally_reserved":0,"allocated":294707200,"data_stored":294527231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":461696718},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,1,3,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":293761024,"data_stored":293761024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269045760,"data_stored":269045760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294215680,"data_stored":294215680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":293568512,"data_stored":293568512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:02.725 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:03.327 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:03.328 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:03.351 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-16T04:47:02.678108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":579184320,"num_objects":70707,"num_object_clones":0,"num_object_copies":282828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130780,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4144,"num_bytes_recovered":31883264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130807,"ondisk_log_size":130807,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2811900,"kb_used_data":1112740,"kb_used_omap":14,"kb_used_meta":1699057,"kb_avail":374675460,"statfs":{"total":386547056640,"available":383667671040,"internally_reserved":0,"allocated":1139445760,"data_stored":1138725884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1739834929},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,11,10,14,5,4,0,0,1,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27246592,"num_objects":-3326,"num_object_clones":0,"num_object_copies":-13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3326,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":1368064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007666"},"pg_stats":[{"pgid":"1.f","version":"33'8214","reported_seq":8307,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.256856+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:01.256856+0000","last_peered":"2024-09-16T04:47:01.256856+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:01.256856+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8214,"log_dups_size":0,"ondisk_log_size":8214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36372480,"num_objects":4440,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8214,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8207","reported_seq":8286,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.073457+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:01.073457+0000","last_peered":"2024-09-16T04:47:01.073457+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:01.073457+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8207,"log_dups_size":0,"ondisk_log_size":8207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36590860,"num_objects":4467,"num_object_clones":0,"num_object_copies":17868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8205,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8195","reported_seq":8283,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:00.762484+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:00.762484+0000","last_peered":"2024-09-16T04:47:00.762484+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:00.762484+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8195,"log_dups_size":0,"ondisk_log_size":8195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36657822,"num_objects":4476,"num_object_clones":0,"num_object_copies":17904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8194,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2254}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8201","reported_seq":8283,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:00.980602+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:00.980602+0000","last_peered":"2024-09-16T04:47:00.980602+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:00.980602+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8201,"log_dups_size":0,"ondisk_log_size":8201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36607244,"num_objects":4469,"num_object_clones":0,"num_object_copies":17876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8199,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2247}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8298","reported_seq":8412,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.332803+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:01.332803+0000","last_peered":"2024-09-16T04:47:01.332803+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:01.332803+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8298,"log_dups_size":0,"ondisk_log_size":8298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36394252,"num_objects":4443,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8296,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2188}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8013","reported_seq":8112,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.336310+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:01.336310+0000","last_peered":"2024-09-16T04:47:01.336310+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:01.336310+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35217408,"num_objects":4299,"num_object_clones":0,"num_object_copies":17196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8013,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8160","reported_seq":8241,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.070318+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:01.070318+0000","last_peered":"2024-09-16T04:47:01.070318+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:01.070318+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36369676,"num_objects":4440,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2221}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8261","reported_seq":12818,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:57.818506+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:57.818506+0000","last_peered":"2024-09-16T04:46:57.818506+0000","last_clean":"2024-09-16T04:46:57.818506+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:57.818506+0000","last_undegraded":"2024-09-16T04:46:57.818506+0000","last_fullsized":"2024-09-16T04:46:57.818506+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37057804,"num_objects":4524,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8408","reported_seq":11800,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.397331+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:01.397331+0000","last_peered":"2024-09-16T04:47:01.397331+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:01.397331+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8408,"log_dups_size":0,"ondisk_log_size":8408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35613208,"num_objects":4348,"num_object_clones":0,"num_object_copies":17392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8405,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1669,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3671}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8246","reported_seq":8344,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:00.611493+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:00.611493+0000","last_peered":"2024-09-16T04:47:00.611493+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:00.611493+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8246,"log_dups_size":0,"ondisk_log_size":8246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36959500,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2321}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8145","reported_seq":8258,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.362712+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:01.362712+0000","last_peered":"2024-09-16T04:47:01.362712+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:01.362712+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35598226,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8142,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7990","reported_seq":8069,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.256837+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:01.256837+0000","last_peered":"2024-09-16T04:47:01.256837+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:01.256837+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7990,"log_dups_size":0,"ondisk_log_size":7990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35560070,"num_objects":4341,"num_object_clones":0,"num_object_copies":17364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7989,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2232}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8122","reported_seq":8202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:00.714478+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:00.714478+0000","last_peered":"2024-09-16T04:47:00.714478+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:00.714478+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8122,"log_dups_size":0,"ondisk_log_size":8122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36133510,"num_objects":4411,"num_object_clones":0,"num_object_copies":17644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8121,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2189}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8279","reported_seq":8398,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.420304+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:01.420304+0000","last_peered":"2024-09-16T04:47:01.420304+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:01.420304+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8279,"log_dups_size":0,"ondisk_log_size":8279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36885772,"num_objects":4503,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8277,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2301}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8034","reported_seq":8127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:00.777638+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:00.777638+0000","last_peered":"2024-09-16T04:47:00.777638+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:00.777638+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8034,"log_dups_size":0,"ondisk_log_size":8034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36164876,"num_objects":4415,"num_object_clones":0,"num_object_copies":17660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8032,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2289}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8034","reported_seq":8121,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.052958+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:01.052958+0000","last_peered":"2024-09-16T04:47:01.052958+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:01.052958+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8034,"log_dups_size":0,"ondisk_log_size":8034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35001612,"num_objects":4273,"num_object_clones":0,"num_object_copies":17092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8032,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2044}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":579184320,"num_objects":70707,"num_object_clones":0,"num_object_copies":282828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130780,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4144,"num_bytes_recovered":31883264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1137131520,"data_stored":1137131520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130807,"ondisk_log_size":130807,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751764,"kb_used_data":282740,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93620076,"statfs":{"total":96636764160,"available":95866957824,"internally_reserved":0,"allocated":289525760,"data_stored":289345791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,5,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743796,"kb_used_data":283620,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93628044,"statfs":{"total":96636764160,"available":95875117056,"internally_reserved":0,"allocated":290426880,"data_stored":290246911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,1,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577932,"kb_used_data":263036,"kb_used_omap":4,"kb_used_meta":314875,"kb_avail":93793908,"statfs":{"total":96636764160,"available":96044961792,"internally_reserved":0,"allocated":269348864,"data_stored":269168895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":322432719},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738408,"kb_used_data":283344,"kb_used_omap":4,"kb_used_meta":455035,"kb_avail":93633432,"statfs":{"total":96636764160,"available":95880634368,"internally_reserved":0,"allocated":290144256,"data_stored":289964287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":465956558},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,4,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":289288192,"data_stored":289288192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269045760,"data_stored":269045760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289697792,"data_stored":289697792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":289099776,"data_stored":289099776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:03.352 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:03.878 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:03.879 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:03.904 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-16T04:47:02.678108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":579184320,"num_objects":70707,"num_object_clones":0,"num_object_copies":282828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130780,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4144,"num_bytes_recovered":31883264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130807,"ondisk_log_size":130807,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2811900,"kb_used_data":1112740,"kb_used_omap":14,"kb_used_meta":1699057,"kb_avail":374675460,"statfs":{"total":386547056640,"available":383667671040,"internally_reserved":0,"allocated":1139445760,"data_stored":1138725884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1739834929},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,11,10,14,5,4,0,0,1,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27246592,"num_objects":-3326,"num_object_clones":0,"num_object_copies":-13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3326,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":1368064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007666"},"pg_stats":[{"pgid":"1.f","version":"33'8214","reported_seq":8307,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.256856+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:01.256856+0000","last_peered":"2024-09-16T04:47:01.256856+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:01.256856+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8214,"log_dups_size":0,"ondisk_log_size":8214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36372480,"num_objects":4440,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8214,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8207","reported_seq":8286,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.073457+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:01.073457+0000","last_peered":"2024-09-16T04:47:01.073457+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:01.073457+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8207,"log_dups_size":0,"ondisk_log_size":8207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36590860,"num_objects":4467,"num_object_clones":0,"num_object_copies":17868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8205,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8195","reported_seq":8283,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:00.762484+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:00.762484+0000","last_peered":"2024-09-16T04:47:00.762484+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:00.762484+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8195,"log_dups_size":0,"ondisk_log_size":8195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36657822,"num_objects":4476,"num_object_clones":0,"num_object_copies":17904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8194,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2254}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8201","reported_seq":8283,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:00.980602+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:00.980602+0000","last_peered":"2024-09-16T04:47:00.980602+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:00.980602+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8201,"log_dups_size":0,"ondisk_log_size":8201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36607244,"num_objects":4469,"num_object_clones":0,"num_object_copies":17876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8199,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2247}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8298","reported_seq":8412,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.332803+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:01.332803+0000","last_peered":"2024-09-16T04:47:01.332803+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:01.332803+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8298,"log_dups_size":0,"ondisk_log_size":8298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36394252,"num_objects":4443,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8296,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2188}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8013","reported_seq":8112,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.336310+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:01.336310+0000","last_peered":"2024-09-16T04:47:01.336310+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:01.336310+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35217408,"num_objects":4299,"num_object_clones":0,"num_object_copies":17196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8013,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8160","reported_seq":8241,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.070318+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:01.070318+0000","last_peered":"2024-09-16T04:47:01.070318+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:01.070318+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36369676,"num_objects":4440,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2221}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8261","reported_seq":12818,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:46:57.818506+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:46:57.818506+0000","last_peered":"2024-09-16T04:46:57.818506+0000","last_clean":"2024-09-16T04:46:57.818506+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:46:57.818506+0000","last_undegraded":"2024-09-16T04:46:57.818506+0000","last_fullsized":"2024-09-16T04:46:57.818506+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8261,"log_dups_size":0,"ondisk_log_size":8261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37057804,"num_objects":4524,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8408","reported_seq":11800,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.397331+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:01.397331+0000","last_peered":"2024-09-16T04:47:01.397331+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:01.397331+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8408,"log_dups_size":0,"ondisk_log_size":8408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35613208,"num_objects":4348,"num_object_clones":0,"num_object_copies":17392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8405,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1669,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3671}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8246","reported_seq":8344,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:00.611493+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:00.611493+0000","last_peered":"2024-09-16T04:47:00.611493+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:00.611493+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8246,"log_dups_size":0,"ondisk_log_size":8246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36959500,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2321}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8145","reported_seq":8258,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.362712+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:01.362712+0000","last_peered":"2024-09-16T04:47:01.362712+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:01.362712+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35598226,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8142,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7990","reported_seq":8069,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.256837+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:01.256837+0000","last_peered":"2024-09-16T04:47:01.256837+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:01.256837+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7990,"log_dups_size":0,"ondisk_log_size":7990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35560070,"num_objects":4341,"num_object_clones":0,"num_object_copies":17364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7989,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2232}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8122","reported_seq":8202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:00.714478+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:00.714478+0000","last_peered":"2024-09-16T04:47:00.714478+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:00.714478+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8122,"log_dups_size":0,"ondisk_log_size":8122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36133510,"num_objects":4411,"num_object_clones":0,"num_object_copies":17644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8121,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2189}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8279","reported_seq":8398,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.420304+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:01.420304+0000","last_peered":"2024-09-16T04:47:01.420304+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:01.420304+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8279,"log_dups_size":0,"ondisk_log_size":8279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36885772,"num_objects":4503,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8277,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2301}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8034","reported_seq":8127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:00.777638+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:00.777638+0000","last_peered":"2024-09-16T04:47:00.777638+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:00.777638+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8034,"log_dups_size":0,"ondisk_log_size":8034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36164876,"num_objects":4415,"num_object_clones":0,"num_object_copies":17660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8032,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2289}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8034","reported_seq":8121,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.052958+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:01.052958+0000","last_peered":"2024-09-16T04:47:01.052958+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:01.052958+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8034,"log_dups_size":0,"ondisk_log_size":8034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35001612,"num_objects":4273,"num_object_clones":0,"num_object_copies":17092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8032,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2044}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":579184320,"num_objects":70707,"num_object_clones":0,"num_object_copies":282828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130780,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4144,"num_bytes_recovered":31883264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1137131520,"data_stored":1137131520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130807,"ondisk_log_size":130807,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751764,"kb_used_data":282740,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93620076,"statfs":{"total":96636764160,"available":95866957824,"internally_reserved":0,"allocated":289525760,"data_stored":289345791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,5,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743796,"kb_used_data":283620,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93628044,"statfs":{"total":96636764160,"available":95875117056,"internally_reserved":0,"allocated":290426880,"data_stored":290246911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,1,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577932,"kb_used_data":263036,"kb_used_omap":4,"kb_used_meta":314875,"kb_avail":93793908,"statfs":{"total":96636764160,"available":96044961792,"internally_reserved":0,"allocated":269348864,"data_stored":269168895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":322432719},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738408,"kb_used_data":283344,"kb_used_omap":4,"kb_used_meta":455035,"kb_avail":93633432,"statfs":{"total":96636764160,"available":95880634368,"internally_reserved":0,"allocated":290144256,"data_stored":289964287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":465956558},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,4,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":289288192,"data_stored":289288192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269045760,"data_stored":269045760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289697792,"data_stored":289697792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":289099776,"data_stored":289099776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:03.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:47:03.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8214", 'reported_seq': 8307, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:01.256856+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:47:01.256856+0000', 'last_peered': '2024-09-16T04:47:01.256856+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:47:01.256856+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8214, 'log_dups_size': 0, 'ondisk_log_size': 8214, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36372480, 'num_objects': 4440, 'num_object_clones': 0, 'num_object_copies': 17760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2195, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8214, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2245}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:03.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:47:03.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8207", 'reported_seq': 8286, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:01.073457+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:47:01.073457+0000', 'last_peered': '2024-09-16T04:47:01.073457+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:47:01.073457+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8207, 'log_dups_size': 0, 'ondisk_log_size': 8207, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36590860, 'num_objects': 4467, 'num_object_clones': 0, 'num_object_copies': 17868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2207, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4467, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8205, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:03.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:47:03.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8195", 'reported_seq': 8283, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:00.762484+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:47:00.762484+0000', 'last_peered': '2024-09-16T04:47:00.762484+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:47:00.762484+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8195, 'log_dups_size': 0, 'ondisk_log_size': 8195, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36657822, 'num_objects': 4476, 'num_object_clones': 0, 'num_object_copies': 17904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4476, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8194, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2254}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:03.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:47:03.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8201", 'reported_seq': 8283, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:00.980602+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:47:00.980602+0000', 'last_peered': '2024-09-16T04:47:00.980602+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:47:00.980602+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8201, 'log_dups_size': 0, 'ondisk_log_size': 8201, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36607244, 'num_objects': 4469, 'num_object_clones': 0, 'num_object_copies': 17876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8199, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2247}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:03.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:47:03.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8298", 'reported_seq': 8412, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:01.332803+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:47:01.332803+0000', 'last_peered': '2024-09-16T04:47:01.332803+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:47:01.332803+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8298, 'log_dups_size': 0, 'ondisk_log_size': 8298, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36394252, 'num_objects': 4443, 'num_object_clones': 0, 'num_object_copies': 17772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4443, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8296, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2188}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:03.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:47:03.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8013", 'reported_seq': 8112, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:01.336310+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:47:01.336310+0000', 'last_peered': '2024-09-16T04:47:01.336310+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:47:01.336310+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8013, 'log_dups_size': 0, 'ondisk_log_size': 8013, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35217408, 'num_objects': 4299, 'num_object_clones': 0, 'num_object_copies': 17196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2180, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4299, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8013, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:03.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:47:03.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8160", 'reported_seq': 8241, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:01.070318+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:47:01.070318+0000', 'last_peered': '2024-09-16T04:47:01.070318+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:47:01.070318+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8160, 'log_dups_size': 0, 'ondisk_log_size': 8160, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36369676, 'num_objects': 4440, 'num_object_clones': 0, 'num_object_copies': 17760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8158, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2221}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:03.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:47:03.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8408", 'reported_seq': 11800, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:01.397331+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:47:01.397331+0000', 'last_peered': '2024-09-16T04:47:01.397331+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:47:01.397331+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8408, 'log_dups_size': 0, 'ondisk_log_size': 8408, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35613208, 'num_objects': 4348, 'num_object_clones': 0, 'num_object_copies': 17392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 677, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4348, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8405, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1669, 'num_bytes_recovered': 13549568, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3671}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:03.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:47:03.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8246", 'reported_seq': 8344, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:00.611493+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:47:00.611493+0000', 'last_peered': '2024-09-16T04:47:00.611493+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:47:00.611493+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8246, 'log_dups_size': 0, 'ondisk_log_size': 8246, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36959500, 'num_objects': 4512, 'num_object_clones': 0, 'num_object_copies': 18048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2193, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8244, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2321}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:03.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:47:03.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8145", 'reported_seq': 8258, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:01.362712+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:47:01.362712+0000', 'last_peered': '2024-09-16T04:47:01.362712+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:47:01.362712+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8145, 'log_dups_size': 0, 'ondisk_log_size': 8145, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35598226, 'num_objects': 4346, 'num_object_clones': 0, 'num_object_copies': 17384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2165, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8142, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2184}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:03.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:47:03.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7990", 'reported_seq': 8069, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:01.256837+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:47:01.256837+0000', 'last_peered': '2024-09-16T04:47:01.256837+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:47:01.256837+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 7990, 'log_dups_size': 0, 'ondisk_log_size': 7990, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35560070, 'num_objects': 4341, 'num_object_clones': 0, 'num_object_copies': 17364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7989, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2232}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:03.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:47:03.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8122", 'reported_seq': 8202, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:00.714478+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:47:00.714478+0000', 'last_peered': '2024-09-16T04:47:00.714478+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:47:00.714478+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8122, 'log_dups_size': 0, 'ondisk_log_size': 8122, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36133510, 'num_objects': 4411, 'num_object_clones': 0, 'num_object_copies': 17644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4411, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8121, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2189}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:03.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:47:03.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8279", 'reported_seq': 8398, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:01.420304+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:47:01.420304+0000', 'last_peered': '2024-09-16T04:47:01.420304+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:47:01.420304+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8279, 'log_dups_size': 0, 'ondisk_log_size': 8279, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36885772, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 18012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2204, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8277, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:03.907 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:47:03.907 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8034", 'reported_seq': 8127, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:00.777638+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:47:00.777638+0000', 'last_peered': '2024-09-16T04:47:00.777638+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:47:00.777638+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8034, 'log_dups_size': 0, 'ondisk_log_size': 8034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36164876, 'num_objects': 4415, 'num_object_clones': 0, 'num_object_copies': 17660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2128, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4415, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8032, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2289}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:03.908 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:47:03.908 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8034", 'reported_seq': 8121, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:01.052958+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:47:01.052958+0000', 'last_peered': '2024-09-16T04:47:01.052958+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:47:01.052958+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8034, 'log_dups_size': 0, 'ondisk_log_size': 8034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35001612, 'num_objects': 4273, 'num_object_clones': 0, 'num_object_copies': 17092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8032, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2044}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:03.908 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:47:04.597 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:04.598 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31438/282828 objects degraded (11.116%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":436,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":70707,"data_bytes":579184320,"bytes_used":2879385600,"bytes_avail":383667671040,"bytes_total":386547056640,"degraded_objects":31438,"degraded_total":282828,"degraded_ratio":0.1111558968701826,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":113932,"recovering_keys_per_sec":0,"num_objects_recovered":191,"num_bytes_recovered":1368064,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":276},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:47:04.616 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:47:04.616 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:05.222 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:05.222 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:05.254 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-09-16T04:47:04.678638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":578545344,"num_objects":70629,"num_object_clones":0,"num_object_copies":282516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70629,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130858,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4144,"num_bytes_recovered":31883264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130885,"ondisk_log_size":130885,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799120,"kb_used_data":1112440,"kb_used_omap":14,"kb_used_meta":1686577,"kb_avail":374688240,"statfs":{"total":386547056640,"available":383680757760,"internally_reserved":0,"allocated":1139138560,"data_stored":1138418684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1727055409},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,9,9,17,7,4,0,0,1,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18882560,"num_objects":-2305,"num_object_clones":0,"num_object_copies":-9220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2305,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003688"},"pg_stats":[{"pgid":"1.f","version":"33'8214","reported_seq":8307,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.256856+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:01.256856+0000","last_peered":"2024-09-16T04:47:01.256856+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:01.256856+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8214,"log_dups_size":0,"ondisk_log_size":8214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36372480,"num_objects":4440,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8214,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8207","reported_seq":8286,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.073457+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:01.073457+0000","last_peered":"2024-09-16T04:47:01.073457+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:01.073457+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8207,"log_dups_size":0,"ondisk_log_size":8207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36590860,"num_objects":4467,"num_object_clones":0,"num_object_copies":17868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8205,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8195","reported_seq":8283,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:00.762484+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:00.762484+0000","last_peered":"2024-09-16T04:47:00.762484+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:00.762484+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8195,"log_dups_size":0,"ondisk_log_size":8195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36657822,"num_objects":4476,"num_object_clones":0,"num_object_copies":17904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8194,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2254}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8201","reported_seq":8283,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:00.980602+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:00.980602+0000","last_peered":"2024-09-16T04:47:00.980602+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:00.980602+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8201,"log_dups_size":0,"ondisk_log_size":8201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36607244,"num_objects":4469,"num_object_clones":0,"num_object_copies":17876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8199,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2247}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8298","reported_seq":8412,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.332803+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:01.332803+0000","last_peered":"2024-09-16T04:47:01.332803+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:01.332803+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8298,"log_dups_size":0,"ondisk_log_size":8298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36394252,"num_objects":4443,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8296,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2188}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8013","reported_seq":8112,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.336310+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:01.336310+0000","last_peered":"2024-09-16T04:47:01.336310+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:01.336310+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35217408,"num_objects":4299,"num_object_clones":0,"num_object_copies":17196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8013,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8160","reported_seq":8241,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.070318+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:01.070318+0000","last_peered":"2024-09-16T04:47:01.070318+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:01.070318+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36369676,"num_objects":4440,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2221}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8339","reported_seq":12896,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:03.013862+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:03.013862+0000","last_peered":"2024-09-16T04:47:03.013862+0000","last_clean":"2024-09-16T04:47:03.013862+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:03.013862+0000","last_undegraded":"2024-09-16T04:47:03.013862+0000","last_fullsized":"2024-09-16T04:47:03.013862+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8339,"log_dups_size":0,"ondisk_log_size":8339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:03.441437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36418828,"num_objects":4446,"num_object_clones":0,"num_object_copies":17784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8337,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8408","reported_seq":11800,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.397331+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:01.397331+0000","last_peered":"2024-09-16T04:47:01.397331+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:01.397331+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8408,"log_dups_size":0,"ondisk_log_size":8408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35613208,"num_objects":4348,"num_object_clones":0,"num_object_copies":17392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8405,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1669,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3671}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8246","reported_seq":8344,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:00.611493+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:00.611493+0000","last_peered":"2024-09-16T04:47:00.611493+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:00.611493+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8246,"log_dups_size":0,"ondisk_log_size":8246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36959500,"num_objects":4512,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2321}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8145","reported_seq":8258,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.362712+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:01.362712+0000","last_peered":"2024-09-16T04:47:01.362712+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:01.362712+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35598226,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8142,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7990","reported_seq":8069,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.256837+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:01.256837+0000","last_peered":"2024-09-16T04:47:01.256837+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:01.256837+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":7990,"log_dups_size":0,"ondisk_log_size":7990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35560070,"num_objects":4341,"num_object_clones":0,"num_object_copies":17364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7989,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2232}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8122","reported_seq":8202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:00.714478+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:00.714478+0000","last_peered":"2024-09-16T04:47:00.714478+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:00.714478+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8122,"log_dups_size":0,"ondisk_log_size":8122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36133510,"num_objects":4411,"num_object_clones":0,"num_object_copies":17644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8121,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2189}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8279","reported_seq":8398,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.420304+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:01.420304+0000","last_peered":"2024-09-16T04:47:01.420304+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:01.420304+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8279,"log_dups_size":0,"ondisk_log_size":8279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36885772,"num_objects":4503,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8277,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2301}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8034","reported_seq":8127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:00.777638+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:00.777638+0000","last_peered":"2024-09-16T04:47:00.777638+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:00.777638+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8034,"log_dups_size":0,"ondisk_log_size":8034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36164876,"num_objects":4415,"num_object_clones":0,"num_object_copies":17660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8032,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2289}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8034","reported_seq":8121,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:01.052958+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:01.052958+0000","last_peered":"2024-09-16T04:47:01.052958+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:01.052958+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8034,"log_dups_size":0,"ondisk_log_size":8034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35001612,"num_objects":4273,"num_object_clones":0,"num_object_copies":17092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8032,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2044}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":578545344,"num_objects":70629,"num_object_clones":0,"num_object_copies":282516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70629,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130858,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4144,"num_bytes_recovered":31883264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1136803840,"data_stored":1136803840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130885,"ondisk_log_size":130885,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751764,"kb_used_data":282740,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93620076,"statfs":{"total":96636764160,"available":95866957824,"internally_reserved":0,"allocated":289525760,"data_stored":289345791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,5,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743796,"kb_used_data":283620,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93628044,"statfs":{"total":96636764160,"available":95875117056,"internally_reserved":0,"allocated":290426880,"data_stored":290246911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,1,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051652,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565152,"kb_used_data":262736,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93806688,"statfs":{"total":96636764160,"available":96058048512,"internally_reserved":0,"allocated":269041664,"data_stored":268861695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309653199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738408,"kb_used_data":283344,"kb_used_omap":4,"kb_used_meta":455035,"kb_avail":93633432,"statfs":{"total":96636764160,"available":95880634368,"internally_reserved":0,"allocated":290144256,"data_stored":289964287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":465956558},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,4,1,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":289288192,"data_stored":289288192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268718080,"data_stored":268718080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289697792,"data_stored":289697792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":289099776,"data_stored":289099776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:08.255 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:08.883 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:08.884 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:08.913 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-09-16T04:47:08.679655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":569779904,"num_objects":69559,"num_object_clones":0,"num_object_copies":278236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69559,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131928,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4225,"num_bytes_recovered":32399360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131955,"ondisk_log_size":131955,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2822436,"kb_used_data":1099676,"kb_used_omap":14,"kb_used_meta":1722673,"kb_avail":374664924,"statfs":{"total":386547056640,"available":383656882176,"internally_reserved":0,"allocated":1126068224,"data_stored":1125348348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1764017713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,4,12,11,5,1,1,0,0,4,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18259968,"num_objects":-2229,"num_object_clones":0,"num_object_copies":-8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2229,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003366"},"pg_stats":[{"pgid":"1.f","version":"33'8287","reported_seq":8386,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.382475+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:06.382475+0000","last_peered":"2024-09-16T04:47:06.382475+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:06.382475+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8287,"log_dups_size":0,"ondisk_log_size":8287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35774464,"num_objects":4367,"num_object_clones":0,"num_object_copies":17468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8287,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2175}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8284","reported_seq":8365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.113122+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:06.113122+0000","last_peered":"2024-09-16T04:47:06.113122+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:06.113122+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8284,"log_dups_size":0,"ondisk_log_size":8284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35960076,"num_objects":4390,"num_object_clones":0,"num_object_copies":17560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8282,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8272","reported_seq":8360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:05.724970+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:05.724970+0000","last_peered":"2024-09-16T04:47:05.724970+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:05.724970+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8272,"log_dups_size":0,"ondisk_log_size":8272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36027038,"num_objects":4399,"num_object_clones":0,"num_object_copies":17596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8271,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2177}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8258","reported_seq":8340,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.061398+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:06.061398+0000","last_peered":"2024-09-16T04:47:06.061398+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:06.061398+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8258,"log_dups_size":0,"ondisk_log_size":8258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36140300,"num_objects":4412,"num_object_clones":0,"num_object_copies":17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2190}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8380","reported_seq":8500,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.354700+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:06.354700+0000","last_peered":"2024-09-16T04:47:06.354700+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:06.354700+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8380,"log_dups_size":0,"ondisk_log_size":8380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35722508,"num_objects":4361,"num_object_clones":0,"num_object_copies":17444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8378,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2109}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8086","reported_seq":8191,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.378221+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:06.378221+0000","last_peered":"2024-09-16T04:47:06.378221+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:06.378221+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8086,"log_dups_size":0,"ondisk_log_size":8086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34619392,"num_objects":4226,"num_object_clones":0,"num_object_copies":16904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2048}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8222","reported_seq":8307,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.263701+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:06.263701+0000","last_peered":"2024-09-16T04:47:06.263701+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:06.263701+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8222,"log_dups_size":0,"ondisk_log_size":8222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35861772,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8220,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2161}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8394","reported_seq":12951,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:07.729367+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:07.729367+0000","last_peered":"2024-09-16T04:47:07.729367+0000","last_clean":"2024-09-16T04:47:07.729367+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:07.729367+0000","last_undegraded":"2024-09-16T04:47:07.729367+0000","last_fullsized":"2024-09-16T04:47:07.729367+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:08.447562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35968268,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8392,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8482","reported_seq":11999,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.466550+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:06.466550+0000","last_peered":"2024-09-16T04:47:06.466550+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:06.466550+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8482,"log_dups_size":0,"ondisk_log_size":8482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35007000,"num_objects":4274,"num_object_clones":0,"num_object_copies":17096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8479,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1732,"num_bytes_recovered":14065664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3660}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8309","reported_seq":8409,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:05.706535+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:05.706535+0000","last_peered":"2024-09-16T04:47:05.706535+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:05.706535+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8309,"log_dups_size":0,"ondisk_log_size":8309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36443404,"num_objects":4449,"num_object_clones":0,"num_object_copies":17796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8307,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2259}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8201","reported_seq":8317,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.422835+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:06.422835+0000","last_peered":"2024-09-16T04:47:06.422835+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:06.422835+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8201,"log_dups_size":0,"ondisk_log_size":8201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35139474,"num_objects":4290,"num_object_clones":0,"num_object_copies":17160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8198,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8059","reported_seq":8140,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.475507+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:06.475507+0000","last_peered":"2024-09-16T04:47:06.475507+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:06.475507+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8059,"log_dups_size":0,"ondisk_log_size":8059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34994822,"num_objects":4272,"num_object_clones":0,"num_object_copies":17088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8058,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2164}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8189","reported_seq":8271,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:05.409252+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:05.409252+0000","last_peered":"2024-09-16T04:47:05.409252+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:05.409252+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8189,"log_dups_size":0,"ondisk_log_size":8189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35584646,"num_objects":4344,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8188,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2122}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8334","reported_seq":8457,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.426990+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:06.426990+0000","last_peered":"2024-09-16T04:47:06.426990+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:06.426990+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8334,"log_dups_size":0,"ondisk_log_size":8334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36435212,"num_objects":4448,"num_object_clones":0,"num_object_copies":17792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8332,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2248}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8088","reported_seq":8183,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:05.702089+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:05.702089+0000","last_peered":"2024-09-16T04:47:05.702089+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:05.702089+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8088,"log_dups_size":0,"ondisk_log_size":8088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35722508,"num_objects":4361,"num_object_clones":0,"num_object_copies":17444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2236}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8110","reported_seq":8198,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.228147+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:06.228147+0000","last_peered":"2024-09-16T04:47:06.228147+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:06.228147+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8110,"log_dups_size":0,"ondisk_log_size":8110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34379020,"num_objects":4197,"num_object_clones":0,"num_object_copies":16788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8108,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1969}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":569779904,"num_objects":69559,"num_object_clones":0,"num_object_copies":278236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69559,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131928,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4225,"num_bytes_recovered":32399360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123483648,"data_stored":1123483648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131955,"ondisk_log_size":131955,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751724,"kb_used_data":278556,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93620116,"statfs":{"total":96636764160,"available":95866998784,"internally_reserved":0,"allocated":285241344,"data_stored":285061375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,2,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743764,"kb_used_data":279428,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93628076,"statfs":{"total":96636764160,"available":95875149824,"internally_reserved":0,"allocated":286134272,"data_stored":285954303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,6,1,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587148,"kb_used_data":262524,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93784692,"statfs":{"total":96636764160,"available":96035524608,"internally_reserved":0,"allocated":268824576,"data_stored":268644607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332394191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739800,"kb_used_data":279168,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93632040,"statfs":{"total":96636764160,"available":95879208960,"internally_reserved":0,"allocated":285868032,"data_stored":285688063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":471658190},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,2,1,1,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":284917760,"data_stored":284917760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268525568,"data_stored":268525568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":285343744,"data_stored":285343744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":284696576,"data_stored":284696576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:08.914 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:09.522 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:09.522 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:09.561 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-09-16T04:47:08.679655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":569779904,"num_objects":69559,"num_object_clones":0,"num_object_copies":278236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69559,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131928,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4225,"num_bytes_recovered":32399360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131955,"ondisk_log_size":131955,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2822436,"kb_used_data":1099676,"kb_used_omap":14,"kb_used_meta":1722673,"kb_avail":374664924,"statfs":{"total":386547056640,"available":383656882176,"internally_reserved":0,"allocated":1126068224,"data_stored":1125348348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1764017713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,4,12,11,5,1,1,0,0,4,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18259968,"num_objects":-2229,"num_object_clones":0,"num_object_copies":-8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2229,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003366"},"pg_stats":[{"pgid":"1.f","version":"33'8287","reported_seq":8386,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.382475+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:06.382475+0000","last_peered":"2024-09-16T04:47:06.382475+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:06.382475+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8287,"log_dups_size":0,"ondisk_log_size":8287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35774464,"num_objects":4367,"num_object_clones":0,"num_object_copies":17468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8287,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2175}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8284","reported_seq":8365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.113122+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:06.113122+0000","last_peered":"2024-09-16T04:47:06.113122+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:06.113122+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8284,"log_dups_size":0,"ondisk_log_size":8284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35960076,"num_objects":4390,"num_object_clones":0,"num_object_copies":17560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8282,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8272","reported_seq":8360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:05.724970+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:05.724970+0000","last_peered":"2024-09-16T04:47:05.724970+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:05.724970+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8272,"log_dups_size":0,"ondisk_log_size":8272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36027038,"num_objects":4399,"num_object_clones":0,"num_object_copies":17596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8271,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2177}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8258","reported_seq":8340,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.061398+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:06.061398+0000","last_peered":"2024-09-16T04:47:06.061398+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:06.061398+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8258,"log_dups_size":0,"ondisk_log_size":8258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36140300,"num_objects":4412,"num_object_clones":0,"num_object_copies":17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2190}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8380","reported_seq":8500,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.354700+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:06.354700+0000","last_peered":"2024-09-16T04:47:06.354700+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:06.354700+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8380,"log_dups_size":0,"ondisk_log_size":8380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35722508,"num_objects":4361,"num_object_clones":0,"num_object_copies":17444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8378,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2109}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8086","reported_seq":8191,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.378221+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:06.378221+0000","last_peered":"2024-09-16T04:47:06.378221+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:06.378221+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8086,"log_dups_size":0,"ondisk_log_size":8086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34619392,"num_objects":4226,"num_object_clones":0,"num_object_copies":16904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2048}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8222","reported_seq":8307,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.263701+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:06.263701+0000","last_peered":"2024-09-16T04:47:06.263701+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:06.263701+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8222,"log_dups_size":0,"ondisk_log_size":8222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35861772,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8220,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2161}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8394","reported_seq":12951,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:07.729367+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:07.729367+0000","last_peered":"2024-09-16T04:47:07.729367+0000","last_clean":"2024-09-16T04:47:07.729367+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:07.729367+0000","last_undegraded":"2024-09-16T04:47:07.729367+0000","last_fullsized":"2024-09-16T04:47:07.729367+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:08.447562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35968268,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8392,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8482","reported_seq":11999,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.466550+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:06.466550+0000","last_peered":"2024-09-16T04:47:06.466550+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:06.466550+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8482,"log_dups_size":0,"ondisk_log_size":8482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35007000,"num_objects":4274,"num_object_clones":0,"num_object_copies":17096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8479,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1732,"num_bytes_recovered":14065664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3660}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8309","reported_seq":8409,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:05.706535+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:05.706535+0000","last_peered":"2024-09-16T04:47:05.706535+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:05.706535+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8309,"log_dups_size":0,"ondisk_log_size":8309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36443404,"num_objects":4449,"num_object_clones":0,"num_object_copies":17796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8307,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2259}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8201","reported_seq":8317,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.422835+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:06.422835+0000","last_peered":"2024-09-16T04:47:06.422835+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:06.422835+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8201,"log_dups_size":0,"ondisk_log_size":8201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35139474,"num_objects":4290,"num_object_clones":0,"num_object_copies":17160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8198,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8059","reported_seq":8140,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.475507+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:06.475507+0000","last_peered":"2024-09-16T04:47:06.475507+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:06.475507+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8059,"log_dups_size":0,"ondisk_log_size":8059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34994822,"num_objects":4272,"num_object_clones":0,"num_object_copies":17088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8058,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2164}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8189","reported_seq":8271,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:05.409252+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:05.409252+0000","last_peered":"2024-09-16T04:47:05.409252+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:05.409252+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8189,"log_dups_size":0,"ondisk_log_size":8189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35584646,"num_objects":4344,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8188,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2122}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8334","reported_seq":8457,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.426990+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:06.426990+0000","last_peered":"2024-09-16T04:47:06.426990+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:06.426990+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8334,"log_dups_size":0,"ondisk_log_size":8334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36435212,"num_objects":4448,"num_object_clones":0,"num_object_copies":17792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8332,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2248}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8088","reported_seq":8183,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:05.702089+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:05.702089+0000","last_peered":"2024-09-16T04:47:05.702089+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:05.702089+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8088,"log_dups_size":0,"ondisk_log_size":8088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35722508,"num_objects":4361,"num_object_clones":0,"num_object_copies":17444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2236}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8110","reported_seq":8198,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.228147+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:06.228147+0000","last_peered":"2024-09-16T04:47:06.228147+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:06.228147+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8110,"log_dups_size":0,"ondisk_log_size":8110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34379020,"num_objects":4197,"num_object_clones":0,"num_object_copies":16788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8108,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1969}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":569779904,"num_objects":69559,"num_object_clones":0,"num_object_copies":278236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69559,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131928,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4225,"num_bytes_recovered":32399360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123483648,"data_stored":1123483648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131955,"ondisk_log_size":131955,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751724,"kb_used_data":278556,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93620116,"statfs":{"total":96636764160,"available":95866998784,"internally_reserved":0,"allocated":285241344,"data_stored":285061375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,2,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743764,"kb_used_data":279428,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93628076,"statfs":{"total":96636764160,"available":95875149824,"internally_reserved":0,"allocated":286134272,"data_stored":285954303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,6,1,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587148,"kb_used_data":262524,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93784692,"statfs":{"total":96636764160,"available":96035524608,"internally_reserved":0,"allocated":268824576,"data_stored":268644607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332394191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739800,"kb_used_data":279168,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93632040,"statfs":{"total":96636764160,"available":95879208960,"internally_reserved":0,"allocated":285868032,"data_stored":285688063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":471658190},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,2,1,1,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":284917760,"data_stored":284917760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268525568,"data_stored":268525568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":285343744,"data_stored":285343744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":284696576,"data_stored":284696576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:09.562 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:47:09.562 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8287", 'reported_seq': 8386, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:06.382475+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:47:06.382475+0000', 'last_peered': '2024-09-16T04:47:06.382475+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:47:06.382475+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8287, 'log_dups_size': 0, 'ondisk_log_size': 8287, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35774464, 'num_objects': 4367, 'num_object_clones': 0, 'num_object_copies': 17468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8287, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2175}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:09.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:47:09.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8284", 'reported_seq': 8365, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:06.113122+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:47:06.113122+0000', 'last_peered': '2024-09-16T04:47:06.113122+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:47:06.113122+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8284, 'log_dups_size': 0, 'ondisk_log_size': 8284, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35960076, 'num_objects': 4390, 'num_object_clones': 0, 'num_object_copies': 17560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2206, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8282, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2186}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:09.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:47:09.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8272", 'reported_seq': 8360, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:05.724970+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:47:05.724970+0000', 'last_peered': '2024-09-16T04:47:05.724970+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:47:05.724970+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8272, 'log_dups_size': 0, 'ondisk_log_size': 8272, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36027038, 'num_objects': 4399, 'num_object_clones': 0, 'num_object_copies': 17596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4399, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8271, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2177}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:09.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:47:09.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8258", 'reported_seq': 8340, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:06.061398+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:47:06.061398+0000', 'last_peered': '2024-09-16T04:47:06.061398+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:47:06.061398+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8258, 'log_dups_size': 0, 'ondisk_log_size': 8258, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36140300, 'num_objects': 4412, 'num_object_clones': 0, 'num_object_copies': 17648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8256, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2190}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:09.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:47:09.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8380", 'reported_seq': 8500, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:06.354700+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:47:06.354700+0000', 'last_peered': '2024-09-16T04:47:06.354700+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:47:06.354700+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8380, 'log_dups_size': 0, 'ondisk_log_size': 8380, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35722508, 'num_objects': 4361, 'num_object_clones': 0, 'num_object_copies': 17444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2254, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4361, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8378, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2109}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:09.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:47:09.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8086", 'reported_seq': 8191, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:06.378221+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:47:06.378221+0000', 'last_peered': '2024-09-16T04:47:06.378221+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:47:06.378221+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8086, 'log_dups_size': 0, 'ondisk_log_size': 8086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34619392, 'num_objects': 4226, 'num_object_clones': 0, 'num_object_copies': 16904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4226, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8086, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2048}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:09.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:47:09.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8222", 'reported_seq': 8307, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:06.263701+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:47:06.263701+0000', 'last_peered': '2024-09-16T04:47:06.263701+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:47:06.263701+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8222, 'log_dups_size': 0, 'ondisk_log_size': 8222, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35861772, 'num_objects': 4378, 'num_object_clones': 0, 'num_object_copies': 17512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8220, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2161}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:09.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:47:09.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8482", 'reported_seq': 11999, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:06.466550+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:47:06.466550+0000', 'last_peered': '2024-09-16T04:47:06.466550+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:47:06.466550+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8482, 'log_dups_size': 0, 'ondisk_log_size': 8482, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35007000, 'num_objects': 4274, 'num_object_clones': 0, 'num_object_copies': 17096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 614, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8479, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1732, 'num_bytes_recovered': 14065664, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3660}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:09.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:47:09.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8309", 'reported_seq': 8409, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:05.706535+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:47:05.706535+0000', 'last_peered': '2024-09-16T04:47:05.706535+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:47:05.706535+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8309, 'log_dups_size': 0, 'ondisk_log_size': 8309, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36443404, 'num_objects': 4449, 'num_object_clones': 0, 'num_object_copies': 17796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8307, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2259}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:09.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:47:09.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8201", 'reported_seq': 8317, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:06.422835+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:47:06.422835+0000', 'last_peered': '2024-09-16T04:47:06.422835+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:47:06.422835+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8201, 'log_dups_size': 0, 'ondisk_log_size': 8201, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35139474, 'num_objects': 4290, 'num_object_clones': 0, 'num_object_copies': 17160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8198, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2129}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:09.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:47:09.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8059", 'reported_seq': 8140, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:06.475507+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:47:06.475507+0000', 'last_peered': '2024-09-16T04:47:06.475507+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:47:06.475507+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8059, 'log_dups_size': 0, 'ondisk_log_size': 8059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34994822, 'num_objects': 4272, 'num_object_clones': 0, 'num_object_copies': 17088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8058, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2164}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:09.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:47:09.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8189", 'reported_seq': 8271, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:05.409252+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:47:05.409252+0000', 'last_peered': '2024-09-16T04:47:05.409252+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:47:05.409252+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8189, 'log_dups_size': 0, 'ondisk_log_size': 8189, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35584646, 'num_objects': 4344, 'num_object_clones': 0, 'num_object_copies': 17376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4344, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8188, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2122}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:09.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:47:09.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8334", 'reported_seq': 8457, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:06.426990+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:47:06.426990+0000', 'last_peered': '2024-09-16T04:47:06.426990+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:47:06.426990+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8334, 'log_dups_size': 0, 'ondisk_log_size': 8334, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36435212, 'num_objects': 4448, 'num_object_clones': 0, 'num_object_copies': 17792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2202, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4448, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8332, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 36, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2248}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:09.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:47:09.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8088", 'reported_seq': 8183, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:05.702089+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:47:05.702089+0000', 'last_peered': '2024-09-16T04:47:05.702089+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:47:05.702089+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8088, 'log_dups_size': 0, 'ondisk_log_size': 8088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35722508, 'num_objects': 4361, 'num_object_clones': 0, 'num_object_copies': 17444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2127, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4361, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8086, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2236}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:09.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:47:09.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8110", 'reported_seq': 8198, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:06.228147+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:47:06.228147+0000', 'last_peered': '2024-09-16T04:47:06.228147+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:47:06.228147+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8110, 'log_dups_size': 0, 'ondisk_log_size': 8110, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34379020, 'num_objects': 4197, 'num_object_clones': 0, 'num_object_copies': 16788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4197, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8108, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1969}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:09.566 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:47:10.272 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:10.272 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31357/278236 objects degraded (11.270%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":442,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":69559,"data_bytes":569779904,"bytes_used":2890174464,"bytes_avail":383656882176,"bytes_total":386547056640,"degraded_objects":31357,"degraded_total":278236,"degraded_ratio":0.1126992912491554,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":87356,"recovering_keys_per_sec":0,"num_objects_recovered":156,"num_bytes_recovered":1048576,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":185},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:47:10.300 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:47:10.300 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:10.937 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:10.938 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:10.976 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-09-16T04:47:10.680183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":569779904,"num_objects":69559,"num_object_clones":0,"num_object_copies":278236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69559,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131928,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4225,"num_bytes_recovered":32399360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131955,"ondisk_log_size":131955,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2822436,"kb_used_data":1099676,"kb_used_omap":14,"kb_used_meta":1722673,"kb_avail":374664924,"statfs":{"total":386547056640,"available":383656882176,"internally_reserved":0,"allocated":1126068224,"data_stored":1125348348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1764017713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,4,12,11,5,1,1,0,0,4,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17743872,"num_objects":-2166,"num_object_clones":0,"num_object_copies":-8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2166,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003054"},"pg_stats":[{"pgid":"1.f","version":"33'8287","reported_seq":8386,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.382475+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:06.382475+0000","last_peered":"2024-09-16T04:47:06.382475+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:06.382475+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8287,"log_dups_size":0,"ondisk_log_size":8287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35774464,"num_objects":4367,"num_object_clones":0,"num_object_copies":17468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8287,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2175}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8284","reported_seq":8365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.113122+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:06.113122+0000","last_peered":"2024-09-16T04:47:06.113122+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:06.113122+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8284,"log_dups_size":0,"ondisk_log_size":8284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35960076,"num_objects":4390,"num_object_clones":0,"num_object_copies":17560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8282,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8272","reported_seq":8360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:05.724970+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:05.724970+0000","last_peered":"2024-09-16T04:47:05.724970+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:05.724970+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8272,"log_dups_size":0,"ondisk_log_size":8272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36027038,"num_objects":4399,"num_object_clones":0,"num_object_copies":17596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8271,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2177}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8258","reported_seq":8340,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.061398+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:06.061398+0000","last_peered":"2024-09-16T04:47:06.061398+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:06.061398+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8258,"log_dups_size":0,"ondisk_log_size":8258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36140300,"num_objects":4412,"num_object_clones":0,"num_object_copies":17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2190}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8380","reported_seq":8500,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.354700+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:06.354700+0000","last_peered":"2024-09-16T04:47:06.354700+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:06.354700+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8380,"log_dups_size":0,"ondisk_log_size":8380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35722508,"num_objects":4361,"num_object_clones":0,"num_object_copies":17444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8378,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2109}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8086","reported_seq":8191,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.378221+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:06.378221+0000","last_peered":"2024-09-16T04:47:06.378221+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:06.378221+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8086,"log_dups_size":0,"ondisk_log_size":8086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34619392,"num_objects":4226,"num_object_clones":0,"num_object_copies":16904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2048}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8222","reported_seq":8307,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.263701+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:06.263701+0000","last_peered":"2024-09-16T04:47:06.263701+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:06.263701+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8222,"log_dups_size":0,"ondisk_log_size":8222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35861772,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8220,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2161}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8394","reported_seq":12951,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:07.729367+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:07.729367+0000","last_peered":"2024-09-16T04:47:07.729367+0000","last_clean":"2024-09-16T04:47:07.729367+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:07.729367+0000","last_undegraded":"2024-09-16T04:47:07.729367+0000","last_fullsized":"2024-09-16T04:47:07.729367+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:08.447562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35968268,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8392,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8482","reported_seq":11999,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.466550+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:06.466550+0000","last_peered":"2024-09-16T04:47:06.466550+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:06.466550+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8482,"log_dups_size":0,"ondisk_log_size":8482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35007000,"num_objects":4274,"num_object_clones":0,"num_object_copies":17096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8479,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1732,"num_bytes_recovered":14065664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3660}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8309","reported_seq":8409,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:05.706535+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:05.706535+0000","last_peered":"2024-09-16T04:47:05.706535+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:05.706535+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8309,"log_dups_size":0,"ondisk_log_size":8309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36443404,"num_objects":4449,"num_object_clones":0,"num_object_copies":17796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8307,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2259}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8201","reported_seq":8317,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.422835+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:06.422835+0000","last_peered":"2024-09-16T04:47:06.422835+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:06.422835+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8201,"log_dups_size":0,"ondisk_log_size":8201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35139474,"num_objects":4290,"num_object_clones":0,"num_object_copies":17160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8198,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8059","reported_seq":8140,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.475507+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:06.475507+0000","last_peered":"2024-09-16T04:47:06.475507+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:06.475507+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8059,"log_dups_size":0,"ondisk_log_size":8059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34994822,"num_objects":4272,"num_object_clones":0,"num_object_copies":17088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8058,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2164}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8189","reported_seq":8271,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:05.409252+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:05.409252+0000","last_peered":"2024-09-16T04:47:05.409252+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:05.409252+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8189,"log_dups_size":0,"ondisk_log_size":8189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35584646,"num_objects":4344,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8188,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2122}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8334","reported_seq":8457,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.426990+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:06.426990+0000","last_peered":"2024-09-16T04:47:06.426990+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:06.426990+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8334,"log_dups_size":0,"ondisk_log_size":8334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36435212,"num_objects":4448,"num_object_clones":0,"num_object_copies":17792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8332,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2248}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8088","reported_seq":8183,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:05.702089+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:05.702089+0000","last_peered":"2024-09-16T04:47:05.702089+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:05.702089+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8088,"log_dups_size":0,"ondisk_log_size":8088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35722508,"num_objects":4361,"num_object_clones":0,"num_object_copies":17444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2236}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8110","reported_seq":8198,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:06.228147+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:06.228147+0000","last_peered":"2024-09-16T04:47:06.228147+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:06.228147+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8110,"log_dups_size":0,"ondisk_log_size":8110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34379020,"num_objects":4197,"num_object_clones":0,"num_object_copies":16788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8108,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1969}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":569779904,"num_objects":69559,"num_object_clones":0,"num_object_copies":278236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69559,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131928,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4225,"num_bytes_recovered":32399360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123483648,"data_stored":1123483648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131955,"ondisk_log_size":131955,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751724,"kb_used_data":278556,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93620116,"statfs":{"total":96636764160,"available":95866998784,"internally_reserved":0,"allocated":285241344,"data_stored":285061375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,2,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743764,"kb_used_data":279428,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93628076,"statfs":{"total":96636764160,"available":95875149824,"internally_reserved":0,"allocated":286134272,"data_stored":285954303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,6,1,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587148,"kb_used_data":262524,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93784692,"statfs":{"total":96636764160,"available":96035524608,"internally_reserved":0,"allocated":268824576,"data_stored":268644607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332394191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739800,"kb_used_data":279168,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93632040,"statfs":{"total":96636764160,"available":95879208960,"internally_reserved":0,"allocated":285868032,"data_stored":285688063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":471658190},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,2,1,1,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":284917760,"data_stored":284917760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268525568,"data_stored":268525568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":285343744,"data_stored":285343744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":284696576,"data_stored":284696576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:13.979 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:14.577 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:14.577 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:14.606 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-16T04:47:12.680644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":561407680,"num_objects":68537,"num_object_clones":0,"num_object_copies":274148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68537,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132950,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4290,"num_bytes_recovered":32890880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132977,"ondisk_log_size":132977,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805012,"kb_used_data":1086420,"kb_used_omap":14,"kb_used_meta":1718513,"kb_avail":374682348,"statfs":{"total":386547056640,"available":383674724352,"internally_reserved":0,"allocated":1112494080,"data_stored":1111774204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1759757873},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,9,8,4,5,3,0,0,0,3,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26116096,"num_objects":-3188,"num_object_clones":0,"num_object_copies":-12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3188,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003027"},"pg_stats":[{"pgid":"1.f","version":"33'8353","reported_seq":8452,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.378851+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:11.378851+0000","last_peered":"2024-09-16T04:47:11.378851+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:11.378851+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8353,"log_dups_size":0,"ondisk_log_size":8353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35233792,"num_objects":4301,"num_object_clones":0,"num_object_copies":17204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8353,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2109}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8345","reported_seq":8426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.292215+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:11.292215+0000","last_peered":"2024-09-16T04:47:11.292215+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:11.292215+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8345,"log_dups_size":0,"ondisk_log_size":8345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35460364,"num_objects":4329,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8343,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2125}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8338","reported_seq":8426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:10.732533+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:10.732533+0000","last_peered":"2024-09-16T04:47:10.732533+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:10.732533+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8338,"log_dups_size":0,"ondisk_log_size":8338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35486366,"num_objects":4333,"num_object_clones":0,"num_object_copies":17332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8337,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8326","reported_seq":8408,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.064660+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:11.064660+0000","last_peered":"2024-09-16T04:47:11.064660+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:11.064660+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35583244,"num_objects":4344,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8324,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2122}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8453","reported_seq":8573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.321092+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:11.321092+0000","last_peered":"2024-09-16T04:47:11.321092+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:11.321092+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8453,"log_dups_size":0,"ondisk_log_size":8453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35124492,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8451,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8155","reported_seq":8259,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.521179+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:11.521179+0000","last_peered":"2024-09-16T04:47:11.521179+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:11.521179+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8155,"log_dups_size":0,"ondisk_log_size":8155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34054144,"num_objects":4157,"num_object_clones":0,"num_object_copies":16628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8155,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8296","reported_seq":8383,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.227215+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:11.227215+0000","last_peered":"2024-09-16T04:47:11.227215+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:11.227215+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8296,"log_dups_size":0,"ondisk_log_size":8296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35255564,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8294,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2088}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8394","reported_seq":12951,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:07.729367+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:07.729367+0000","last_peered":"2024-09-16T04:47:07.729367+0000","last_clean":"2024-09-16T04:47:07.729367+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:07.729367+0000","last_undegraded":"2024-09-16T04:47:07.729367+0000","last_fullsized":"2024-09-16T04:47:07.729367+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:08.447562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35968268,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8392,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8546","reported_seq":12184,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.430819+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:11.430819+0000","last_peered":"2024-09-16T04:47:11.430819+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:11.430819+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8546,"log_dups_size":0,"ondisk_log_size":8546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34482712,"num_objects":4210,"num_object_clones":0,"num_object_copies":16840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8543,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1792,"num_bytes_recovered":14557184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3656}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8375","reported_seq":8475,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:10.621032+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:10.621032+0000","last_peered":"2024-09-16T04:47:10.621032+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:10.621032+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8375,"log_dups_size":0,"ondisk_log_size":8375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35902732,"num_objects":4383,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8373,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2193}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8263","reported_seq":8385,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.480931+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:11.480931+0000","last_peered":"2024-09-16T04:47:11.480931+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:11.480931+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8263,"log_dups_size":0,"ondisk_log_size":8263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34631570,"num_objects":4228,"num_object_clones":0,"num_object_copies":16912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8260,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2070}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8120","reported_seq":8201,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.524538+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:11.524538+0000","last_peered":"2024-09-16T04:47:11.524538+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:11.524538+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8120,"log_dups_size":0,"ondisk_log_size":8120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34495110,"num_objects":4211,"num_object_clones":0,"num_object_copies":16844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8119,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2103}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8269","reported_seq":8351,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:10.635104+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:10.635104+0000","last_peered":"2024-09-16T04:47:10.635104+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:10.635104+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8269,"log_dups_size":0,"ondisk_log_size":8269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34929286,"num_objects":4264,"num_object_clones":0,"num_object_copies":17056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8268,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2042}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8405","reported_seq":8530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.491171+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:11.491171+0000","last_peered":"2024-09-16T04:47:11.491171+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:11.491171+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8405,"log_dups_size":0,"ondisk_log_size":8405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35853580,"num_objects":4377,"num_object_clones":0,"num_object_copies":17508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8403,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2178}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8156","reported_seq":8251,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:10.770909+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:10.770909+0000","last_peered":"2024-09-16T04:47:10.770909+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:10.770909+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8156,"log_dups_size":0,"ondisk_log_size":8156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35165452,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2168}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8183","reported_seq":8272,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.078910+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:11.078910+0000","last_peered":"2024-09-16T04:47:11.078910+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:11.078910+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8183,"log_dups_size":0,"ondisk_log_size":8183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33781004,"num_objects":4124,"num_object_clones":0,"num_object_copies":16496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8181,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":561407680,"num_objects":68537,"num_object_clones":0,"num_object_copies":274148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68537,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132950,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4290,"num_bytes_recovered":32890880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1110065152,"data_stored":1110065152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132977,"ondisk_log_size":132977,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734740,"kb_used_data":274052,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93637100,"statfs":{"total":96636764160,"available":95884390400,"internally_reserved":0,"allocated":280629248,"data_stored":280449279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471723727},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,0,1,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743532,"kb_used_data":275036,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93628308,"statfs":{"total":96636764160,"available":95875387392,"internally_reserved":0,"allocated":281636864,"data_stored":281456895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1,1,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587148,"kb_used_data":262524,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93784692,"statfs":{"total":96636764160,"available":96035524608,"internally_reserved":0,"allocated":268824576,"data_stored":268644607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332394191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739592,"kb_used_data":274808,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93632248,"statfs":{"total":96636764160,"available":95879421952,"internally_reserved":0,"allocated":281403392,"data_stored":281223423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":475918030},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280457216,"data_stored":280457216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268525568,"data_stored":268525568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280850432,"data_stored":280850432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280231936,"data_stored":280231936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:14.607 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:15.218 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:15.219 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:15.247 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-16T04:47:14.681184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":560858816,"num_objects":68470,"num_object_clones":0,"num_object_copies":273880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68470,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133017,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4290,"num_bytes_recovered":32890880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133044,"ondisk_log_size":133044,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2808928,"kb_used_data":1086176,"kb_used_omap":14,"kb_used_meta":1722673,"kb_avail":374678432,"statfs":{"total":386547056640,"available":383670714368,"internally_reserved":0,"allocated":1112244224,"data_stored":1111524348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1764017713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,9,10,3,6,3,0,0,0,3,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18325504,"num_objects":-2237,"num_object_clones":0,"num_object_copies":-8948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2237,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003076"},"pg_stats":[{"pgid":"1.f","version":"33'8353","reported_seq":8452,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.378851+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:11.378851+0000","last_peered":"2024-09-16T04:47:11.378851+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:11.378851+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8353,"log_dups_size":0,"ondisk_log_size":8353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35233792,"num_objects":4301,"num_object_clones":0,"num_object_copies":17204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8353,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2109}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8345","reported_seq":8426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.292215+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:11.292215+0000","last_peered":"2024-09-16T04:47:11.292215+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:11.292215+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8345,"log_dups_size":0,"ondisk_log_size":8345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35460364,"num_objects":4329,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8343,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2125}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8338","reported_seq":8426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:10.732533+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:10.732533+0000","last_peered":"2024-09-16T04:47:10.732533+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:10.732533+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8338,"log_dups_size":0,"ondisk_log_size":8338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35486366,"num_objects":4333,"num_object_clones":0,"num_object_copies":17332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8337,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8326","reported_seq":8408,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.064660+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:11.064660+0000","last_peered":"2024-09-16T04:47:11.064660+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:11.064660+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35583244,"num_objects":4344,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8324,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2122}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8453","reported_seq":8573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.321092+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:11.321092+0000","last_peered":"2024-09-16T04:47:11.321092+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:11.321092+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8453,"log_dups_size":0,"ondisk_log_size":8453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35124492,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8451,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8155","reported_seq":8259,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.521179+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:11.521179+0000","last_peered":"2024-09-16T04:47:11.521179+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:11.521179+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8155,"log_dups_size":0,"ondisk_log_size":8155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34054144,"num_objects":4157,"num_object_clones":0,"num_object_copies":16628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8155,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8296","reported_seq":8383,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.227215+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:11.227215+0000","last_peered":"2024-09-16T04:47:11.227215+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:11.227215+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8296,"log_dups_size":0,"ondisk_log_size":8296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35255564,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8294,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2088}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8461","reported_seq":13018,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.098003+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:13.098003+0000","last_peered":"2024-09-16T04:47:13.098003+0000","last_clean":"2024-09-16T04:47:13.098003+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:13.098003+0000","last_undegraded":"2024-09-16T04:47:13.098003+0000","last_fullsized":"2024-09-16T04:47:13.098003+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8461,"log_dups_size":0,"ondisk_log_size":8461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:13.577331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35419404,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8459,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8546","reported_seq":12184,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.430819+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:11.430819+0000","last_peered":"2024-09-16T04:47:11.430819+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:11.430819+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8546,"log_dups_size":0,"ondisk_log_size":8546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34482712,"num_objects":4210,"num_object_clones":0,"num_object_copies":16840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8543,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1792,"num_bytes_recovered":14557184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3656}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8375","reported_seq":8475,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:10.621032+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:10.621032+0000","last_peered":"2024-09-16T04:47:10.621032+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:10.621032+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8375,"log_dups_size":0,"ondisk_log_size":8375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35902732,"num_objects":4383,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8373,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2193}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8263","reported_seq":8385,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.480931+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:11.480931+0000","last_peered":"2024-09-16T04:47:11.480931+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:11.480931+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8263,"log_dups_size":0,"ondisk_log_size":8263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34631570,"num_objects":4228,"num_object_clones":0,"num_object_copies":16912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8260,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2070}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8120","reported_seq":8201,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.524538+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:11.524538+0000","last_peered":"2024-09-16T04:47:11.524538+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:11.524538+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8120,"log_dups_size":0,"ondisk_log_size":8120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34495110,"num_objects":4211,"num_object_clones":0,"num_object_copies":16844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8119,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2103}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8269","reported_seq":8351,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:10.635104+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:10.635104+0000","last_peered":"2024-09-16T04:47:10.635104+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:10.635104+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8269,"log_dups_size":0,"ondisk_log_size":8269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34929286,"num_objects":4264,"num_object_clones":0,"num_object_copies":17056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8268,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2042}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8405","reported_seq":8530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.491171+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:11.491171+0000","last_peered":"2024-09-16T04:47:11.491171+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:11.491171+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8405,"log_dups_size":0,"ondisk_log_size":8405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35853580,"num_objects":4377,"num_object_clones":0,"num_object_copies":17508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8403,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2178}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8156","reported_seq":8251,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:10.770909+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:10.770909+0000","last_peered":"2024-09-16T04:47:10.770909+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:10.770909+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8156,"log_dups_size":0,"ondisk_log_size":8156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35165452,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2168}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8183","reported_seq":8272,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.078910+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:11.078910+0000","last_peered":"2024-09-16T04:47:11.078910+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:11.078910+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8183,"log_dups_size":0,"ondisk_log_size":8183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33781004,"num_objects":4124,"num_object_clones":0,"num_object_copies":16496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8181,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":560858816,"num_objects":68470,"num_object_clones":0,"num_object_copies":273880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68470,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133017,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4290,"num_bytes_recovered":32890880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1109807104,"data_stored":1109807104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133044,"ondisk_log_size":133044,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734740,"kb_used_data":274052,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93637100,"statfs":{"total":96636764160,"available":95884390400,"internally_reserved":0,"allocated":280629248,"data_stored":280449279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471723727},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,0,1,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743532,"kb_used_data":275036,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93628308,"statfs":{"total":96636764160,"available":95875387392,"internally_reserved":0,"allocated":281636864,"data_stored":281456895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1,1,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051654,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591064,"kb_used_data":262280,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93780776,"statfs":{"total":96636764160,"available":96031514624,"internally_reserved":0,"allocated":268574720,"data_stored":268394751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336654031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739592,"kb_used_data":274808,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93632248,"statfs":{"total":96636764160,"available":95879421952,"internally_reserved":0,"allocated":281403392,"data_stored":281223423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":475918030},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280457216,"data_stored":280457216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268267520,"data_stored":268267520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280850432,"data_stored":280850432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280231936,"data_stored":280231936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:15.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:47:15.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8353", 'reported_seq': 8452, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:11.378851+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:47:11.378851+0000', 'last_peered': '2024-09-16T04:47:11.378851+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:47:11.378851+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8353, 'log_dups_size': 0, 'ondisk_log_size': 8353, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35233792, 'num_objects': 4301, 'num_object_clones': 0, 'num_object_copies': 17204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8353, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2109}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:15.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:47:15.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8345", 'reported_seq': 8426, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:11.292215+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:47:11.292215+0000', 'last_peered': '2024-09-16T04:47:11.292215+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:47:11.292215+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8345, 'log_dups_size': 0, 'ondisk_log_size': 8345, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35460364, 'num_objects': 4329, 'num_object_clones': 0, 'num_object_copies': 17316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2206, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4329, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8343, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2125}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:15.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:47:15.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8338", 'reported_seq': 8426, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:10.732533+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:47:10.732533+0000', 'last_peered': '2024-09-16T04:47:10.732533+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:47:10.732533+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8338, 'log_dups_size': 0, 'ondisk_log_size': 8338, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35486366, 'num_objects': 4333, 'num_object_clones': 0, 'num_object_copies': 17332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4333, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8337, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2111}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:15.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:47:15.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8326", 'reported_seq': 8408, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:11.064660+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:47:11.064660+0000', 'last_peered': '2024-09-16T04:47:11.064660+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:47:11.064660+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8326, 'log_dups_size': 0, 'ondisk_log_size': 8326, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35583244, 'num_objects': 4344, 'num_object_clones': 0, 'num_object_copies': 17376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4344, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8324, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2122}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:15.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:47:15.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8453", 'reported_seq': 8573, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:11.321092+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:47:11.321092+0000', 'last_peered': '2024-09-16T04:47:11.321092+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:47:11.321092+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8453, 'log_dups_size': 0, 'ondisk_log_size': 8453, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35124492, 'num_objects': 4288, 'num_object_clones': 0, 'num_object_copies': 17152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2254, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8451, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2036}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:15.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:47:15.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8155", 'reported_seq': 8259, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:11.521179+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:47:11.521179+0000', 'last_peered': '2024-09-16T04:47:11.521179+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:47:11.521179+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8155, 'log_dups_size': 0, 'ondisk_log_size': 8155, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34054144, 'num_objects': 4157, 'num_object_clones': 0, 'num_object_copies': 16628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8155, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1979}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:15.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:47:15.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8296", 'reported_seq': 8383, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:11.227215+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:47:11.227215+0000', 'last_peered': '2024-09-16T04:47:11.227215+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:47:11.227215+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8296, 'log_dups_size': 0, 'ondisk_log_size': 8296, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35255564, 'num_objects': 4304, 'num_object_clones': 0, 'num_object_copies': 17216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2218, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4304, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8294, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2088}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:15.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:47:15.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8546", 'reported_seq': 12184, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:11.430819+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:47:11.430819+0000', 'last_peered': '2024-09-16T04:47:11.430819+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:47:11.430819+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8546, 'log_dups_size': 0, 'ondisk_log_size': 8546, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34482712, 'num_objects': 4210, 'num_object_clones': 0, 'num_object_copies': 16840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 554, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8543, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1792, 'num_bytes_recovered': 14557184, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3656}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:15.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:47:15.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8375", 'reported_seq': 8475, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:10.621032+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:47:10.621032+0000', 'last_peered': '2024-09-16T04:47:10.621032+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:47:10.621032+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8375, 'log_dups_size': 0, 'ondisk_log_size': 8375, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35902732, 'num_objects': 4383, 'num_object_clones': 0, 'num_object_copies': 17532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8373, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2193}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:15.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:47:15.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8263", 'reported_seq': 8385, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:11.480931+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:47:11.480931+0000', 'last_peered': '2024-09-16T04:47:11.480931+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:47:11.480931+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8263, 'log_dups_size': 0, 'ondisk_log_size': 8263, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34631570, 'num_objects': 4228, 'num_object_clones': 0, 'num_object_copies': 16912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2161, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4228, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8260, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2070}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:15.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:47:15.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8120", 'reported_seq': 8201, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:11.524538+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:47:11.524538+0000', 'last_peered': '2024-09-16T04:47:11.524538+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:47:11.524538+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8120, 'log_dups_size': 0, 'ondisk_log_size': 8120, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34495110, 'num_objects': 4211, 'num_object_clones': 0, 'num_object_copies': 16844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4211, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8119, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2103}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:15.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:47:15.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8269", 'reported_seq': 8351, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:10.635104+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:47:10.635104+0000', 'last_peered': '2024-09-16T04:47:10.635104+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:47:10.635104+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8269, 'log_dups_size': 0, 'ondisk_log_size': 8269, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34929286, 'num_objects': 4264, 'num_object_clones': 0, 'num_object_copies': 17056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8268, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2042}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:15.251 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:47:15.251 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8405", 'reported_seq': 8530, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:11.491171+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:47:11.491171+0000', 'last_peered': '2024-09-16T04:47:11.491171+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:47:11.491171+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8405, 'log_dups_size': 0, 'ondisk_log_size': 8405, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35853580, 'num_objects': 4377, 'num_object_clones': 0, 'num_object_copies': 17508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2201, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8403, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2178}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:15.251 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:47:15.251 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8156", 'reported_seq': 8251, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:10.770909+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:47:10.770909+0000', 'last_peered': '2024-09-16T04:47:10.770909+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:47:10.770909+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8156, 'log_dups_size': 0, 'ondisk_log_size': 8156, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35165452, 'num_objects': 4293, 'num_object_clones': 0, 'num_object_copies': 17172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2127, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8154, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2168}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:15.251 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:47:15.251 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8183", 'reported_seq': 8272, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:11.078910+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:47:11.078910+0000', 'last_peered': '2024-09-16T04:47:11.078910+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:47:11.078910+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8183, 'log_dups_size': 0, 'ondisk_log_size': 8183, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33781004, 'num_objects': 4124, 'num_object_clones': 0, 'num_object_copies': 16496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8181, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1896}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:15.251 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:47:15.990 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:15.990 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31292/273880 objects degraded (11.425%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":448,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":68470,"data_bytes":560858816,"bytes_used":2876342272,"bytes_avail":383670714368,"bytes_total":386547056640,"degraded_objects":31292,"degraded_total":273880,"degraded_ratio":0.11425441799328173,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":83946,"recovering_keys_per_sec":0,"num_objects_recovered":146,"num_bytes_recovered":1007616,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":186},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:47:16.021 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:47:16.021 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:16.621 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:16.622 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:16.645 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-16T04:47:14.681184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":560858816,"num_objects":68470,"num_object_clones":0,"num_object_copies":273880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68470,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133017,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4290,"num_bytes_recovered":32890880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133044,"ondisk_log_size":133044,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2808928,"kb_used_data":1086176,"kb_used_omap":14,"kb_used_meta":1722673,"kb_avail":374678432,"statfs":{"total":386547056640,"available":383670714368,"internally_reserved":0,"allocated":1112244224,"data_stored":1111524348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1764017713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,9,10,3,6,3,0,0,0,3,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18325504,"num_objects":-2237,"num_object_clones":0,"num_object_copies":-8948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2237,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003076"},"pg_stats":[{"pgid":"1.f","version":"33'8353","reported_seq":8452,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.378851+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:11.378851+0000","last_peered":"2024-09-16T04:47:11.378851+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:11.378851+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8353,"log_dups_size":0,"ondisk_log_size":8353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35233792,"num_objects":4301,"num_object_clones":0,"num_object_copies":17204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8353,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2109}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8345","reported_seq":8426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.292215+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:11.292215+0000","last_peered":"2024-09-16T04:47:11.292215+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:11.292215+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8345,"log_dups_size":0,"ondisk_log_size":8345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35460364,"num_objects":4329,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8343,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2125}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8338","reported_seq":8426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:10.732533+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:10.732533+0000","last_peered":"2024-09-16T04:47:10.732533+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:10.732533+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8338,"log_dups_size":0,"ondisk_log_size":8338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35486366,"num_objects":4333,"num_object_clones":0,"num_object_copies":17332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8337,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8326","reported_seq":8408,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.064660+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:11.064660+0000","last_peered":"2024-09-16T04:47:11.064660+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:11.064660+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35583244,"num_objects":4344,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8324,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2122}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8453","reported_seq":8573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.321092+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:11.321092+0000","last_peered":"2024-09-16T04:47:11.321092+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:11.321092+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8453,"log_dups_size":0,"ondisk_log_size":8453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35124492,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8451,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8155","reported_seq":8259,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.521179+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:11.521179+0000","last_peered":"2024-09-16T04:47:11.521179+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:11.521179+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8155,"log_dups_size":0,"ondisk_log_size":8155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34054144,"num_objects":4157,"num_object_clones":0,"num_object_copies":16628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8155,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8296","reported_seq":8383,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.227215+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:11.227215+0000","last_peered":"2024-09-16T04:47:11.227215+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:11.227215+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8296,"log_dups_size":0,"ondisk_log_size":8296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35255564,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8294,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2088}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8461","reported_seq":13018,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.098003+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:13.098003+0000","last_peered":"2024-09-16T04:47:13.098003+0000","last_clean":"2024-09-16T04:47:13.098003+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:13.098003+0000","last_undegraded":"2024-09-16T04:47:13.098003+0000","last_fullsized":"2024-09-16T04:47:13.098003+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8461,"log_dups_size":0,"ondisk_log_size":8461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:13.577331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35419404,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8459,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8546","reported_seq":12184,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.430819+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:11.430819+0000","last_peered":"2024-09-16T04:47:11.430819+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:11.430819+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8546,"log_dups_size":0,"ondisk_log_size":8546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34482712,"num_objects":4210,"num_object_clones":0,"num_object_copies":16840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8543,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1792,"num_bytes_recovered":14557184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3656}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8375","reported_seq":8475,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:10.621032+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:10.621032+0000","last_peered":"2024-09-16T04:47:10.621032+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:10.621032+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8375,"log_dups_size":0,"ondisk_log_size":8375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35902732,"num_objects":4383,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8373,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2193}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8263","reported_seq":8385,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.480931+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:11.480931+0000","last_peered":"2024-09-16T04:47:11.480931+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:11.480931+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8263,"log_dups_size":0,"ondisk_log_size":8263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34631570,"num_objects":4228,"num_object_clones":0,"num_object_copies":16912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8260,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2070}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8120","reported_seq":8201,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.524538+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:11.524538+0000","last_peered":"2024-09-16T04:47:11.524538+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:11.524538+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8120,"log_dups_size":0,"ondisk_log_size":8120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34495110,"num_objects":4211,"num_object_clones":0,"num_object_copies":16844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8119,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2103}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8269","reported_seq":8351,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:10.635104+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:10.635104+0000","last_peered":"2024-09-16T04:47:10.635104+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:10.635104+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8269,"log_dups_size":0,"ondisk_log_size":8269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34929286,"num_objects":4264,"num_object_clones":0,"num_object_copies":17056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8268,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2042}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8405","reported_seq":8530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.491171+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:11.491171+0000","last_peered":"2024-09-16T04:47:11.491171+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:11.491171+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8405,"log_dups_size":0,"ondisk_log_size":8405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35853580,"num_objects":4377,"num_object_clones":0,"num_object_copies":17508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8403,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2178}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8156","reported_seq":8251,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:10.770909+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:10.770909+0000","last_peered":"2024-09-16T04:47:10.770909+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:10.770909+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8156,"log_dups_size":0,"ondisk_log_size":8156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35165452,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2168}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8183","reported_seq":8272,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:11.078910+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:11.078910+0000","last_peered":"2024-09-16T04:47:11.078910+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:11.078910+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8183,"log_dups_size":0,"ondisk_log_size":8183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33781004,"num_objects":4124,"num_object_clones":0,"num_object_copies":16496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8181,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":560858816,"num_objects":68470,"num_object_clones":0,"num_object_copies":273880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68470,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133017,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4290,"num_bytes_recovered":32890880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1109807104,"data_stored":1109807104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133044,"ondisk_log_size":133044,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734740,"kb_used_data":274052,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93637100,"statfs":{"total":96636764160,"available":95884390400,"internally_reserved":0,"allocated":280629248,"data_stored":280449279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471723727},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,0,1,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743532,"kb_used_data":275036,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93628308,"statfs":{"total":96636764160,"available":95875387392,"internally_reserved":0,"allocated":281636864,"data_stored":281456895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1,1,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051654,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591064,"kb_used_data":262280,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93780776,"statfs":{"total":96636764160,"available":96031514624,"internally_reserved":0,"allocated":268574720,"data_stored":268394751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336654031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739592,"kb_used_data":274808,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93632248,"statfs":{"total":96636764160,"available":95879421952,"internally_reserved":0,"allocated":281403392,"data_stored":281223423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":475918030},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280457216,"data_stored":280457216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268267520,"data_stored":268267520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280850432,"data_stored":280850432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280231936,"data_stored":280231936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:19.647 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:20.221 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:20.222 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:20.245 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-16T04:47:18.682248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":551888576,"num_objects":67375,"num_object_clones":0,"num_object_copies":269500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67375,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134112,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":33374208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134139,"ondisk_log_size":134139,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803756,"kb_used_data":1072672,"kb_used_omap":14,"kb_used_meta":1730993,"kb_avail":374683604,"statfs":{"total":386547056640,"available":383676010496,"internally_reserved":0,"allocated":1098416128,"data_stored":1097696252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1772537393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,3,4,8,5,6,4,0,0,0,0,2,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18341888,"num_objects":-2239,"num_object_clones":0,"num_object_copies":-8956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2239,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003152"},"pg_stats":[{"pgid":"1.f","version":"33'8421","reported_seq":8530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.559337+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:16.559337+0000","last_peered":"2024-09-16T04:47:16.559337+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:16.559337+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8421,"log_dups_size":0,"ondisk_log_size":8421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34676736,"num_objects":4233,"num_object_clones":0,"num_object_copies":16932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8421,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2046}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8418","reported_seq":8499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.279791+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:16.279791+0000","last_peered":"2024-09-16T04:47:16.279791+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:16.279791+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8418,"log_dups_size":0,"ondisk_log_size":8418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34862348,"num_objects":4256,"num_object_clones":0,"num_object_copies":17024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8416,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2052}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8419","reported_seq":8507,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:15.710199+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:15.710199+0000","last_peered":"2024-09-16T04:47:15.710199+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:15.710199+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8419,"log_dups_size":0,"ondisk_log_size":8419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34822814,"num_objects":4252,"num_object_clones":0,"num_object_copies":17008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8418,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2032}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8401","reported_seq":8481,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.108762+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:16.108762+0000","last_peered":"2024-09-16T04:47:16.108762+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:16.108762+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8401,"log_dups_size":0,"ondisk_log_size":8401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34968844,"num_objects":4269,"num_object_clones":0,"num_object_copies":17076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8399,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8521","reported_seq":8644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.510861+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:16.510861+0000","last_peered":"2024-09-16T04:47:16.510861+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:16.510861+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8521,"log_dups_size":0,"ondisk_log_size":8521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34567436,"num_objects":4220,"num_object_clones":0,"num_object_copies":16880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8519,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8231","reported_seq":8336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.539172+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:16.539172+0000","last_peered":"2024-09-16T04:47:16.539172+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:16.539172+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8231,"log_dups_size":0,"ondisk_log_size":8231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33431552,"num_objects":4081,"num_object_clones":0,"num_object_copies":16324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8231,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1903}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8373","reported_seq":8460,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:15.968024+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:15.968024+0000","last_peered":"2024-09-16T04:47:15.968024+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:15.968024+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8373,"log_dups_size":0,"ondisk_log_size":8373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34624780,"num_objects":4227,"num_object_clones":0,"num_object_copies":16908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8371,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2011}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8529","reported_seq":13086,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:17.917733+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:17.917733+0000","last_peered":"2024-09-16T04:47:17.917733+0000","last_clean":"2024-09-16T04:47:17.917733+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:17.917733+0000","last_undegraded":"2024-09-16T04:47:17.917733+0000","last_fullsized":"2024-09-16T04:47:17.917733+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8529,"log_dups_size":0,"ondisk_log_size":8529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:18.655748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34862348,"num_objects":4256,"num_object_clones":0,"num_object_copies":17024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8527,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8612","reported_seq":12366,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.627200+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:16.627200+0000","last_peered":"2024-09-16T04:47:16.627200+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:16.627200+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8612,"log_dups_size":0,"ondisk_log_size":8612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33942040,"num_objects":4144,"num_object_clones":0,"num_object_copies":16576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8609,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1851,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8439","reported_seq":8539,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:15.764863+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:15.764863+0000","last_peered":"2024-09-16T04:47:15.764863+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:15.764863+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8439,"log_dups_size":0,"ondisk_log_size":8439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35378444,"num_objects":4319,"num_object_clones":0,"num_object_copies":17276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8437,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2129}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8315","reported_seq":8441,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.154074+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:16.154074+0000","last_peered":"2024-09-16T04:47:16.154074+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:16.154074+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8315,"log_dups_size":0,"ondisk_log_size":8315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34205586,"num_objects":4176,"num_object_clones":0,"num_object_copies":16704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8312,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8174","reported_seq":8259,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.535466+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:16.535466+0000","last_peered":"2024-09-16T04:47:16.535466+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:16.535466+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8174,"log_dups_size":0,"ondisk_log_size":8174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34052742,"num_objects":4157,"num_object_clones":0,"num_object_copies":16628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8173,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2051}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8342","reported_seq":8426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:15.784089+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:15.784089+0000","last_peered":"2024-09-16T04:47:15.784089+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:15.784089+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8342,"log_dups_size":0,"ondisk_log_size":8342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34331270,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8341,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1970}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8475","reported_seq":8602,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.476871+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:16.476871+0000","last_peered":"2024-09-16T04:47:16.476871+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:16.476871+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8475,"log_dups_size":0,"ondisk_log_size":8475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35280140,"num_objects":4307,"num_object_clones":0,"num_object_copies":17228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8473,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2109}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8223","reported_seq":8320,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:15.720824+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:15.720824+0000","last_peered":"2024-09-16T04:47:15.720824+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:15.720824+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8223,"log_dups_size":0,"ondisk_log_size":8223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34616588,"num_objects":4226,"num_object_clones":0,"num_object_copies":16904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8221,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2102}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8246","reported_seq":8335,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.306127+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:16.306127+0000","last_peered":"2024-09-16T04:47:16.306127+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:16.306127+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8246,"log_dups_size":0,"ondisk_log_size":8246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264908,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1833}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":551888576,"num_objects":67375,"num_object_clones":0,"num_object_copies":269500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67375,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134112,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":33374208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1096019968,"data_stored":1096019968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134139,"ondisk_log_size":134139,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738616,"kb_used_data":269600,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93633224,"statfs":{"total":96636764160,"available":95880421376,"internally_reserved":0,"allocated":276070400,"data_stored":275890431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,3,1,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747468,"kb_used_data":270648,"kb_used_omap":1,"kb_used_meta":476798,"kb_avail":93624372,"statfs":{"total":96636764160,"available":95871356928,"internally_reserved":0,"allocated":277143552,"data_stored":276963583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":488241605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,0,1,0,2,3,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578288,"kb_used_data":261984,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93793552,"statfs":{"total":96636764160,"available":96044597248,"internally_reserved":0,"allocated":268271616,"data_stored":268091647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":323874511},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739384,"kb_used_data":270440,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93632456,"statfs":{"total":96636764160,"available":95879634944,"internally_reserved":0,"allocated":276930560,"data_stored":276750591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":480177870},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":275968000,"data_stored":275968000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267997184,"data_stored":267997184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":276344832,"data_stored":276344832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":275709952,"data_stored":275709952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:20.247 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:20.850 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:20.850 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:20.880 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-16T04:47:20.682735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":551888576,"num_objects":67375,"num_object_clones":0,"num_object_copies":269500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67375,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134112,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":33374208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134139,"ondisk_log_size":134139,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803756,"kb_used_data":1072672,"kb_used_omap":14,"kb_used_meta":1730993,"kb_avail":374683604,"statfs":{"total":386547056640,"available":383676010496,"internally_reserved":0,"allocated":1098416128,"data_stored":1097696252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1772537393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,3,4,8,5,6,4,0,0,0,0,2,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17891328,"num_objects":-2184,"num_object_clones":0,"num_object_copies":-8736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2184,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003080"},"pg_stats":[{"pgid":"1.f","version":"33'8421","reported_seq":8530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.559337+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:16.559337+0000","last_peered":"2024-09-16T04:47:16.559337+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:16.559337+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8421,"log_dups_size":0,"ondisk_log_size":8421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34676736,"num_objects":4233,"num_object_clones":0,"num_object_copies":16932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8421,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2046}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8418","reported_seq":8499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.279791+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:16.279791+0000","last_peered":"2024-09-16T04:47:16.279791+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:16.279791+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8418,"log_dups_size":0,"ondisk_log_size":8418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34862348,"num_objects":4256,"num_object_clones":0,"num_object_copies":17024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8416,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2052}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8419","reported_seq":8507,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:15.710199+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:15.710199+0000","last_peered":"2024-09-16T04:47:15.710199+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:15.710199+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8419,"log_dups_size":0,"ondisk_log_size":8419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34822814,"num_objects":4252,"num_object_clones":0,"num_object_copies":17008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8418,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2032}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8401","reported_seq":8481,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.108762+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:16.108762+0000","last_peered":"2024-09-16T04:47:16.108762+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:16.108762+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8401,"log_dups_size":0,"ondisk_log_size":8401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34968844,"num_objects":4269,"num_object_clones":0,"num_object_copies":17076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8399,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8521","reported_seq":8644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.510861+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:16.510861+0000","last_peered":"2024-09-16T04:47:16.510861+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:16.510861+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8521,"log_dups_size":0,"ondisk_log_size":8521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34567436,"num_objects":4220,"num_object_clones":0,"num_object_copies":16880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8519,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8231","reported_seq":8336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.539172+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:16.539172+0000","last_peered":"2024-09-16T04:47:16.539172+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:16.539172+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8231,"log_dups_size":0,"ondisk_log_size":8231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33431552,"num_objects":4081,"num_object_clones":0,"num_object_copies":16324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8231,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1903}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8373","reported_seq":8460,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:15.968024+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:15.968024+0000","last_peered":"2024-09-16T04:47:15.968024+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:15.968024+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8373,"log_dups_size":0,"ondisk_log_size":8373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34624780,"num_objects":4227,"num_object_clones":0,"num_object_copies":16908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8371,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2011}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8529","reported_seq":13086,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:17.917733+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:17.917733+0000","last_peered":"2024-09-16T04:47:17.917733+0000","last_clean":"2024-09-16T04:47:17.917733+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:17.917733+0000","last_undegraded":"2024-09-16T04:47:17.917733+0000","last_fullsized":"2024-09-16T04:47:17.917733+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8529,"log_dups_size":0,"ondisk_log_size":8529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:18.655748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34862348,"num_objects":4256,"num_object_clones":0,"num_object_copies":17024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8527,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8612","reported_seq":12366,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.627200+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:16.627200+0000","last_peered":"2024-09-16T04:47:16.627200+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:16.627200+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8612,"log_dups_size":0,"ondisk_log_size":8612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33942040,"num_objects":4144,"num_object_clones":0,"num_object_copies":16576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8609,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1851,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8439","reported_seq":8539,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:15.764863+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:15.764863+0000","last_peered":"2024-09-16T04:47:15.764863+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:15.764863+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8439,"log_dups_size":0,"ondisk_log_size":8439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35378444,"num_objects":4319,"num_object_clones":0,"num_object_copies":17276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8437,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2129}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8315","reported_seq":8441,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.154074+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:16.154074+0000","last_peered":"2024-09-16T04:47:16.154074+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:16.154074+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8315,"log_dups_size":0,"ondisk_log_size":8315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34205586,"num_objects":4176,"num_object_clones":0,"num_object_copies":16704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8312,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8174","reported_seq":8259,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.535466+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:16.535466+0000","last_peered":"2024-09-16T04:47:16.535466+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:16.535466+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8174,"log_dups_size":0,"ondisk_log_size":8174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34052742,"num_objects":4157,"num_object_clones":0,"num_object_copies":16628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8173,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2051}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8342","reported_seq":8426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:15.784089+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:15.784089+0000","last_peered":"2024-09-16T04:47:15.784089+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:15.784089+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8342,"log_dups_size":0,"ondisk_log_size":8342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34331270,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8341,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1970}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8475","reported_seq":8602,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.476871+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:16.476871+0000","last_peered":"2024-09-16T04:47:16.476871+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:16.476871+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8475,"log_dups_size":0,"ondisk_log_size":8475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35280140,"num_objects":4307,"num_object_clones":0,"num_object_copies":17228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8473,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2109}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8223","reported_seq":8320,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:15.720824+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:15.720824+0000","last_peered":"2024-09-16T04:47:15.720824+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:15.720824+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8223,"log_dups_size":0,"ondisk_log_size":8223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34616588,"num_objects":4226,"num_object_clones":0,"num_object_copies":16904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8221,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2102}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8246","reported_seq":8335,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.306127+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:16.306127+0000","last_peered":"2024-09-16T04:47:16.306127+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:16.306127+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8246,"log_dups_size":0,"ondisk_log_size":8246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264908,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1833}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":551888576,"num_objects":67375,"num_object_clones":0,"num_object_copies":269500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67375,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134112,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":33374208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1096019968,"data_stored":1096019968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134139,"ondisk_log_size":134139,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738616,"kb_used_data":269600,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93633224,"statfs":{"total":96636764160,"available":95880421376,"internally_reserved":0,"allocated":276070400,"data_stored":275890431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,3,1,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747468,"kb_used_data":270648,"kb_used_omap":1,"kb_used_meta":476798,"kb_avail":93624372,"statfs":{"total":96636764160,"available":95871356928,"internally_reserved":0,"allocated":277143552,"data_stored":276963583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":488241605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,0,1,0,2,3,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578288,"kb_used_data":261984,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93793552,"statfs":{"total":96636764160,"available":96044597248,"internally_reserved":0,"allocated":268271616,"data_stored":268091647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":323874511},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739384,"kb_used_data":270440,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93632456,"statfs":{"total":96636764160,"available":95879634944,"internally_reserved":0,"allocated":276930560,"data_stored":276750591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":480177870},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":275968000,"data_stored":275968000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267997184,"data_stored":267997184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":276344832,"data_stored":276344832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":275709952,"data_stored":275709952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:20.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:47:20.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8421", 'reported_seq': 8530, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:16.559337+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:47:16.559337+0000', 'last_peered': '2024-09-16T04:47:16.559337+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:47:16.559337+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8421, 'log_dups_size': 0, 'ondisk_log_size': 8421, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34676736, 'num_objects': 4233, 'num_object_clones': 0, 'num_object_copies': 16932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2187, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4233, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8421, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2046}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:20.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:47:20.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8418", 'reported_seq': 8499, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:16.279791+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:47:16.279791+0000', 'last_peered': '2024-09-16T04:47:16.279791+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:47:16.279791+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8418, 'log_dups_size': 0, 'ondisk_log_size': 8418, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34862348, 'num_objects': 4256, 'num_object_clones': 0, 'num_object_copies': 17024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2206, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4256, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8416, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2052}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:20.882 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:47:20.882 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8419", 'reported_seq': 8507, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:15.710199+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:47:15.710199+0000', 'last_peered': '2024-09-16T04:47:15.710199+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:47:15.710199+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8419, 'log_dups_size': 0, 'ondisk_log_size': 8419, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34822814, 'num_objects': 4252, 'num_object_clones': 0, 'num_object_copies': 17008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4252, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8418, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2032}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:20.882 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:47:20.882 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8401", 'reported_seq': 8481, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:16.108762+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:47:16.108762+0000', 'last_peered': '2024-09-16T04:47:16.108762+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:47:16.108762+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8401, 'log_dups_size': 0, 'ondisk_log_size': 8401, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34968844, 'num_objects': 4269, 'num_object_clones': 0, 'num_object_copies': 17076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8399, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2047}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:20.882 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:47:20.882 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8521", 'reported_seq': 8644, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:16.510861+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:47:16.510861+0000', 'last_peered': '2024-09-16T04:47:16.510861+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:47:16.510861+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8521, 'log_dups_size': 0, 'ondisk_log_size': 8521, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34567436, 'num_objects': 4220, 'num_object_clones': 0, 'num_object_copies': 16880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8519, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1970}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:20.882 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:47:20.882 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8231", 'reported_seq': 8336, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:16.539172+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:47:16.539172+0000', 'last_peered': '2024-09-16T04:47:16.539172+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:47:16.539172+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8231, 'log_dups_size': 0, 'ondisk_log_size': 8231, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33431552, 'num_objects': 4081, 'num_object_clones': 0, 'num_object_copies': 16324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4081, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8231, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1903}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:20.882 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:47:20.882 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8373", 'reported_seq': 8460, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:15.968024+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:47:15.968024+0000', 'last_peered': '2024-09-16T04:47:15.968024+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:47:15.968024+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8373, 'log_dups_size': 0, 'ondisk_log_size': 8373, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34624780, 'num_objects': 4227, 'num_object_clones': 0, 'num_object_copies': 16908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2218, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8371, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2011}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:20.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:47:20.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8612", 'reported_seq': 12366, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:16.627200+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:47:16.627200+0000', 'last_peered': '2024-09-16T04:47:16.627200+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:47:16.627200+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8612, 'log_dups_size': 0, 'ondisk_log_size': 8612, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33942040, 'num_objects': 4144, 'num_object_clones': 0, 'num_object_copies': 16576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 495, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4144, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8609, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1851, 'num_bytes_recovered': 15040512, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3649}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:20.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:47:20.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8439", 'reported_seq': 8539, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:15.764863+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:47:15.764863+0000', 'last_peered': '2024-09-16T04:47:15.764863+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:47:15.764863+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8439, 'log_dups_size': 0, 'ondisk_log_size': 8439, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35378444, 'num_objects': 4319, 'num_object_clones': 0, 'num_object_copies': 17276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4319, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8437, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2129}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:20.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:47:20.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8315", 'reported_seq': 8441, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:16.154074+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:47:16.154074+0000', 'last_peered': '2024-09-16T04:47:16.154074+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:47:16.154074+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8315, 'log_dups_size': 0, 'ondisk_log_size': 8315, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34205586, 'num_objects': 4176, 'num_object_clones': 0, 'num_object_copies': 16704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2159, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4176, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8312, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 36, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2020}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:20.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:47:20.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8174", 'reported_seq': 8259, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:16.535466+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:47:16.535466+0000', 'last_peered': '2024-09-16T04:47:16.535466+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:47:16.535466+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8174, 'log_dups_size': 0, 'ondisk_log_size': 8174, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34052742, 'num_objects': 4157, 'num_object_clones': 0, 'num_object_copies': 16628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2107, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8173, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2051}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:20.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:47:20.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8342", 'reported_seq': 8426, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:15.784089+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:47:15.784089+0000', 'last_peered': '2024-09-16T04:47:15.784089+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:47:15.784089+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8342, 'log_dups_size': 0, 'ondisk_log_size': 8342, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34331270, 'num_objects': 4191, 'num_object_clones': 0, 'num_object_copies': 16764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8341, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1970}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:20.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:47:20.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8475", 'reported_seq': 8602, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:16.476871+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:47:16.476871+0000', 'last_peered': '2024-09-16T04:47:16.476871+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:47:16.476871+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8475, 'log_dups_size': 0, 'ondisk_log_size': 8475, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35280140, 'num_objects': 4307, 'num_object_clones': 0, 'num_object_copies': 17228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8473, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:20.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:47:20.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8223", 'reported_seq': 8320, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:15.720824+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:47:15.720824+0000', 'last_peered': '2024-09-16T04:47:15.720824+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:47:15.720824+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8223, 'log_dups_size': 0, 'ondisk_log_size': 8223, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34616588, 'num_objects': 4226, 'num_object_clones': 0, 'num_object_copies': 16904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2126, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4226, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8221, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2102}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:20.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:47:20.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8246", 'reported_seq': 8335, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:16.306127+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:47:16.306127+0000', 'last_peered': '2024-09-16T04:47:16.306127+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:47:16.306127+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8246, 'log_dups_size': 0, 'ondisk_log_size': 8246, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33264908, 'num_objects': 4061, 'num_object_clones': 0, 'num_object_copies': 16244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8244, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1833}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:20.884 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:47:21.569 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:21.569 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31217/269500 objects degraded (11.583%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":453,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":67375,"data_bytes":551888576,"bytes_used":2871046144,"bytes_avail":383676010496,"bytes_total":386547056640,"degraded_objects":31217,"degraded_total":269500,"degraded_ratio":0.1158330241187384,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":81216,"recovering_keys_per_sec":0,"num_objects_recovered":140,"num_bytes_recovered":974848,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":181},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:47:21.597 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:47:21.597 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:22.185 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:22.185 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:22.217 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-16T04:47:20.682735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":551888576,"num_objects":67375,"num_object_clones":0,"num_object_copies":269500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67375,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134112,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":33374208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134139,"ondisk_log_size":134139,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803756,"kb_used_data":1072672,"kb_used_omap":14,"kb_used_meta":1730993,"kb_avail":374683604,"statfs":{"total":386547056640,"available":383676010496,"internally_reserved":0,"allocated":1098416128,"data_stored":1097696252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1772537393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,3,4,8,5,6,4,0,0,0,0,2,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17891328,"num_objects":-2184,"num_object_clones":0,"num_object_copies":-8736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2184,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003080"},"pg_stats":[{"pgid":"1.f","version":"33'8421","reported_seq":8530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.559337+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:16.559337+0000","last_peered":"2024-09-16T04:47:16.559337+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:16.559337+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8421,"log_dups_size":0,"ondisk_log_size":8421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34676736,"num_objects":4233,"num_object_clones":0,"num_object_copies":16932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8421,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2046}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8418","reported_seq":8499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.279791+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:16.279791+0000","last_peered":"2024-09-16T04:47:16.279791+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:16.279791+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8418,"log_dups_size":0,"ondisk_log_size":8418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34862348,"num_objects":4256,"num_object_clones":0,"num_object_copies":17024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8416,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2052}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8419","reported_seq":8507,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:15.710199+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:15.710199+0000","last_peered":"2024-09-16T04:47:15.710199+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:15.710199+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8419,"log_dups_size":0,"ondisk_log_size":8419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34822814,"num_objects":4252,"num_object_clones":0,"num_object_copies":17008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8418,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2032}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8401","reported_seq":8481,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.108762+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:16.108762+0000","last_peered":"2024-09-16T04:47:16.108762+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:16.108762+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8401,"log_dups_size":0,"ondisk_log_size":8401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34968844,"num_objects":4269,"num_object_clones":0,"num_object_copies":17076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8399,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8521","reported_seq":8644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.510861+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:16.510861+0000","last_peered":"2024-09-16T04:47:16.510861+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:16.510861+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8521,"log_dups_size":0,"ondisk_log_size":8521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34567436,"num_objects":4220,"num_object_clones":0,"num_object_copies":16880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8519,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8231","reported_seq":8336,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.539172+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:16.539172+0000","last_peered":"2024-09-16T04:47:16.539172+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:16.539172+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8231,"log_dups_size":0,"ondisk_log_size":8231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33431552,"num_objects":4081,"num_object_clones":0,"num_object_copies":16324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8231,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1903}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8373","reported_seq":8460,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:15.968024+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:15.968024+0000","last_peered":"2024-09-16T04:47:15.968024+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:15.968024+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8373,"log_dups_size":0,"ondisk_log_size":8373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34624780,"num_objects":4227,"num_object_clones":0,"num_object_copies":16908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8371,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2011}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8529","reported_seq":13086,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:17.917733+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:17.917733+0000","last_peered":"2024-09-16T04:47:17.917733+0000","last_clean":"2024-09-16T04:47:17.917733+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:17.917733+0000","last_undegraded":"2024-09-16T04:47:17.917733+0000","last_fullsized":"2024-09-16T04:47:17.917733+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8529,"log_dups_size":0,"ondisk_log_size":8529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:18.655748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34862348,"num_objects":4256,"num_object_clones":0,"num_object_copies":17024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8527,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8612","reported_seq":12366,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.627200+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:16.627200+0000","last_peered":"2024-09-16T04:47:16.627200+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:16.627200+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8612,"log_dups_size":0,"ondisk_log_size":8612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33942040,"num_objects":4144,"num_object_clones":0,"num_object_copies":16576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8609,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1851,"num_bytes_recovered":15040512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8439","reported_seq":8539,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:15.764863+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:15.764863+0000","last_peered":"2024-09-16T04:47:15.764863+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:15.764863+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8439,"log_dups_size":0,"ondisk_log_size":8439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35378444,"num_objects":4319,"num_object_clones":0,"num_object_copies":17276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8437,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2129}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8315","reported_seq":8441,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.154074+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:16.154074+0000","last_peered":"2024-09-16T04:47:16.154074+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:16.154074+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8315,"log_dups_size":0,"ondisk_log_size":8315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34205586,"num_objects":4176,"num_object_clones":0,"num_object_copies":16704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8312,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8174","reported_seq":8259,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.535466+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:16.535466+0000","last_peered":"2024-09-16T04:47:16.535466+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:16.535466+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8174,"log_dups_size":0,"ondisk_log_size":8174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34052742,"num_objects":4157,"num_object_clones":0,"num_object_copies":16628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8173,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2051}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8342","reported_seq":8426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:15.784089+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:15.784089+0000","last_peered":"2024-09-16T04:47:15.784089+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:15.784089+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8342,"log_dups_size":0,"ondisk_log_size":8342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34331270,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8341,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1970}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8475","reported_seq":8602,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.476871+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:16.476871+0000","last_peered":"2024-09-16T04:47:16.476871+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:16.476871+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8475,"log_dups_size":0,"ondisk_log_size":8475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35280140,"num_objects":4307,"num_object_clones":0,"num_object_copies":17228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8473,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2109}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8223","reported_seq":8320,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:15.720824+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:15.720824+0000","last_peered":"2024-09-16T04:47:15.720824+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:15.720824+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8223,"log_dups_size":0,"ondisk_log_size":8223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34616588,"num_objects":4226,"num_object_clones":0,"num_object_copies":16904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8221,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2102}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8246","reported_seq":8335,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:16.306127+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:16.306127+0000","last_peered":"2024-09-16T04:47:16.306127+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:16.306127+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8246,"log_dups_size":0,"ondisk_log_size":8246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264908,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1833}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":551888576,"num_objects":67375,"num_object_clones":0,"num_object_copies":269500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67375,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134112,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":33374208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1096019968,"data_stored":1096019968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134139,"ondisk_log_size":134139,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738616,"kb_used_data":269600,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93633224,"statfs":{"total":96636764160,"available":95880421376,"internally_reserved":0,"allocated":276070400,"data_stored":275890431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,3,1,1,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747468,"kb_used_data":270648,"kb_used_omap":1,"kb_used_meta":476798,"kb_avail":93624372,"statfs":{"total":96636764160,"available":95871356928,"internally_reserved":0,"allocated":277143552,"data_stored":276963583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":488241605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,0,1,0,2,3,0,0,0,0,0,0,4],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578288,"kb_used_data":261984,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93793552,"statfs":{"total":96636764160,"available":96044597248,"internally_reserved":0,"allocated":268271616,"data_stored":268091647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":323874511},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739384,"kb_used_data":270440,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93632456,"statfs":{"total":96636764160,"available":95879634944,"internally_reserved":0,"allocated":276930560,"data_stored":276750591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":480177870},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":275968000,"data_stored":275968000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267997184,"data_stored":267997184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":276344832,"data_stored":276344832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":275709952,"data_stored":275709952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:25.219 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:25.788 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:25.788 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:25.816 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-16T04:47:24.683780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":542877376,"num_objects":66275,"num_object_clones":0,"num_object_copies":265100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66275,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135212,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4439,"num_bytes_recovered":33947648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135239,"ondisk_log_size":135239,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2790432,"kb_used_data":1059352,"kb_used_omap":14,"kb_used_meta":1730993,"kb_avail":374696928,"statfs":{"total":386547056640,"available":383689654272,"internally_reserved":0,"allocated":1084776448,"data_stored":1084056572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1772537393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,18,7,12,8,4,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18530304,"num_objects":-2262,"num_object_clones":0,"num_object_copies":-9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2262,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003135"},"pg_stats":[{"pgid":"1.f","version":"33'8488","reported_seq":8599,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.383907+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:21.383907+0000","last_peered":"2024-09-16T04:47:21.383907+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:21.383907+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8488,"log_dups_size":0,"ondisk_log_size":8488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34127872,"num_objects":4166,"num_object_clones":0,"num_object_copies":16664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1980}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8487","reported_seq":8568,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.277688+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:21.277688+0000","last_peered":"2024-09-16T04:47:21.277688+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:21.277688+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8487,"log_dups_size":0,"ondisk_log_size":8487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34297100,"num_objects":4187,"num_object_clones":0,"num_object_copies":16748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8485,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1983}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8481","reported_seq":8573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:20.914301+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:20.914301+0000","last_peered":"2024-09-16T04:47:20.914301+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:20.914301+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8481,"log_dups_size":0,"ondisk_log_size":8481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34314910,"num_objects":4190,"num_object_clones":0,"num_object_copies":16760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4190,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8480,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1970}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8493","reported_seq":8575,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.237040+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:21.237040+0000","last_peered":"2024-09-16T04:47:21.237040+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:21.237040+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8493,"log_dups_size":0,"ondisk_log_size":8493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34215180,"num_objects":4177,"num_object_clones":0,"num_object_copies":16708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8491,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1955}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8582","reported_seq":8706,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.648848+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:21.648848+0000","last_peered":"2024-09-16T04:47:21.648848+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:21.648848+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8582,"log_dups_size":0,"ondisk_log_size":8582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34067724,"num_objects":4159,"num_object_clones":0,"num_object_copies":16636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8580,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8306","reported_seq":8411,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.231973+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:21.231973+0000","last_peered":"2024-09-16T04:47:21.231973+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:21.231973+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8306,"log_dups_size":0,"ondisk_log_size":8306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32817152,"num_objects":4006,"num_object_clones":0,"num_object_copies":16024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8306,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1828}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8454","reported_seq":8543,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.326127+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:21.326127+0000","last_peered":"2024-09-16T04:47:21.326127+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:21.326127+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8454,"log_dups_size":0,"ondisk_log_size":8454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33961228,"num_objects":4146,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8452,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1931}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8596","reported_seq":13151,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:23.109670+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:23.109670+0000","last_peered":"2024-09-16T04:47:23.109670+0000","last_clean":"2024-09-16T04:47:23.109670+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:23.109670+0000","last_undegraded":"2024-09-16T04:47:23.109670+0000","last_fullsized":"2024-09-16T04:47:23.109670+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8596,"log_dups_size":0,"ondisk_log_size":8596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:24.620806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34313484,"num_objects":4189,"num_object_clones":0,"num_object_copies":16756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8594,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8663","reported_seq":12557,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.683240+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:21.683240+0000","last_peered":"2024-09-16T04:47:21.683240+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:21.683240+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8663,"log_dups_size":0,"ondisk_log_size":8663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33524248,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":425,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1921,"num_bytes_recovered":15613952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3668}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8514","reported_seq":8614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:20.921543+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:20.921543+0000","last_peered":"2024-09-16T04:47:20.921543+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:20.921543+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8514,"log_dups_size":0,"ondisk_log_size":8514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34764044,"num_objects":4244,"num_object_clones":0,"num_object_copies":16976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8512,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8380","reported_seq":8508,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.547572+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:21.547572+0000","last_peered":"2024-09-16T04:47:21.547572+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:21.547572+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8380,"log_dups_size":0,"ondisk_log_size":8380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33673106,"num_objects":4111,"num_object_clones":0,"num_object_copies":16444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8377,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1956}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8238","reported_seq":8325,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.657527+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:21.657527+0000","last_peered":"2024-09-16T04:47:21.657527+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:21.657527+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8238,"log_dups_size":0,"ondisk_log_size":8238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528454,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8237,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1988}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8414","reported_seq":8498,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:20.732896+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:20.732896+0000","last_peered":"2024-09-16T04:47:20.732896+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:20.732896+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8414,"log_dups_size":0,"ondisk_log_size":8414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33741446,"num_objects":4119,"num_object_clones":0,"num_object_copies":16476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1898}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8542","reported_seq":8670,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.591057+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:21.591057+0000","last_peered":"2024-09-16T04:47:21.591057+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:21.591057+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8542,"log_dups_size":0,"ondisk_log_size":8542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34731276,"num_objects":4240,"num_object_clones":0,"num_object_copies":16960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8540,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2042}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8285","reported_seq":8382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:20.837255+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:20.837255+0000","last_peered":"2024-09-16T04:47:20.837255+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:20.837255+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8285,"log_dups_size":0,"ondisk_log_size":8285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34108684,"num_objects":4164,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8283,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2040}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8316","reported_seq":8406,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.148404+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:21.148404+0000","last_peered":"2024-09-16T04:47:21.148404+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:21.148404+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8316,"log_dups_size":0,"ondisk_log_size":8316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32691468,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8314,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":542877376,"num_objects":66275,"num_object_clones":0,"num_object_copies":265100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66275,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135212,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4439,"num_bytes_recovered":33947648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1082413056,"data_stored":1082413056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135239,"ondisk_log_size":135239,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738396,"kb_used_data":265220,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93633444,"statfs":{"total":96636764160,"available":95880646656,"internally_reserved":0,"allocated":271585280,"data_stored":271405311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730632,"kb_used_data":266296,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93641208,"statfs":{"total":96636764160,"available":95888596992,"internally_reserved":0,"allocated":272687104,"data_stored":272507135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,3,2,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051656,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582292,"kb_used_data":261828,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93789548,"statfs":{"total":96636764160,"available":96040497152,"internally_reserved":0,"allocated":268111872,"data_stored":267931903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328134351},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739112,"kb_used_data":266008,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93632728,"statfs":{"total":96636764160,"available":95879913472,"internally_reserved":0,"allocated":272392192,"data_stored":272212223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":484437710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,9,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271507456,"data_stored":271507456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267808768,"data_stored":267808768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":271851520,"data_stored":271851520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271245312,"data_stored":271245312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:25.817 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:26.385 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:26.386 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:26.422 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-16T04:47:24.683780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":542877376,"num_objects":66275,"num_object_clones":0,"num_object_copies":265100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66275,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135212,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4439,"num_bytes_recovered":33947648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135239,"ondisk_log_size":135239,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2790432,"kb_used_data":1059352,"kb_used_omap":14,"kb_used_meta":1730993,"kb_avail":374696928,"statfs":{"total":386547056640,"available":383689654272,"internally_reserved":0,"allocated":1084776448,"data_stored":1084056572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1772537393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,18,7,12,8,4,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18530304,"num_objects":-2262,"num_object_clones":0,"num_object_copies":-9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2262,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003135"},"pg_stats":[{"pgid":"1.f","version":"33'8488","reported_seq":8599,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.383907+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:21.383907+0000","last_peered":"2024-09-16T04:47:21.383907+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:21.383907+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8488,"log_dups_size":0,"ondisk_log_size":8488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34127872,"num_objects":4166,"num_object_clones":0,"num_object_copies":16664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1980}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8487","reported_seq":8568,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.277688+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:21.277688+0000","last_peered":"2024-09-16T04:47:21.277688+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:21.277688+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8487,"log_dups_size":0,"ondisk_log_size":8487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34297100,"num_objects":4187,"num_object_clones":0,"num_object_copies":16748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8485,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1983}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8481","reported_seq":8573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:20.914301+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:20.914301+0000","last_peered":"2024-09-16T04:47:20.914301+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:20.914301+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8481,"log_dups_size":0,"ondisk_log_size":8481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34314910,"num_objects":4190,"num_object_clones":0,"num_object_copies":16760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4190,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8480,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1970}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8493","reported_seq":8575,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.237040+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:21.237040+0000","last_peered":"2024-09-16T04:47:21.237040+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:21.237040+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8493,"log_dups_size":0,"ondisk_log_size":8493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34215180,"num_objects":4177,"num_object_clones":0,"num_object_copies":16708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8491,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1955}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8582","reported_seq":8706,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.648848+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:21.648848+0000","last_peered":"2024-09-16T04:47:21.648848+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:21.648848+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8582,"log_dups_size":0,"ondisk_log_size":8582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34067724,"num_objects":4159,"num_object_clones":0,"num_object_copies":16636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8580,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8306","reported_seq":8411,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.231973+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:21.231973+0000","last_peered":"2024-09-16T04:47:21.231973+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:21.231973+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8306,"log_dups_size":0,"ondisk_log_size":8306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32817152,"num_objects":4006,"num_object_clones":0,"num_object_copies":16024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8306,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1828}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8454","reported_seq":8543,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.326127+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:21.326127+0000","last_peered":"2024-09-16T04:47:21.326127+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:21.326127+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8454,"log_dups_size":0,"ondisk_log_size":8454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33961228,"num_objects":4146,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8452,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1931}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8596","reported_seq":13151,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:23.109670+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:23.109670+0000","last_peered":"2024-09-16T04:47:23.109670+0000","last_clean":"2024-09-16T04:47:23.109670+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:23.109670+0000","last_undegraded":"2024-09-16T04:47:23.109670+0000","last_fullsized":"2024-09-16T04:47:23.109670+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8596,"log_dups_size":0,"ondisk_log_size":8596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:24.620806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34313484,"num_objects":4189,"num_object_clones":0,"num_object_copies":16756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8594,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8663","reported_seq":12557,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.683240+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:21.683240+0000","last_peered":"2024-09-16T04:47:21.683240+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:21.683240+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8663,"log_dups_size":0,"ondisk_log_size":8663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33524248,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":425,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1921,"num_bytes_recovered":15613952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3668}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8514","reported_seq":8614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:20.921543+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:20.921543+0000","last_peered":"2024-09-16T04:47:20.921543+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:20.921543+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8514,"log_dups_size":0,"ondisk_log_size":8514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34764044,"num_objects":4244,"num_object_clones":0,"num_object_copies":16976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8512,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8380","reported_seq":8508,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.547572+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:21.547572+0000","last_peered":"2024-09-16T04:47:21.547572+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:21.547572+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8380,"log_dups_size":0,"ondisk_log_size":8380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33673106,"num_objects":4111,"num_object_clones":0,"num_object_copies":16444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8377,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1956}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8238","reported_seq":8325,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.657527+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:21.657527+0000","last_peered":"2024-09-16T04:47:21.657527+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:21.657527+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8238,"log_dups_size":0,"ondisk_log_size":8238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528454,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8237,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1988}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8414","reported_seq":8498,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:20.732896+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:20.732896+0000","last_peered":"2024-09-16T04:47:20.732896+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:20.732896+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8414,"log_dups_size":0,"ondisk_log_size":8414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33741446,"num_objects":4119,"num_object_clones":0,"num_object_copies":16476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1898}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8542","reported_seq":8670,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.591057+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:21.591057+0000","last_peered":"2024-09-16T04:47:21.591057+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:21.591057+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8542,"log_dups_size":0,"ondisk_log_size":8542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34731276,"num_objects":4240,"num_object_clones":0,"num_object_copies":16960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8540,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2042}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8285","reported_seq":8382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:20.837255+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:20.837255+0000","last_peered":"2024-09-16T04:47:20.837255+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:20.837255+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8285,"log_dups_size":0,"ondisk_log_size":8285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34108684,"num_objects":4164,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8283,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2040}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8316","reported_seq":8406,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.148404+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:21.148404+0000","last_peered":"2024-09-16T04:47:21.148404+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:21.148404+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8316,"log_dups_size":0,"ondisk_log_size":8316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32691468,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8314,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":542877376,"num_objects":66275,"num_object_clones":0,"num_object_copies":265100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66275,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135212,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4439,"num_bytes_recovered":33947648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1082413056,"data_stored":1082413056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135239,"ondisk_log_size":135239,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738396,"kb_used_data":265220,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93633444,"statfs":{"total":96636764160,"available":95880646656,"internally_reserved":0,"allocated":271585280,"data_stored":271405311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730632,"kb_used_data":266296,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93641208,"statfs":{"total":96636764160,"available":95888596992,"internally_reserved":0,"allocated":272687104,"data_stored":272507135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,3,2,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051656,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582292,"kb_used_data":261828,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93789548,"statfs":{"total":96636764160,"available":96040497152,"internally_reserved":0,"allocated":268111872,"data_stored":267931903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328134351},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739112,"kb_used_data":266008,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93632728,"statfs":{"total":96636764160,"available":95879913472,"internally_reserved":0,"allocated":272392192,"data_stored":272212223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":484437710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,9,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271507456,"data_stored":271507456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267808768,"data_stored":267808768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":271851520,"data_stored":271851520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271245312,"data_stored":271245312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:26.423 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:47:26.423 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8488", 'reported_seq': 8599, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:21.383907+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:47:21.383907+0000', 'last_peered': '2024-09-16T04:47:21.383907+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:47:21.383907+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8488, 'log_dups_size': 0, 'ondisk_log_size': 8488, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34127872, 'num_objects': 4166, 'num_object_clones': 0, 'num_object_copies': 16664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4166, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8488, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1980}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:26.424 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:47:26.424 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8487", 'reported_seq': 8568, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:21.277688+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:47:21.277688+0000', 'last_peered': '2024-09-16T04:47:21.277688+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:47:21.277688+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8487, 'log_dups_size': 0, 'ondisk_log_size': 8487, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34297100, 'num_objects': 4187, 'num_object_clones': 0, 'num_object_copies': 16748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2206, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4187, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8485, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1983}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:26.424 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:47:26.424 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8481", 'reported_seq': 8573, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:20.914301+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:47:20.914301+0000', 'last_peered': '2024-09-16T04:47:20.914301+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:47:20.914301+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8481, 'log_dups_size': 0, 'ondisk_log_size': 8481, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34314910, 'num_objects': 4190, 'num_object_clones': 0, 'num_object_copies': 16760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4190, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8480, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1970}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:26.424 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:47:26.424 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8493", 'reported_seq': 8575, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:21.237040+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:47:21.237040+0000', 'last_peered': '2024-09-16T04:47:21.237040+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:47:21.237040+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8493, 'log_dups_size': 0, 'ondisk_log_size': 8493, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34215180, 'num_objects': 4177, 'num_object_clones': 0, 'num_object_copies': 16708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8491, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1955}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:26.424 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:47:26.424 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8582", 'reported_seq': 8706, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:21.648848+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:47:21.648848+0000', 'last_peered': '2024-09-16T04:47:21.648848+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:47:21.648848+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8582, 'log_dups_size': 0, 'ondisk_log_size': 8582, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34067724, 'num_objects': 4159, 'num_object_clones': 0, 'num_object_copies': 16636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8580, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1909}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:26.424 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:47:26.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8306", 'reported_seq': 8411, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:21.231973+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:47:21.231973+0000', 'last_peered': '2024-09-16T04:47:21.231973+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:47:21.231973+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8306, 'log_dups_size': 0, 'ondisk_log_size': 8306, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32817152, 'num_objects': 4006, 'num_object_clones': 0, 'num_object_copies': 16024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4006, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8306, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1828}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:26.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:47:26.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8454", 'reported_seq': 8543, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:21.326127+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:47:21.326127+0000', 'last_peered': '2024-09-16T04:47:21.326127+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:47:21.326127+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8454, 'log_dups_size': 0, 'ondisk_log_size': 8454, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33961228, 'num_objects': 4146, 'num_object_clones': 0, 'num_object_copies': 16584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2217, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8452, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1931}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:26.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:47:26.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8663", 'reported_seq': 12557, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:21.683240+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:47:21.683240+0000', 'last_peered': '2024-09-16T04:47:21.683240+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:47:21.683240+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8663, 'log_dups_size': 0, 'ondisk_log_size': 8663, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33524248, 'num_objects': 4093, 'num_object_clones': 0, 'num_object_copies': 16372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 425, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8660, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1921, 'num_bytes_recovered': 15613952, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3668}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:26.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:47:26.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8514", 'reported_seq': 8614, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:20.921543+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:47:20.921543+0000', 'last_peered': '2024-09-16T04:47:20.921543+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:47:20.921543+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8514, 'log_dups_size': 0, 'ondisk_log_size': 8514, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34764044, 'num_objects': 4244, 'num_object_clones': 0, 'num_object_copies': 16976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4244, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8512, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2054}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:26.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:47:26.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8380", 'reported_seq': 8508, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:21.547572+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:47:21.547572+0000', 'last_peered': '2024-09-16T04:47:21.547572+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:47:21.547572+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8380, 'log_dups_size': 0, 'ondisk_log_size': 8380, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33673106, 'num_objects': 4111, 'num_object_clones': 0, 'num_object_copies': 16444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4111, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8377, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1956}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:26.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:47:26.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8238", 'reported_seq': 8325, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:21.657527+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:47:21.657527+0000', 'last_peered': '2024-09-16T04:47:21.657527+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:47:21.657527+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8238, 'log_dups_size': 0, 'ondisk_log_size': 8238, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33528454, 'num_objects': 4093, 'num_object_clones': 0, 'num_object_copies': 16372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8237, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1988}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:26.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:47:26.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8414", 'reported_seq': 8498, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:20.732896+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:47:20.732896+0000', 'last_peered': '2024-09-16T04:47:20.732896+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:47:20.732896+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8414, 'log_dups_size': 0, 'ondisk_log_size': 8414, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33741446, 'num_objects': 4119, 'num_object_clones': 0, 'num_object_copies': 16476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8413, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1898}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:26.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:47:26.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8542", 'reported_seq': 8670, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:21.591057+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:47:21.591057+0000', 'last_peered': '2024-09-16T04:47:21.591057+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:47:21.591057+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8542, 'log_dups_size': 0, 'ondisk_log_size': 8542, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34731276, 'num_objects': 4240, 'num_object_clones': 0, 'num_object_copies': 16960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4240, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8540, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2042}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:26.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:47:26.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8285", 'reported_seq': 8382, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:20.837255+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:47:20.837255+0000', 'last_peered': '2024-09-16T04:47:20.837255+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:47:20.837255+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8285, 'log_dups_size': 0, 'ondisk_log_size': 8285, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34108684, 'num_objects': 4164, 'num_object_clones': 0, 'num_object_copies': 16656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2126, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4164, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8283, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2040}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:26.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:47:26.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8316", 'reported_seq': 8406, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:21.148404+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:47:21.148404+0000', 'last_peered': '2024-09-16T04:47:21.148404+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:47:21.148404+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8316, 'log_dups_size': 0, 'ondisk_log_size': 8316, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32691468, 'num_objects': 3991, 'num_object_clones': 0, 'num_object_copies': 15964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3991, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8314, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1763}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:26.427 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:47:27.108 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:27.108 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31143/265100 objects degraded (11.748%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":459,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":66275,"data_bytes":542877376,"bytes_used":2857402368,"bytes_avail":383689654272,"bytes_total":386547056640,"degraded_objects":31143,"degraded_total":265100,"degraded_ratio":0.11747642399094679,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":88040,"recovering_keys_per_sec":0,"num_objects_recovered":149,"num_bytes_recovered":1056768,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":188},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T04:42:26.585812+0000","services":{}},"progress_events":{}} 2024-09-16T04:47:27.127 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:47:27.127 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:27.701 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:27.702 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:27.725 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-09-16T04:47:26.684254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":538478272,"num_objects":65738,"num_object_clones":0,"num_object_copies":262952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65738,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135749,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4439,"num_bytes_recovered":33947648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135776,"ondisk_log_size":135776,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773580,"kb_used_data":1050816,"kb_used_omap":14,"kb_used_meta":1722673,"kb_avail":374713780,"statfs":{"total":386547056640,"available":383706910720,"internally_reserved":0,"allocated":1076035584,"data_stored":1075315708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1764017713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,4,4,10,8,4,0,0,0,0,4,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22380544,"num_objects":-2732,"num_object_clones":0,"num_object_copies":-10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2732,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003070"},"pg_stats":[{"pgid":"1.f","version":"33'8488","reported_seq":8599,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.383907+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:21.383907+0000","last_peered":"2024-09-16T04:47:21.383907+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:21.383907+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8488,"log_dups_size":0,"ondisk_log_size":8488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34127872,"num_objects":4166,"num_object_clones":0,"num_object_copies":16664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1980}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8547","reported_seq":8629,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.388304+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:26.388304+0000","last_peered":"2024-09-16T04:47:26.388304+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:26.388304+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8547,"log_dups_size":0,"ondisk_log_size":8547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33805580,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8545,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1923}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8545","reported_seq":8637,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:25.645464+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:25.645464+0000","last_peered":"2024-09-16T04:47:25.645464+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:25.645464+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8545,"log_dups_size":0,"ondisk_log_size":8545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33790622,"num_objects":4126,"num_object_clones":0,"num_object_copies":16504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4126,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8544,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1906}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8562","reported_seq":8644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.234822+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:26.234822+0000","last_peered":"2024-09-16T04:47:26.234822+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:26.234822+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8562,"log_dups_size":0,"ondisk_log_size":8562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33649932,"num_objects":4108,"num_object_clones":0,"num_object_copies":16432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8560,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1886}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8582","reported_seq":8706,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.648848+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:21.648848+0000","last_peered":"2024-09-16T04:47:21.648848+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:21.648848+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8582,"log_dups_size":0,"ondisk_log_size":8582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34067724,"num_objects":4159,"num_object_clones":0,"num_object_copies":16636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8580,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8306","reported_seq":8411,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.231973+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:21.231973+0000","last_peered":"2024-09-16T04:47:21.231973+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:21.231973+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8306,"log_dups_size":0,"ondisk_log_size":8306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32817152,"num_objects":4006,"num_object_clones":0,"num_object_copies":16024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8306,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1828}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8526","reported_seq":8615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.312758+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:26.312758+0000","last_peered":"2024-09-16T04:47:26.312758+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:26.312758+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8526,"log_dups_size":0,"ondisk_log_size":8526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33371404,"num_objects":4074,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8524,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1859}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8596","reported_seq":13151,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:23.109670+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:23.109670+0000","last_peered":"2024-09-16T04:47:23.109670+0000","last_clean":"2024-09-16T04:47:23.109670+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:23.109670+0000","last_undegraded":"2024-09-16T04:47:23.109670+0000","last_fullsized":"2024-09-16T04:47:23.109670+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8596,"log_dups_size":0,"ondisk_log_size":8596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:24.620806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34313484,"num_objects":4189,"num_object_clones":0,"num_object_copies":16756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8594,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8663","reported_seq":12557,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.683240+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:21.683240+0000","last_peered":"2024-09-16T04:47:21.683240+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:21.683240+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8663,"log_dups_size":0,"ondisk_log_size":8663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33524248,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":425,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8660,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1921,"num_bytes_recovered":15613952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3668}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8573","reported_seq":8673,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:25.854498+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:25.854498+0000","last_peered":"2024-09-16T04:47:25.854498+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:25.854498+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8573,"log_dups_size":0,"ondisk_log_size":8573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34280716,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8571,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1995}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8380","reported_seq":8508,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.547572+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:21.547572+0000","last_peered":"2024-09-16T04:47:21.547572+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:21.547572+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8380,"log_dups_size":0,"ondisk_log_size":8380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33673106,"num_objects":4111,"num_object_clones":0,"num_object_copies":16444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8377,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1956}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8238","reported_seq":8325,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.657527+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:21.657527+0000","last_peered":"2024-09-16T04:47:21.657527+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:21.657527+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8238,"log_dups_size":0,"ondisk_log_size":8238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528454,"num_objects":4093,"num_object_clones":0,"num_object_copies":16372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8237,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1988}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8484","reported_seq":8567,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:25.972910+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:25.972910+0000","last_peered":"2024-09-16T04:47:25.972910+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:25.972910+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8484,"log_dups_size":0,"ondisk_log_size":8484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33168006,"num_objects":4049,"num_object_clones":0,"num_object_copies":16196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8483,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1828}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8542","reported_seq":8670,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:21.591057+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:21.591057+0000","last_peered":"2024-09-16T04:47:21.591057+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:21.591057+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8542,"log_dups_size":0,"ondisk_log_size":8542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34731276,"num_objects":4240,"num_object_clones":0,"num_object_copies":16960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8540,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2042}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8351","reported_seq":8448,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:25.713569+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:25.713569+0000","last_peered":"2024-09-16T04:47:25.713569+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:25.713569+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8351,"log_dups_size":0,"ondisk_log_size":8351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33568012,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8349,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1974}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8393","reported_seq":8483,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.321378+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:26.321378+0000","last_peered":"2024-09-16T04:47:26.321378+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:26.321378+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8393,"log_dups_size":0,"ondisk_log_size":8393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32060684,"num_objects":3914,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":538478272,"num_objects":65738,"num_object_clones":0,"num_object_copies":262952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65738,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135749,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4439,"num_bytes_recovered":33947648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073766400,"data_stored":1073766400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135776,"ondisk_log_size":135776,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738396,"kb_used_data":265220,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93633444,"statfs":{"total":96636764160,"available":95880646656,"internally_reserved":0,"allocated":271585280,"data_stored":271405311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730492,"kb_used_data":261992,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93641348,"statfs":{"total":96636764160,"available":95888740352,"internally_reserved":0,"allocated":268279808,"data_stored":268099839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,1,1,1,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051656,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582292,"kb_used_data":261828,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93789548,"statfs":{"total":96636764160,"available":96040497152,"internally_reserved":0,"allocated":268111872,"data_stored":267931903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328134351},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722400,"kb_used_data":261776,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93649440,"statfs":{"total":96636764160,"available":95897026560,"internally_reserved":0,"allocated":268058624,"data_stored":267878655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":471658190},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,2,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":267194368,"data_stored":267194368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267808768,"data_stored":267808768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":267517952,"data_stored":267517952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271245312,"data_stored":271245312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:30.729 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:31.332 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:31.332 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:31.359 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-16T04:47:30.685301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":534308544,"num_objects":65229,"num_object_clones":0,"num_object_copies":260916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136258,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136285,"ondisk_log_size":136285,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773324,"kb_used_data":1046400,"kb_used_omap":14,"kb_used_meta":1726833,"kb_avail":374714036,"statfs":{"total":386547056640,"available":383707172864,"internally_reserved":0,"allocated":1071513600,"data_stored":1070793724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1768277553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,8,5,10,4,0,0,0,0,0,4,2,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17580032,"num_objects":-2146,"num_object_clones":0,"num_object_copies":-8584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2146,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003053"},"pg_stats":[{"pgid":"1.f","version":"33'8546","reported_seq":8657,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.633079+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:26.633079+0000","last_peered":"2024-09-16T04:47:26.633079+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:26.633079+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8546,"log_dups_size":0,"ondisk_log_size":8546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33652736,"num_objects":4108,"num_object_clones":0,"num_object_copies":16432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1922}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8547","reported_seq":8629,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.388304+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:26.388304+0000","last_peered":"2024-09-16T04:47:26.388304+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:26.388304+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8547,"log_dups_size":0,"ondisk_log_size":8547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33805580,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8545,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1923}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8545","reported_seq":8637,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:25.645464+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:25.645464+0000","last_peered":"2024-09-16T04:47:25.645464+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:25.645464+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8545,"log_dups_size":0,"ondisk_log_size":8545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33790622,"num_objects":4126,"num_object_clones":0,"num_object_copies":16504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4126,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8544,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1906}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8562","reported_seq":8644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.234822+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:26.234822+0000","last_peered":"2024-09-16T04:47:26.234822+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:26.234822+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8562,"log_dups_size":0,"ondisk_log_size":8562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33649932,"num_objects":4108,"num_object_clones":0,"num_object_copies":16432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8560,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1886}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8655","reported_seq":8779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.489662+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:26.489662+0000","last_peered":"2024-09-16T04:47:26.489662+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:26.489662+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8655,"log_dups_size":0,"ondisk_log_size":8655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33469708,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8653,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1836}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8365","reported_seq":8470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.576000+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:26.576000+0000","last_peered":"2024-09-16T04:47:26.576000+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:26.576000+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8365,"log_dups_size":0,"ondisk_log_size":8365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32333824,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8365,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1769}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8526","reported_seq":8615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.312758+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:26.312758+0000","last_peered":"2024-09-16T04:47:26.312758+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:26.312758+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8526,"log_dups_size":0,"ondisk_log_size":8526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33371404,"num_objects":4074,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8524,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1859}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8654","reported_seq":13211,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:27.946838+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:27.946838+0000","last_peered":"2024-09-16T04:47:27.946838+0000","last_clean":"2024-09-16T04:47:27.946838+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:27.946838+0000","last_undegraded":"2024-09-16T04:47:27.946838+0000","last_fullsized":"2024-09-16T04:47:27.946838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8654,"log_dups_size":0,"ondisk_log_size":8654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:29.665540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33838348,"num_objects":4131,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8652,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8723","reported_seq":12837,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.709804+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:26.709804+0000","last_peered":"2024-09-16T04:47:26.709804+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:26.709804+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8723,"log_dups_size":0,"ondisk_log_size":8723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33032728,"num_objects":4033,"num_object_clones":0,"num_object_copies":16132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8720,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2030,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8573","reported_seq":8673,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:25.854498+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:25.854498+0000","last_peered":"2024-09-16T04:47:25.854498+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:25.854498+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8573,"log_dups_size":0,"ondisk_log_size":8573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34280716,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8571,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1995}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8437","reported_seq":8565,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.559939+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:26.559939+0000","last_peered":"2024-09-16T04:47:26.559939+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:26.559939+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8437,"log_dups_size":0,"ondisk_log_size":8437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33206162,"num_objects":4054,"num_object_clones":0,"num_object_copies":16216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8434,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8296","reported_seq":8383,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.512807+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:26.512807+0000","last_peered":"2024-09-16T04:47:26.512807+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:26.512807+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8296,"log_dups_size":0,"ondisk_log_size":8296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33053318,"num_objects":4035,"num_object_clones":0,"num_object_copies":16140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8295,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1930}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8484","reported_seq":8567,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:25.972910+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:25.972910+0000","last_peered":"2024-09-16T04:47:25.972910+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:25.972910+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8484,"log_dups_size":0,"ondisk_log_size":8484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33168006,"num_objects":4049,"num_object_clones":0,"num_object_copies":16196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8483,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1828}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8628","reported_seq":8756,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.638998+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:26.638998+0000","last_peered":"2024-09-16T04:47:26.638998+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:26.638998+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8628,"log_dups_size":0,"ondisk_log_size":8628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34026764,"num_objects":4154,"num_object_clones":0,"num_object_copies":16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8626,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1956}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8351","reported_seq":8448,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:25.713569+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:25.713569+0000","last_peered":"2024-09-16T04:47:25.713569+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:25.713569+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8351,"log_dups_size":0,"ondisk_log_size":8351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33568012,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8349,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1974}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8393","reported_seq":8483,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.321378+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:26.321378+0000","last_peered":"2024-09-16T04:47:26.321378+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:26.321378+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8393,"log_dups_size":0,"ondisk_log_size":8393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32060684,"num_objects":3914,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":534308544,"num_objects":65229,"num_object_clones":0,"num_object_copies":260916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136258,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1067991040,"data_stored":1067991040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136285,"ondisk_log_size":136285,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734924,"kb_used_data":261748,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93636916,"statfs":{"total":96636764160,"available":95884201984,"internally_reserved":0,"allocated":268029952,"data_stored":267849983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,8,1,4,1,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730492,"kb_used_data":261992,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93641348,"statfs":{"total":96636764160,"available":95888740352,"internally_reserved":0,"allocated":268279808,"data_stored":268099839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,1,1,1,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585508,"kb_used_data":260884,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93786332,"statfs":{"total":96636764160,"available":96037203968,"internally_reserved":0,"allocated":267145216,"data_stored":266965247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332394191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722400,"kb_used_data":261776,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93649440,"statfs":{"total":96636764160,"available":95897026560,"internally_reserved":0,"allocated":268058624,"data_stored":267878655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":471658190},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,2,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":267194368,"data_stored":267194368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266338304,"data_stored":266338304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":267517952,"data_stored":267517952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":266940416,"data_stored":266940416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:31.360 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:31.913 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:31.913 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:31.943 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-16T04:47:30.685301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":534308544,"num_objects":65229,"num_object_clones":0,"num_object_copies":260916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136258,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136285,"ondisk_log_size":136285,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773324,"kb_used_data":1046400,"kb_used_omap":14,"kb_used_meta":1726833,"kb_avail":374714036,"statfs":{"total":386547056640,"available":383707172864,"internally_reserved":0,"allocated":1071513600,"data_stored":1070793724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1768277553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,8,5,10,4,0,0,0,0,0,4,2,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17580032,"num_objects":-2146,"num_object_clones":0,"num_object_copies":-8584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2146,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003053"},"pg_stats":[{"pgid":"1.f","version":"33'8546","reported_seq":8657,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.633079+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:26.633079+0000","last_peered":"2024-09-16T04:47:26.633079+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:26.633079+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8546,"log_dups_size":0,"ondisk_log_size":8546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33652736,"num_objects":4108,"num_object_clones":0,"num_object_copies":16432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1922}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8547","reported_seq":8629,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.388304+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:26.388304+0000","last_peered":"2024-09-16T04:47:26.388304+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:26.388304+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8547,"log_dups_size":0,"ondisk_log_size":8547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33805580,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8545,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1923}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8545","reported_seq":8637,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:25.645464+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:25.645464+0000","last_peered":"2024-09-16T04:47:25.645464+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:25.645464+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8545,"log_dups_size":0,"ondisk_log_size":8545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33790622,"num_objects":4126,"num_object_clones":0,"num_object_copies":16504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4126,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8544,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1906}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8562","reported_seq":8644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.234822+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:26.234822+0000","last_peered":"2024-09-16T04:47:26.234822+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:26.234822+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8562,"log_dups_size":0,"ondisk_log_size":8562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33649932,"num_objects":4108,"num_object_clones":0,"num_object_copies":16432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8560,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1886}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8655","reported_seq":8779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.489662+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:26.489662+0000","last_peered":"2024-09-16T04:47:26.489662+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:26.489662+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8655,"log_dups_size":0,"ondisk_log_size":8655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33469708,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8653,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1836}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8365","reported_seq":8470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.576000+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:26.576000+0000","last_peered":"2024-09-16T04:47:26.576000+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:26.576000+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8365,"log_dups_size":0,"ondisk_log_size":8365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32333824,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8365,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1769}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8526","reported_seq":8615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.312758+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:26.312758+0000","last_peered":"2024-09-16T04:47:26.312758+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:26.312758+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8526,"log_dups_size":0,"ondisk_log_size":8526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33371404,"num_objects":4074,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8524,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1859}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8654","reported_seq":13211,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:27.946838+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:27.946838+0000","last_peered":"2024-09-16T04:47:27.946838+0000","last_clean":"2024-09-16T04:47:27.946838+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:27.946838+0000","last_undegraded":"2024-09-16T04:47:27.946838+0000","last_fullsized":"2024-09-16T04:47:27.946838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8654,"log_dups_size":0,"ondisk_log_size":8654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:29.665540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33838348,"num_objects":4131,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8652,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8723","reported_seq":12837,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.709804+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:26.709804+0000","last_peered":"2024-09-16T04:47:26.709804+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:26.709804+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8723,"log_dups_size":0,"ondisk_log_size":8723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33032728,"num_objects":4033,"num_object_clones":0,"num_object_copies":16132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8720,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2030,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8573","reported_seq":8673,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:25.854498+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:25.854498+0000","last_peered":"2024-09-16T04:47:25.854498+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:25.854498+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8573,"log_dups_size":0,"ondisk_log_size":8573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34280716,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8571,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1995}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8437","reported_seq":8565,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.559939+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:26.559939+0000","last_peered":"2024-09-16T04:47:26.559939+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:26.559939+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8437,"log_dups_size":0,"ondisk_log_size":8437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33206162,"num_objects":4054,"num_object_clones":0,"num_object_copies":16216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8434,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8296","reported_seq":8383,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.512807+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:26.512807+0000","last_peered":"2024-09-16T04:47:26.512807+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:26.512807+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8296,"log_dups_size":0,"ondisk_log_size":8296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33053318,"num_objects":4035,"num_object_clones":0,"num_object_copies":16140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8295,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1930}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8484","reported_seq":8567,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:25.972910+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:25.972910+0000","last_peered":"2024-09-16T04:47:25.972910+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:25.972910+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8484,"log_dups_size":0,"ondisk_log_size":8484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33168006,"num_objects":4049,"num_object_clones":0,"num_object_copies":16196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8483,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1828}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8628","reported_seq":8756,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.638998+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:26.638998+0000","last_peered":"2024-09-16T04:47:26.638998+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:26.638998+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8628,"log_dups_size":0,"ondisk_log_size":8628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34026764,"num_objects":4154,"num_object_clones":0,"num_object_copies":16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8626,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1956}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8351","reported_seq":8448,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:25.713569+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:25.713569+0000","last_peered":"2024-09-16T04:47:25.713569+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:25.713569+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8351,"log_dups_size":0,"ondisk_log_size":8351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33568012,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8349,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1974}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8393","reported_seq":8483,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:26.321378+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:26.321378+0000","last_peered":"2024-09-16T04:47:26.321378+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:26.321378+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8393,"log_dups_size":0,"ondisk_log_size":8393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32060684,"num_objects":3914,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":534308544,"num_objects":65229,"num_object_clones":0,"num_object_copies":260916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136258,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1067991040,"data_stored":1067991040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136285,"ondisk_log_size":136285,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734924,"kb_used_data":261748,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93636916,"statfs":{"total":96636764160,"available":95884201984,"internally_reserved":0,"allocated":268029952,"data_stored":267849983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,8,1,4,1,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730492,"kb_used_data":261992,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93641348,"statfs":{"total":96636764160,"available":95888740352,"internally_reserved":0,"allocated":268279808,"data_stored":268099839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,1,1,1,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585508,"kb_used_data":260884,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93786332,"statfs":{"total":96636764160,"available":96037203968,"internally_reserved":0,"allocated":267145216,"data_stored":266965247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332394191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722400,"kb_used_data":261776,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93649440,"statfs":{"total":96636764160,"available":95897026560,"internally_reserved":0,"allocated":268058624,"data_stored":267878655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":471658190},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,2,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":267194368,"data_stored":267194368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266338304,"data_stored":266338304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":267517952,"data_stored":267517952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":266940416,"data_stored":266940416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:31.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:47:31.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8546", 'reported_seq': 8657, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:26.633079+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:47:26.633079+0000', 'last_peered': '2024-09-16T04:47:26.633079+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:47:26.633079+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8546, 'log_dups_size': 0, 'ondisk_log_size': 8546, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33652736, 'num_objects': 4108, 'num_object_clones': 0, 'num_object_copies': 16432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8546, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1922}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:31.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:47:31.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8547", 'reported_seq': 8629, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:26.388304+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:47:26.388304+0000', 'last_peered': '2024-09-16T04:47:26.388304+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:47:26.388304+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8547, 'log_dups_size': 0, 'ondisk_log_size': 8547, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33805580, 'num_objects': 4127, 'num_object_clones': 0, 'num_object_copies': 16508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2206, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8545, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1923}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:31.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:47:31.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8545", 'reported_seq': 8637, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:25.645464+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:47:25.645464+0000', 'last_peered': '2024-09-16T04:47:25.645464+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:47:25.645464+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8545, 'log_dups_size': 0, 'ondisk_log_size': 8545, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33790622, 'num_objects': 4126, 'num_object_clones': 0, 'num_object_copies': 16504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2221, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4126, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8544, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1906}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:31.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:47:31.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8562", 'reported_seq': 8644, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:26.234822+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:47:26.234822+0000', 'last_peered': '2024-09-16T04:47:26.234822+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:47:26.234822+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8562, 'log_dups_size': 0, 'ondisk_log_size': 8562, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33649932, 'num_objects': 4108, 'num_object_clones': 0, 'num_object_copies': 16432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2224, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8560, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1886}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:31.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:47:31.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8655", 'reported_seq': 8779, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:26.489662+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:47:26.489662+0000', 'last_peered': '2024-09-16T04:47:26.489662+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:47:26.489662+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8655, 'log_dups_size': 0, 'ondisk_log_size': 8655, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33469708, 'num_objects': 4086, 'num_object_clones': 0, 'num_object_copies': 16344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2252, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8653, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1836}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:31.946 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:47:31.946 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8365", 'reported_seq': 8470, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:26.576000+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:47:26.576000+0000', 'last_peered': '2024-09-16T04:47:26.576000+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:47:26.576000+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8365, 'log_dups_size': 0, 'ondisk_log_size': 8365, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32333824, 'num_objects': 3947, 'num_object_clones': 0, 'num_object_copies': 15788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8365, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1769}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:31.946 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:47:31.946 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8526", 'reported_seq': 8615, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:26.312758+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:47:26.312758+0000', 'last_peered': '2024-09-16T04:47:26.312758+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:47:26.312758+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8526, 'log_dups_size': 0, 'ondisk_log_size': 8526, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33371404, 'num_objects': 4074, 'num_object_clones': 0, 'num_object_copies': 16296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2217, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4074, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8524, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1859}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:31.946 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:47:31.946 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8723", 'reported_seq': 12837, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:26.709804+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:47:26.709804+0000', 'last_peered': '2024-09-16T04:47:26.709804+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:47:26.709804+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8723, 'log_dups_size': 0, 'ondisk_log_size': 8723, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33032728, 'num_objects': 4033, 'num_object_clones': 0, 'num_object_copies': 16132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4033, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8720, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2030, 'num_bytes_recovered': 15925248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3717}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:31.946 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:47:31.946 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8573", 'reported_seq': 8673, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:25.854498+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:47:25.854498+0000', 'last_peered': '2024-09-16T04:47:25.854498+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:47:25.854498+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8573, 'log_dups_size': 0, 'ondisk_log_size': 8573, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34280716, 'num_objects': 4185, 'num_object_clones': 0, 'num_object_copies': 16740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8571, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1995}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:31.946 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:47:31.946 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8437", 'reported_seq': 8565, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:26.559939+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:47:26.559939+0000', 'last_peered': '2024-09-16T04:47:26.559939+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:47:26.559939+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8437, 'log_dups_size': 0, 'ondisk_log_size': 8437, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33206162, 'num_objects': 4054, 'num_object_clones': 0, 'num_object_copies': 16216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8434, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1899}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:31.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:47:31.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8296", 'reported_seq': 8383, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:26.512807+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:47:26.512807+0000', 'last_peered': '2024-09-16T04:47:26.512807+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:47:26.512807+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8296, 'log_dups_size': 0, 'ondisk_log_size': 8296, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33053318, 'num_objects': 4035, 'num_object_clones': 0, 'num_object_copies': 16140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4035, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8295, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1930}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:31.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:47:31.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8484", 'reported_seq': 8567, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:25.972910+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:47:25.972910+0000', 'last_peered': '2024-09-16T04:47:25.972910+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:47:25.972910+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8484, 'log_dups_size': 0, 'ondisk_log_size': 8484, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33168006, 'num_objects': 4049, 'num_object_clones': 0, 'num_object_copies': 16196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8483, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1828}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:31.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:47:31.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8628", 'reported_seq': 8756, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:26.638998+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:47:26.638998+0000', 'last_peered': '2024-09-16T04:47:26.638998+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:47:26.638998+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8628, 'log_dups_size': 0, 'ondisk_log_size': 8628, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34026764, 'num_objects': 4154, 'num_object_clones': 0, 'num_object_copies': 16616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4154, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8626, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1956}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:31.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:47:31.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8351", 'reported_seq': 8448, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:25.713569+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:47:25.713569+0000', 'last_peered': '2024-09-16T04:47:25.713569+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:47:25.713569+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8351, 'log_dups_size': 0, 'ondisk_log_size': 8351, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33568012, 'num_objects': 4098, 'num_object_clones': 0, 'num_object_copies': 16392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2126, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4098, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8349, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1974}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:31.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:47:31.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8393", 'reported_seq': 8483, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:26.321378+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:47:26.321378+0000', 'last_peered': '2024-09-16T04:47:26.321378+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:47:26.321378+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8393, 'log_dups_size': 0, 'ondisk_log_size': 8393, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32060684, 'num_objects': 3914, 'num_object_clones': 0, 'num_object_copies': 15656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3914, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8391, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1686}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:31.948 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:47:32.624 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:32.624 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31034/260916 objects degraded (11.894%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":464,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":65229,"data_bytes":534308544,"bytes_used":2839883776,"bytes_avail":383707172864,"bytes_total":386547056640,"degraded_objects":31034,"degraded_total":260916,"degraded_ratio":0.11894249490257401,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":73709,"recovering_keys_per_sec":0,"num_objects_recovered":183,"num_bytes_recovered":884736,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":178},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T04:47:26.684344+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:47:32.658 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:47:32.659 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:33.301 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:33.302 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:33.329 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-16T04:47:32.685775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":526108352,"num_objects":64228,"num_object_clones":0,"num_object_copies":256912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64228,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137259,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4629,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137286,"ondisk_log_size":137286,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2760936,"kb_used_data":1034004,"kb_used_omap":14,"kb_used_meta":1726833,"kb_avail":374726424,"statfs":{"total":386547056640,"available":383719858176,"internally_reserved":0,"allocated":1058820096,"data_stored":1058100220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1768277553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,5,8,18,4,4,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25780224,"num_objects":-3147,"num_object_clones":0,"num_object_copies":-12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3147,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003040"},"pg_stats":[{"pgid":"1.f","version":"33'8612","reported_seq":8724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.691341+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:31.691341+0000","last_peered":"2024-09-16T04:47:31.691341+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:31.691341+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8612,"log_dups_size":0,"ondisk_log_size":8612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33112064,"num_objects":4042,"num_object_clones":0,"num_object_copies":16168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8612,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1856}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8614","reported_seq":8698,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.340764+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:31.340764+0000","last_peered":"2024-09-16T04:47:31.340764+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:31.340764+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8614,"log_dups_size":0,"ondisk_log_size":8614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33256716,"num_objects":4060,"num_object_clones":0,"num_object_copies":16240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8612,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1857}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8617","reported_seq":8711,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:30.910343+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:30.910343+0000","last_peered":"2024-09-16T04:47:30.910343+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:30.910343+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8617,"log_dups_size":0,"ondisk_log_size":8617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33200798,"num_objects":4054,"num_object_clones":0,"num_object_copies":16216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8616,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1835}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8632","reported_seq":8716,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.326319+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:31.326319+0000","last_peered":"2024-09-16T04:47:31.326319+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:31.326319+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8632,"log_dups_size":0,"ondisk_log_size":8632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076492,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8630,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1817}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8726","reported_seq":8854,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.649890+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:31.649890+0000","last_peered":"2024-09-16T04:47:31.649890+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:31.649890+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8726,"log_dups_size":0,"ondisk_log_size":8726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32888076,"num_objects":4015,"num_object_clones":0,"num_object_copies":16060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8724,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1767}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8425","reported_seq":8530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.624853+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:31.624853+0000","last_peered":"2024-09-16T04:47:31.624853+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:31.624853+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31842304,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8425,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8612","reported_seq":8705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.320122+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:31.320122+0000","last_peered":"2024-09-16T04:47:31.320122+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:31.320122+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8612,"log_dups_size":0,"ondisk_log_size":8612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32666892,"num_objects":3988,"num_object_clones":0,"num_object_copies":15952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8610,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1775}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8654","reported_seq":13211,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:27.946838+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:27.946838+0000","last_peered":"2024-09-16T04:47:27.946838+0000","last_clean":"2024-09-16T04:47:27.946838+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:27.946838+0000","last_undegraded":"2024-09-16T04:47:27.946838+0000","last_fullsized":"2024-09-16T04:47:27.946838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8654,"log_dups_size":0,"ondisk_log_size":8654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:29.665540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33838348,"num_objects":4131,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8652,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8785","reported_seq":13043,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.704739+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:31.704739+0000","last_peered":"2024-09-16T04:47:31.704739+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:31.704739+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8785,"log_dups_size":0,"ondisk_log_size":8785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32524824,"num_objects":3971,"num_object_clones":0,"num_object_copies":15884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8782,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2102,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3727}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8641","reported_seq":8743,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:30.985743+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:30.985743+0000","last_peered":"2024-09-16T04:47:30.985743+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:30.985743+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8641,"log_dups_size":0,"ondisk_log_size":8641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33723660,"num_objects":4117,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8639,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1928}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8501","reported_seq":8628,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.666695+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:31.666695+0000","last_peered":"2024-09-16T04:47:31.666695+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:31.666695+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8501,"log_dups_size":0,"ondisk_log_size":8501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32681874,"num_objects":3990,"num_object_clones":0,"num_object_copies":15960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8498,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1835}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8361","reported_seq":8446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.766962+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:31.766962+0000","last_peered":"2024-09-16T04:47:31.766962+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:31.766962+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8361,"log_dups_size":0,"ondisk_log_size":8361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32520838,"num_objects":3970,"num_object_clones":0,"num_object_copies":15880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8360,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1865}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8557","reported_seq":8641,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:30.978024+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:30.978024+0000","last_peered":"2024-09-16T04:47:30.978024+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:30.978024+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8557,"log_dups_size":0,"ondisk_log_size":8557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32569990,"num_objects":3976,"num_object_clones":0,"num_object_copies":15904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8556,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1755}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8696","reported_seq":8824,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.459878+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:31.459878+0000","last_peered":"2024-09-16T04:47:31.459878+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:31.459878+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8696,"log_dups_size":0,"ondisk_log_size":8696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33469708,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8694,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1888}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8403","reported_seq":8502,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:30.895913+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:30.895913+0000","last_peered":"2024-09-16T04:47:30.895913+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:30.895913+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8403,"log_dups_size":0,"ondisk_log_size":8403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33142028,"num_objects":4046,"num_object_clones":0,"num_object_copies":16184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8401,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1923}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8450","reported_seq":8540,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.257820+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:31.257820+0000","last_peered":"2024-09-16T04:47:31.257820+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:31.257820+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8450,"log_dups_size":0,"ondisk_log_size":8450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31593740,"num_objects":3857,"num_object_clones":0,"num_object_copies":15428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8448,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1629}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":526108352,"num_objects":64228,"num_object_clones":0,"num_object_copies":256912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64228,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137259,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4629,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1055039488,"data_stored":1055039488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137286,"ondisk_log_size":137286,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722368,"kb_used_data":257512,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93649472,"statfs":{"total":96636764160,"available":95897059328,"internally_reserved":0,"allocated":263692288,"data_stored":263512319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475983567},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,5,0,11,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730544,"kb_used_data":257876,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93641296,"statfs":{"total":96636764160,"available":95888687104,"internally_reserved":0,"allocated":264065024,"data_stored":263885055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,4,1,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585508,"kb_used_data":260884,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93786332,"statfs":{"total":96636764160,"available":96037203968,"internally_reserved":0,"allocated":267145216,"data_stored":266965247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332394191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722516,"kb_used_data":257732,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93649324,"statfs":{"total":96636764160,"available":95896907776,"internally_reserved":0,"allocated":263917568,"data_stored":263737599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":475918030},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262873088,"data_stored":262873088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266338304,"data_stored":266338304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":263217152,"data_stored":263217152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":262610944,"data_stored":262610944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:36.331 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:36.918 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:36.918 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:36.947 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-09-16T04:47:36.686777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":520857280,"num_objects":63587,"num_object_clones":0,"num_object_copies":254348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63587,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137900,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4629,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137927,"ondisk_log_size":137927,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2743664,"kb_used_data":1020852,"kb_used_omap":14,"kb_used_meta":1722673,"kb_avail":374743696,"statfs":{"total":386547056640,"available":383737544704,"internally_reserved":0,"allocated":1045352448,"data_stored":1044632572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1764017713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,7,10,17,17,11,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22020096,"num_objects":-2688,"num_object_clones":0,"num_object_copies":-10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2688,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":311296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002996"},"pg_stats":[{"pgid":"1.f","version":"33'8612","reported_seq":8724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.691341+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:31.691341+0000","last_peered":"2024-09-16T04:47:31.691341+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:31.691341+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8612,"log_dups_size":0,"ondisk_log_size":8612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33112064,"num_objects":4042,"num_object_clones":0,"num_object_copies":16168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8612,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1856}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8687","reported_seq":8771,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.473517+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:36.473517+0000","last_peered":"2024-09-16T04:47:36.473517+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:36.473517+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8687,"log_dups_size":0,"ondisk_log_size":8687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32658700,"num_objects":3987,"num_object_clones":0,"num_object_copies":15948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8685,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1784}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8696","reported_seq":8790,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.932890+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:35.932890+0000","last_peered":"2024-09-16T04:47:35.932890+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:35.932890+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8696,"log_dups_size":0,"ondisk_log_size":8696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32553630,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8695,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1756}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8701","reported_seq":8785,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.383096+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:36.383096+0000","last_peered":"2024-09-16T04:47:36.383096+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:36.383096+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8701,"log_dups_size":0,"ondisk_log_size":8701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32511244,"num_objects":3969,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8699,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8726","reported_seq":8854,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.649890+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:31.649890+0000","last_peered":"2024-09-16T04:47:31.649890+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:31.649890+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8726,"log_dups_size":0,"ondisk_log_size":8726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32888076,"num_objects":4015,"num_object_clones":0,"num_object_copies":16060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8724,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1767}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8425","reported_seq":8530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.624853+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:31.624853+0000","last_peered":"2024-09-16T04:47:31.624853+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:31.624853+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31842304,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8425,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8703","reported_seq":8795,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.600333+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:36.600333+0000","last_peered":"2024-09-16T04:47:36.600333+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:36.600333+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8703,"log_dups_size":0,"ondisk_log_size":8703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31921420,"num_objects":3897,"num_object_clones":0,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8701,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1684}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8723","reported_seq":13280,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:33.425905+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:33.425905+0000","last_peered":"2024-09-16T04:47:33.425905+0000","last_clean":"2024-09-16T04:47:33.425905+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:33.425905+0000","last_undegraded":"2024-09-16T04:47:33.425905+0000","last_fullsized":"2024-09-16T04:47:33.425905+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8723,"log_dups_size":0,"ondisk_log_size":8723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:34.689485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33273100,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8721,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8785","reported_seq":13043,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.704739+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:31.704739+0000","last_peered":"2024-09-16T04:47:31.704739+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:31.704739+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8785,"log_dups_size":0,"ondisk_log_size":8785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32524824,"num_objects":3971,"num_object_clones":0,"num_object_copies":15884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8782,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2102,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3727}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8721","reported_seq":8822,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.994450+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:35.994450+0000","last_peered":"2024-09-16T04:47:35.994450+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:35.994450+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8721,"log_dups_size":0,"ondisk_log_size":8721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33068300,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8719,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1848}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8501","reported_seq":8628,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.666695+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:31.666695+0000","last_peered":"2024-09-16T04:47:31.666695+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:31.666695+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8501,"log_dups_size":0,"ondisk_log_size":8501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32681874,"num_objects":3990,"num_object_clones":0,"num_object_copies":15960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8498,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1835}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8361","reported_seq":8446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.766962+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:31.766962+0000","last_peered":"2024-09-16T04:47:31.766962+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:31.766962+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8361,"log_dups_size":0,"ondisk_log_size":8361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32520838,"num_objects":3970,"num_object_clones":0,"num_object_copies":15880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8360,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1865}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8621","reported_seq":8705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.523940+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:35.523940+0000","last_peered":"2024-09-16T04:47:35.523940+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:35.523940+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8621,"log_dups_size":0,"ondisk_log_size":8621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32045702,"num_objects":3912,"num_object_clones":0,"num_object_copies":15648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8620,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1691}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8696","reported_seq":8824,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.459878+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:31.459878+0000","last_peered":"2024-09-16T04:47:31.459878+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:31.459878+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8696,"log_dups_size":0,"ondisk_log_size":8696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33469708,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8694,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1888}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8458","reported_seq":8557,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.784579+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:35.784579+0000","last_peered":"2024-09-16T04:47:35.784579+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:35.784579+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8458,"log_dups_size":0,"ondisk_log_size":8458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32691468,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8456,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1868}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8511","reported_seq":8601,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.422806+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:36.422806+0000","last_peered":"2024-09-16T04:47:36.422806+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:36.422806+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8511,"log_dups_size":0,"ondisk_log_size":8511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31094028,"num_objects":3796,"num_object_clones":0,"num_object_copies":15184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8509,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1568}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":520857280,"num_objects":63587,"num_object_clones":0,"num_object_copies":254348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63587,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137900,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4629,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1041256448,"data_stored":1041256448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137927,"ondisk_log_size":137927,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722368,"kb_used_data":257512,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93649472,"statfs":{"total":96636764160,"available":95897059328,"internally_reserved":0,"allocated":263692288,"data_stored":263512319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475983567},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,5,0,11,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713732,"kb_used_data":253520,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93658108,"statfs":{"total":96636764160,"available":95905902592,"internally_reserved":0,"allocated":259604480,"data_stored":259424511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,13,3,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051658,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585340,"kb_used_data":256556,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93786500,"statfs":{"total":96636764160,"available":96037376000,"internally_reserved":0,"allocated":262713344,"data_stored":262533375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336654031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722224,"kb_used_data":253264,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93649616,"statfs":{"total":96636764160,"available":95897206784,"internally_reserved":0,"allocated":259342336,"data_stored":259162367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":480177870},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,2,2,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":258187264,"data_stored":258187264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261758976,"data_stored":261758976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":258699264,"data_stored":258699264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":262610944,"data_stored":262610944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:36.949 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:37.537 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:37.538 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:37.575 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-09-16T04:47:36.686777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":520857280,"num_objects":63587,"num_object_clones":0,"num_object_copies":254348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63587,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137900,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4629,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137927,"ondisk_log_size":137927,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2743664,"kb_used_data":1020852,"kb_used_omap":14,"kb_used_meta":1722673,"kb_avail":374743696,"statfs":{"total":386547056640,"available":383737544704,"internally_reserved":0,"allocated":1045352448,"data_stored":1044632572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1764017713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,7,10,17,17,11,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22020096,"num_objects":-2688,"num_object_clones":0,"num_object_copies":-10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2688,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":311296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002996"},"pg_stats":[{"pgid":"1.f","version":"33'8612","reported_seq":8724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.691341+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:31.691341+0000","last_peered":"2024-09-16T04:47:31.691341+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:31.691341+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8612,"log_dups_size":0,"ondisk_log_size":8612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33112064,"num_objects":4042,"num_object_clones":0,"num_object_copies":16168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8612,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1856}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8687","reported_seq":8771,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.473517+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:36.473517+0000","last_peered":"2024-09-16T04:47:36.473517+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:36.473517+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8687,"log_dups_size":0,"ondisk_log_size":8687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32658700,"num_objects":3987,"num_object_clones":0,"num_object_copies":15948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8685,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1784}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8696","reported_seq":8790,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.932890+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:35.932890+0000","last_peered":"2024-09-16T04:47:35.932890+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:35.932890+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8696,"log_dups_size":0,"ondisk_log_size":8696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32553630,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8695,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1756}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8701","reported_seq":8785,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.383096+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:36.383096+0000","last_peered":"2024-09-16T04:47:36.383096+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:36.383096+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8701,"log_dups_size":0,"ondisk_log_size":8701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32511244,"num_objects":3969,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8699,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8726","reported_seq":8854,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.649890+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:31.649890+0000","last_peered":"2024-09-16T04:47:31.649890+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:31.649890+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8726,"log_dups_size":0,"ondisk_log_size":8726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32888076,"num_objects":4015,"num_object_clones":0,"num_object_copies":16060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8724,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1767}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8425","reported_seq":8530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.624853+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:31.624853+0000","last_peered":"2024-09-16T04:47:31.624853+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:31.624853+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31842304,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8425,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8703","reported_seq":8795,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.600333+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:36.600333+0000","last_peered":"2024-09-16T04:47:36.600333+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:36.600333+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8703,"log_dups_size":0,"ondisk_log_size":8703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31921420,"num_objects":3897,"num_object_clones":0,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8701,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1684}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8723","reported_seq":13280,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:33.425905+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:33.425905+0000","last_peered":"2024-09-16T04:47:33.425905+0000","last_clean":"2024-09-16T04:47:33.425905+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:33.425905+0000","last_undegraded":"2024-09-16T04:47:33.425905+0000","last_fullsized":"2024-09-16T04:47:33.425905+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8723,"log_dups_size":0,"ondisk_log_size":8723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:34.689485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33273100,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8721,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8785","reported_seq":13043,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.704739+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:31.704739+0000","last_peered":"2024-09-16T04:47:31.704739+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:31.704739+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8785,"log_dups_size":0,"ondisk_log_size":8785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32524824,"num_objects":3971,"num_object_clones":0,"num_object_copies":15884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8782,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2102,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3727}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8721","reported_seq":8822,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.994450+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:35.994450+0000","last_peered":"2024-09-16T04:47:35.994450+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:35.994450+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8721,"log_dups_size":0,"ondisk_log_size":8721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33068300,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8719,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1848}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8501","reported_seq":8628,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.666695+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:31.666695+0000","last_peered":"2024-09-16T04:47:31.666695+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:31.666695+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8501,"log_dups_size":0,"ondisk_log_size":8501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32681874,"num_objects":3990,"num_object_clones":0,"num_object_copies":15960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8498,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1835}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8361","reported_seq":8446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.766962+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:31.766962+0000","last_peered":"2024-09-16T04:47:31.766962+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:31.766962+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8361,"log_dups_size":0,"ondisk_log_size":8361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32520838,"num_objects":3970,"num_object_clones":0,"num_object_copies":15880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8360,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1865}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8621","reported_seq":8705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.523940+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:35.523940+0000","last_peered":"2024-09-16T04:47:35.523940+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:35.523940+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8621,"log_dups_size":0,"ondisk_log_size":8621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32045702,"num_objects":3912,"num_object_clones":0,"num_object_copies":15648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8620,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1691}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8696","reported_seq":8824,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:31.459878+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:31.459878+0000","last_peered":"2024-09-16T04:47:31.459878+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:31.459878+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8696,"log_dups_size":0,"ondisk_log_size":8696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33469708,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8694,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1888}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8458","reported_seq":8557,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.784579+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:35.784579+0000","last_peered":"2024-09-16T04:47:35.784579+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:35.784579+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8458,"log_dups_size":0,"ondisk_log_size":8458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32691468,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8456,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1868}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8511","reported_seq":8601,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.422806+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:36.422806+0000","last_peered":"2024-09-16T04:47:36.422806+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:36.422806+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8511,"log_dups_size":0,"ondisk_log_size":8511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31094028,"num_objects":3796,"num_object_clones":0,"num_object_copies":15184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8509,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1568}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":520857280,"num_objects":63587,"num_object_clones":0,"num_object_copies":254348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63587,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137900,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4629,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1041256448,"data_stored":1041256448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137927,"ondisk_log_size":137927,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722368,"kb_used_data":257512,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93649472,"statfs":{"total":96636764160,"available":95897059328,"internally_reserved":0,"allocated":263692288,"data_stored":263512319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475983567},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,5,0,11,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713732,"kb_used_data":253520,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93658108,"statfs":{"total":96636764160,"available":95905902592,"internally_reserved":0,"allocated":259604480,"data_stored":259424511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,13,3,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051658,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585340,"kb_used_data":256556,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93786500,"statfs":{"total":96636764160,"available":96037376000,"internally_reserved":0,"allocated":262713344,"data_stored":262533375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336654031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722224,"kb_used_data":253264,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93649616,"statfs":{"total":96636764160,"available":95897206784,"internally_reserved":0,"allocated":259342336,"data_stored":259162367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":480177870},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,2,2,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":258187264,"data_stored":258187264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261758976,"data_stored":261758976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":258699264,"data_stored":258699264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":262610944,"data_stored":262610944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:37.576 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:47:37.576 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8612", 'reported_seq': 8724, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:31.691341+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:47:31.691341+0000', 'last_peered': '2024-09-16T04:47:31.691341+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:47:31.691341+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8612, 'log_dups_size': 0, 'ondisk_log_size': 8612, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33112064, 'num_objects': 4042, 'num_object_clones': 0, 'num_object_copies': 16168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4042, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8612, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1856}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:37.576 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:47:37.577 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8687", 'reported_seq': 8771, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:36.473517+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:47:36.473517+0000', 'last_peered': '2024-09-16T04:47:36.473517+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:47:36.473517+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8687, 'log_dups_size': 0, 'ondisk_log_size': 8687, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32658700, 'num_objects': 3987, 'num_object_clones': 0, 'num_object_copies': 15948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3987, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8685, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1784}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:37.577 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:47:37.577 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8696", 'reported_seq': 8790, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:35.932890+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:47:35.932890+0000', 'last_peered': '2024-09-16T04:47:35.932890+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:47:35.932890+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8696, 'log_dups_size': 0, 'ondisk_log_size': 8696, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32553630, 'num_objects': 3975, 'num_object_clones': 0, 'num_object_copies': 15900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3975, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8695, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1756}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:37.577 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:47:37.577 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8701", 'reported_seq': 8785, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:36.383096+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:47:36.383096+0000', 'last_peered': '2024-09-16T04:47:36.383096+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:47:36.383096+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8701, 'log_dups_size': 0, 'ondisk_log_size': 8701, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32511244, 'num_objects': 3969, 'num_object_clones': 0, 'num_object_copies': 15876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3969, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8699, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1748}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:37.577 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:47:37.577 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8726", 'reported_seq': 8854, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:31.649890+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:47:31.649890+0000', 'last_peered': '2024-09-16T04:47:31.649890+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:47:31.649890+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8726, 'log_dups_size': 0, 'ondisk_log_size': 8726, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32888076, 'num_objects': 4015, 'num_object_clones': 0, 'num_object_copies': 16060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4015, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8724, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1767}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:37.577 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:47:37.577 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8425", 'reported_seq': 8530, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:31.624853+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:47:31.624853+0000', 'last_peered': '2024-09-16T04:47:31.624853+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:47:31.624853+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8425, 'log_dups_size': 0, 'ondisk_log_size': 8425, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31842304, 'num_objects': 3887, 'num_object_clones': 0, 'num_object_copies': 15548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8425, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1709}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:37.578 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:47:37.578 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8703", 'reported_seq': 8795, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:36.600333+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:47:36.600333+0000', 'last_peered': '2024-09-16T04:47:36.600333+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:47:36.600333+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8703, 'log_dups_size': 0, 'ondisk_log_size': 8703, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31921420, 'num_objects': 3897, 'num_object_clones': 0, 'num_object_copies': 15588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3897, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8701, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1684}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:37.578 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:47:37.578 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8785", 'reported_seq': 13043, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:31.704739+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:47:31.704739+0000', 'last_peered': '2024-09-16T04:47:31.704739+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:47:31.704739+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8785, 'log_dups_size': 0, 'ondisk_log_size': 8785, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32524824, 'num_objects': 3971, 'num_object_clones': 0, 'num_object_copies': 15884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 244, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8782, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2102, 'num_bytes_recovered': 15925248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3727}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:37.578 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:47:37.578 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8721", 'reported_seq': 8822, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:35.994450+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:47:35.994450+0000', 'last_peered': '2024-09-16T04:47:35.994450+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:47:35.994450+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8721, 'log_dups_size': 0, 'ondisk_log_size': 8721, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33068300, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 16148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8719, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1848}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:37.578 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:47:37.578 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8501", 'reported_seq': 8628, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:31.666695+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:47:31.666695+0000', 'last_peered': '2024-09-16T04:47:31.666695+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:47:31.666695+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8501, 'log_dups_size': 0, 'ondisk_log_size': 8501, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32681874, 'num_objects': 3990, 'num_object_clones': 0, 'num_object_copies': 15960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3990, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8498, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1835}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:37.578 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:47:37.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8361", 'reported_seq': 8446, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:31.766962+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:47:31.766962+0000', 'last_peered': '2024-09-16T04:47:31.766962+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:47:31.766962+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8361, 'log_dups_size': 0, 'ondisk_log_size': 8361, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32520838, 'num_objects': 3970, 'num_object_clones': 0, 'num_object_copies': 15880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3970, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8360, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1865}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:37.579 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:47:37.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8621", 'reported_seq': 8705, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:35.523940+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:47:35.523940+0000', 'last_peered': '2024-09-16T04:47:35.523940+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:47:35.523940+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8621, 'log_dups_size': 0, 'ondisk_log_size': 8621, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32045702, 'num_objects': 3912, 'num_object_clones': 0, 'num_object_copies': 15648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8620, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1691}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:37.579 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:47:37.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8696", 'reported_seq': 8824, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:31.459878+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:47:31.459878+0000', 'last_peered': '2024-09-16T04:47:31.459878+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:47:31.459878+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8696, 'log_dups_size': 0, 'ondisk_log_size': 8696, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33469708, 'num_objects': 4086, 'num_object_clones': 0, 'num_object_copies': 16344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8694, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1888}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:37.579 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:47:37.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8458", 'reported_seq': 8557, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:35.784579+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:47:35.784579+0000', 'last_peered': '2024-09-16T04:47:35.784579+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:47:35.784579+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8458, 'log_dups_size': 0, 'ondisk_log_size': 8458, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32691468, 'num_objects': 3991, 'num_object_clones': 0, 'num_object_copies': 15964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3991, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8456, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1868}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:37.579 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:47:37.579 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8511", 'reported_seq': 8601, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:36.422806+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:47:36.422806+0000', 'last_peered': '2024-09-16T04:47:36.422806+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:47:36.422806+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8511, 'log_dups_size': 0, 'ondisk_log_size': 8511, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31094028, 'num_objects': 3796, 'num_object_clones': 0, 'num_object_copies': 15184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3796, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8509, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1568}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:37.580 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:47:38.311 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:38.311 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30953/254348 objects degraded (12.170%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":470,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":63587,"data_bytes":520857280,"bytes_used":2809511936,"bytes_avail":383737544704,"bytes_total":386547056640,"degraded_objects":30953,"degraded_total":254348,"degraded_ratio":0.12169547234497617,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":25934,"recovering_keys_per_sec":0,"num_objects_recovered":190,"num_bytes_recovered":311296,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":223},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T04:47:26.684344+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:47:38.349 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:47:38.349 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:38.966 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:38.969 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:38.996 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-16T04:47:38.687318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":516425408,"num_objects":63046,"num_object_clones":0,"num_object_copies":252184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63046,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138441,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4687,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138468,"ondisk_log_size":138468,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2730608,"kb_used_data":1011940,"kb_used_omap":14,"kb_used_meta":1718513,"kb_avail":374756752,"statfs":{"total":386547056640,"available":383750914048,"internally_reserved":0,"allocated":1036226560,"data_stored":1035506684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1759757873},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,7,16,10,14,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22052864,"num_objects":-2692,"num_object_clones":0,"num_object_copies":-10768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2692,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":311296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003064"},"pg_stats":[{"pgid":"1.f","version":"33'8680","reported_seq":8792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.627019+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:36.627019+0000","last_peered":"2024-09-16T04:47:36.627019+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:36.627019+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8680,"log_dups_size":0,"ondisk_log_size":8680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32555008,"num_objects":3974,"num_object_clones":0,"num_object_copies":15896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8680,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1788}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8687","reported_seq":8771,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.473517+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:36.473517+0000","last_peered":"2024-09-16T04:47:36.473517+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:36.473517+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8687,"log_dups_size":0,"ondisk_log_size":8687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32658700,"num_objects":3987,"num_object_clones":0,"num_object_copies":15948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8685,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1784}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8696","reported_seq":8790,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.932890+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:35.932890+0000","last_peered":"2024-09-16T04:47:35.932890+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:35.932890+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8696,"log_dups_size":0,"ondisk_log_size":8696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32553630,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8695,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1756}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8701","reported_seq":8785,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.383096+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:36.383096+0000","last_peered":"2024-09-16T04:47:36.383096+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:36.383096+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8701,"log_dups_size":0,"ondisk_log_size":8701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32511244,"num_objects":3969,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8699,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8792","reported_seq":8920,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.730902+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:36.730902+0000","last_peered":"2024-09-16T04:47:36.730902+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:36.730902+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32347404,"num_objects":3949,"num_object_clones":0,"num_object_copies":15796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8790,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1701}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8494","reported_seq":8600,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.807783+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:36.807783+0000","last_peered":"2024-09-16T04:47:36.807783+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:36.807783+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8494,"log_dups_size":0,"ondisk_log_size":8494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31277056,"num_objects":3818,"num_object_clones":0,"num_object_copies":15272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8494,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1640}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8703","reported_seq":8795,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.600333+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:36.600333+0000","last_peered":"2024-09-16T04:47:36.600333+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:36.600333+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8703,"log_dups_size":0,"ondisk_log_size":8703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31921420,"num_objects":3897,"num_object_clones":0,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8701,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1684}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8797","reported_seq":13354,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:38.569750+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:38.569750+0000","last_peered":"2024-09-16T04:47:38.569750+0000","last_clean":"2024-09-16T04:47:38.569750+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:38.569750+0000","last_undegraded":"2024-09-16T04:47:38.569750+0000","last_fullsized":"2024-09-16T04:47:38.569750+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8797,"log_dups_size":0,"ondisk_log_size":8797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:39.722133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32666892,"num_objects":3988,"num_object_clones":0,"num_object_copies":15952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8795,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8857","reported_seq":13231,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.822691+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:36.822691+0000","last_peered":"2024-09-16T04:47:36.822691+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:36.822691+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8857,"log_dups_size":0,"ondisk_log_size":8857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31935000,"num_objects":3899,"num_object_clones":0,"num_object_copies":15596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2160,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3713}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8721","reported_seq":8822,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.994450+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:35.994450+0000","last_peered":"2024-09-16T04:47:35.994450+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:35.994450+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8721,"log_dups_size":0,"ondisk_log_size":8721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33068300,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8719,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1848}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8559","reported_seq":8687,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.816021+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:36.816021+0000","last_peered":"2024-09-16T04:47:36.816021+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:36.816021+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8559,"log_dups_size":0,"ondisk_log_size":8559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32206738,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8556,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1777}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8425","reported_seq":8513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.271512+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:36.271512+0000","last_peered":"2024-09-16T04:47:36.271512+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:36.271512+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31996550,"num_objects":3906,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8424,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1801}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8621","reported_seq":8705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.523940+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:35.523940+0000","last_peered":"2024-09-16T04:47:35.523940+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:35.523940+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8621,"log_dups_size":0,"ondisk_log_size":8621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32045702,"num_objects":3912,"num_object_clones":0,"num_object_copies":15648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8620,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1691}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8766","reported_seq":8894,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.562954+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:36.562954+0000","last_peered":"2024-09-16T04:47:36.562954+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:36.562954+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8766,"log_dups_size":0,"ondisk_log_size":8766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32896268,"num_objects":4016,"num_object_clones":0,"num_object_copies":16064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8764,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1818}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8458","reported_seq":8557,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.784579+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:35.784579+0000","last_peered":"2024-09-16T04:47:35.784579+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:35.784579+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8458,"log_dups_size":0,"ondisk_log_size":8458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32691468,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8456,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1868}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8511","reported_seq":8601,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.422806+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:36.422806+0000","last_peered":"2024-09-16T04:47:36.422806+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:36.422806+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8511,"log_dups_size":0,"ondisk_log_size":8511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31094028,"num_objects":3796,"num_object_clones":0,"num_object_copies":15184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8509,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1568}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":516425408,"num_objects":63046,"num_object_clones":0,"num_object_copies":252184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63046,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138441,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4687,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1031933952,"data_stored":1031933952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138468,"ondisk_log_size":138468,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722112,"kb_used_data":253096,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93649728,"statfs":{"total":96636764160,"available":95897321472,"internally_reserved":0,"allocated":259170304,"data_stored":258990335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713732,"kb_used_data":253520,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93658108,"statfs":{"total":96636764160,"available":95905902592,"internally_reserved":0,"allocated":259604480,"data_stored":259424511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,13,3,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572540,"kb_used_data":252060,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93799300,"statfs":{"total":96636764160,"available":96050483200,"internally_reserved":0,"allocated":258109440,"data_stored":257929471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328134351},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722224,"kb_used_data":253264,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93649616,"statfs":{"total":96636764160,"available":95897206784,"internally_reserved":0,"allocated":259342336,"data_stored":259162367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":480177870},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,2,2,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":258187264,"data_stored":258187264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257073152,"data_stored":257073152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":258699264,"data_stored":258699264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":257974272,"data_stored":257974272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:41.998 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:42.586 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:42.586 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:42.617 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-09-16T04:47:40.687848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":516425408,"num_objects":63046,"num_object_clones":0,"num_object_copies":252184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63046,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138441,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4687,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138468,"ondisk_log_size":138468,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2730608,"kb_used_data":1011940,"kb_used_omap":14,"kb_used_meta":1718513,"kb_avail":374756752,"statfs":{"total":386547056640,"available":383750914048,"internally_reserved":0,"allocated":1036226560,"data_stored":1035506684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1759757873},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,7,16,10,14,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17883136,"num_objects":-2183,"num_object_clones":0,"num_object_copies":-8732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2183,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003071"},"pg_stats":[{"pgid":"1.f","version":"33'8680","reported_seq":8792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.627019+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:36.627019+0000","last_peered":"2024-09-16T04:47:36.627019+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:36.627019+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8680,"log_dups_size":0,"ondisk_log_size":8680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32555008,"num_objects":3974,"num_object_clones":0,"num_object_copies":15896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8680,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1788}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8687","reported_seq":8771,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.473517+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:36.473517+0000","last_peered":"2024-09-16T04:47:36.473517+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:36.473517+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8687,"log_dups_size":0,"ondisk_log_size":8687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32658700,"num_objects":3987,"num_object_clones":0,"num_object_copies":15948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8685,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1784}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8696","reported_seq":8790,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.932890+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:35.932890+0000","last_peered":"2024-09-16T04:47:35.932890+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:35.932890+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8696,"log_dups_size":0,"ondisk_log_size":8696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32553630,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8695,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1756}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8701","reported_seq":8785,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.383096+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:36.383096+0000","last_peered":"2024-09-16T04:47:36.383096+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:36.383096+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8701,"log_dups_size":0,"ondisk_log_size":8701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32511244,"num_objects":3969,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8699,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1748}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8792","reported_seq":8920,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.730902+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:36.730902+0000","last_peered":"2024-09-16T04:47:36.730902+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:36.730902+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32347404,"num_objects":3949,"num_object_clones":0,"num_object_copies":15796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8790,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1701}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8494","reported_seq":8600,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.807783+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:36.807783+0000","last_peered":"2024-09-16T04:47:36.807783+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:36.807783+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8494,"log_dups_size":0,"ondisk_log_size":8494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31277056,"num_objects":3818,"num_object_clones":0,"num_object_copies":15272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8494,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1640}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8703","reported_seq":8795,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.600333+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:36.600333+0000","last_peered":"2024-09-16T04:47:36.600333+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:36.600333+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8703,"log_dups_size":0,"ondisk_log_size":8703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31921420,"num_objects":3897,"num_object_clones":0,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8701,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1684}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8797","reported_seq":13354,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:38.569750+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:38.569750+0000","last_peered":"2024-09-16T04:47:38.569750+0000","last_clean":"2024-09-16T04:47:38.569750+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:38.569750+0000","last_undegraded":"2024-09-16T04:47:38.569750+0000","last_fullsized":"2024-09-16T04:47:38.569750+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8797,"log_dups_size":0,"ondisk_log_size":8797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:39.722133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32666892,"num_objects":3988,"num_object_clones":0,"num_object_copies":15952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8795,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8857","reported_seq":13231,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.822691+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:36.822691+0000","last_peered":"2024-09-16T04:47:36.822691+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:36.822691+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8857,"log_dups_size":0,"ondisk_log_size":8857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31935000,"num_objects":3899,"num_object_clones":0,"num_object_copies":15596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2160,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3713}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8721","reported_seq":8822,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.994450+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:35.994450+0000","last_peered":"2024-09-16T04:47:35.994450+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:35.994450+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8721,"log_dups_size":0,"ondisk_log_size":8721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33068300,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8719,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1848}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8559","reported_seq":8687,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.816021+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:36.816021+0000","last_peered":"2024-09-16T04:47:36.816021+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:36.816021+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8559,"log_dups_size":0,"ondisk_log_size":8559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32206738,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8556,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1777}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8425","reported_seq":8513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.271512+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:36.271512+0000","last_peered":"2024-09-16T04:47:36.271512+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:36.271512+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31996550,"num_objects":3906,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8424,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1801}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8621","reported_seq":8705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.523940+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:35.523940+0000","last_peered":"2024-09-16T04:47:35.523940+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:35.523940+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8621,"log_dups_size":0,"ondisk_log_size":8621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32045702,"num_objects":3912,"num_object_clones":0,"num_object_copies":15648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8620,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1691}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8766","reported_seq":8894,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.562954+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:36.562954+0000","last_peered":"2024-09-16T04:47:36.562954+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:36.562954+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8766,"log_dups_size":0,"ondisk_log_size":8766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32896268,"num_objects":4016,"num_object_clones":0,"num_object_copies":16064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8764,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1818}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8458","reported_seq":8557,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:35.784579+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:35.784579+0000","last_peered":"2024-09-16T04:47:35.784579+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:35.784579+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8458,"log_dups_size":0,"ondisk_log_size":8458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32691468,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8456,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1868}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8511","reported_seq":8601,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:36.422806+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:36.422806+0000","last_peered":"2024-09-16T04:47:36.422806+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:36.422806+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8511,"log_dups_size":0,"ondisk_log_size":8511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31094028,"num_objects":3796,"num_object_clones":0,"num_object_copies":15184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8509,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1568}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":516425408,"num_objects":63046,"num_object_clones":0,"num_object_copies":252184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63046,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138441,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4687,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1031933952,"data_stored":1031933952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138468,"ondisk_log_size":138468,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722112,"kb_used_data":253096,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93649728,"statfs":{"total":96636764160,"available":95897321472,"internally_reserved":0,"allocated":259170304,"data_stored":258990335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713732,"kb_used_data":253520,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93658108,"statfs":{"total":96636764160,"available":95905902592,"internally_reserved":0,"allocated":259604480,"data_stored":259424511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,13,3,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572540,"kb_used_data":252060,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93799300,"statfs":{"total":96636764160,"available":96050483200,"internally_reserved":0,"allocated":258109440,"data_stored":257929471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328134351},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722224,"kb_used_data":253264,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93649616,"statfs":{"total":96636764160,"available":95897206784,"internally_reserved":0,"allocated":259342336,"data_stored":259162367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":480177870},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,2,2,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":258187264,"data_stored":258187264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257073152,"data_stored":257073152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":258699264,"data_stored":258699264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":257974272,"data_stored":257974272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:42.619 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:43.163 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:43.163 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:43.207 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2024-09-16T04:47:42.688329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":507930304,"num_objects":62009,"num_object_clones":0,"num_object_copies":248036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30830,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62009,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139478,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4752,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139505,"ondisk_log_size":139505,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2729052,"kb_used_data":997944,"kb_used_omap":14,"kb_used_meta":1730993,"kb_avail":374758308,"statfs":{"total":386547056640,"available":383752507392,"internally_reserved":0,"allocated":1021894656,"data_stored":1021174780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1772537393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,4,15,18,22,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26378240,"num_objects":-3220,"num_object_clones":0,"num_object_copies":-12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3220,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003027"},"pg_stats":[{"pgid":"1.f","version":"33'8742","reported_seq":8854,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.746805+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:41.746805+0000","last_peered":"2024-09-16T04:47:41.746805+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:41.746805+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8742,"log_dups_size":0,"ondisk_log_size":8742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32047104,"num_objects":3912,"num_object_clones":0,"num_object_copies":15648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8742,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8752","reported_seq":8835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.605468+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:41.605468+0000","last_peered":"2024-09-16T04:47:41.605468+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:41.605468+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8752,"log_dups_size":0,"ondisk_log_size":8752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32126220,"num_objects":3922,"num_object_clones":0,"num_object_copies":15688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8750,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1719}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8773","reported_seq":8864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.068607+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:41.068607+0000","last_peered":"2024-09-16T04:47:41.068607+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:41.068607+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8773,"log_dups_size":0,"ondisk_log_size":8773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31922846,"num_objects":3898,"num_object_clones":0,"num_object_copies":15592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8772,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1679}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8782","reported_seq":8864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.628749+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:41.628749+0000","last_peered":"2024-09-16T04:47:41.628749+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:41.628749+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8782,"log_dups_size":0,"ondisk_log_size":8782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31847692,"num_objects":3888,"num_object_clones":0,"num_object_copies":15552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8780,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8861","reported_seq":8989,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.823500+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:41.823500+0000","last_peered":"2024-09-16T04:47:41.823500+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:41.823500+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8861,"log_dups_size":0,"ondisk_log_size":8861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31782156,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8859,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8556","reported_seq":8662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.691101+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:41.691101+0000","last_peered":"2024-09-16T04:47:41.691101+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:41.691101+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8556,"log_dups_size":0,"ondisk_log_size":8556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30769152,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8556,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1578}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8754","reported_seq":8846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.541058+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:41.541058+0000","last_peered":"2024-09-16T04:47:41.541058+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:41.541058+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8754,"log_dups_size":0,"ondisk_log_size":8754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31503628,"num_objects":3846,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8752,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1633}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8797","reported_seq":13354,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:38.569750+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:38.569750+0000","last_peered":"2024-09-16T04:47:38.569750+0000","last_clean":"2024-09-16T04:47:38.569750+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:38.569750+0000","last_undegraded":"2024-09-16T04:47:38.569750+0000","last_fullsized":"2024-09-16T04:47:38.569750+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8797,"log_dups_size":0,"ondisk_log_size":8797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:39.722133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32666892,"num_objects":3988,"num_object_clones":0,"num_object_copies":15952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8795,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8930","reported_seq":13434,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.869955+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:41.869955+0000","last_peered":"2024-09-16T04:47:41.869955+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:41.869955+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8930,"log_dups_size":0,"ondisk_log_size":8930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31336984,"num_objects":3826,"num_object_clones":0,"num_object_copies":15304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8927,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2225,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3705}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8783","reported_seq":8885,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.110192+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:41.110192+0000","last_peered":"2024-09-16T04:47:41.110192+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:41.110192+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8783,"log_dups_size":0,"ondisk_log_size":8783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32560396,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8781,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1786}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8639","reported_seq":8766,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.845703+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:41.845703+0000","last_peered":"2024-09-16T04:47:41.845703+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:41.845703+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8639,"log_dups_size":0,"ondisk_log_size":8639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31551378,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8636,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8488","reported_seq":8576,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.667795+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:41.667795+0000","last_peered":"2024-09-16T04:47:41.667795+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:41.667795+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8488,"log_dups_size":0,"ondisk_log_size":8488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31480454,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8487,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1738}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8696","reported_seq":8780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.008871+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:41.008871+0000","last_peered":"2024-09-16T04:47:41.008871+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:41.008871+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8696,"log_dups_size":0,"ondisk_log_size":8696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31431302,"num_objects":3837,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8695,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1616}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8836","reported_seq":8964,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.822316+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:41.822316+0000","last_peered":"2024-09-16T04:47:41.822316+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:41.822316+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8836,"log_dups_size":0,"ondisk_log_size":8836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32322828,"num_objects":3946,"num_object_clones":0,"num_object_copies":15784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1748}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8531","reported_seq":8629,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:40.977968+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:40.977968+0000","last_peered":"2024-09-16T04:47:40.977968+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:40.977968+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8531,"log_dups_size":0,"ondisk_log_size":8531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32093452,"num_objects":3918,"num_object_clones":0,"num_object_copies":15672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8529,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1795}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8585","reported_seq":8674,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.633898+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:41.633898+0000","last_peered":"2024-09-16T04:47:41.633898+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:41.633898+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8585,"log_dups_size":0,"ondisk_log_size":8585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30487820,"num_objects":3722,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8583,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1494}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":507930304,"num_objects":62009,"num_object_clones":0,"num_object_copies":248036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30830,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62009,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139478,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4752,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1018482688,"data_stored":1018482688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139505,"ondisk_log_size":139505,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721872,"kb_used_data":248700,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93649968,"statfs":{"total":96636764160,"available":95897567232,"internally_reserved":0,"allocated":254668800,"data_stored":254488831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713440,"kb_used_data":249092,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93658400,"statfs":{"total":96636764160,"available":95906201600,"internally_reserved":0,"allocated":255070208,"data_stored":254890239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,2,14,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572540,"kb_used_data":252060,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93799300,"statfs":{"total":96636764160,"available":96050483200,"internally_reserved":0,"allocated":258109440,"data_stored":257929471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328134351},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721200,"kb_used_data":248092,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93650640,"statfs":{"total":96636764160,"available":95898255360,"internally_reserved":0,"allocated":254046208,"data_stored":253866239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":484437710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,5,6,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":253747200,"data_stored":253747200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257073152,"data_stored":257073152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254173184,"data_stored":254173184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253489152,"data_stored":253489152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:43.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:47:43.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8742", 'reported_seq': 8854, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:41.746805+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:47:41.746805+0000', 'last_peered': '2024-09-16T04:47:41.746805+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:47:41.746805+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8742, 'log_dups_size': 0, 'ondisk_log_size': 8742, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32047104, 'num_objects': 3912, 'num_object_clones': 0, 'num_object_copies': 15648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8742, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1726}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:43.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:47:43.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8752", 'reported_seq': 8835, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:41.605468+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:47:41.605468+0000', 'last_peered': '2024-09-16T04:47:41.605468+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:47:41.605468+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8752, 'log_dups_size': 0, 'ondisk_log_size': 8752, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32126220, 'num_objects': 3922, 'num_object_clones': 0, 'num_object_copies': 15688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8750, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1719}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:43.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:47:43.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8773", 'reported_seq': 8864, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:41.068607+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:47:41.068607+0000', 'last_peered': '2024-09-16T04:47:41.068607+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:47:41.068607+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8773, 'log_dups_size': 0, 'ondisk_log_size': 8773, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31922846, 'num_objects': 3898, 'num_object_clones': 0, 'num_object_copies': 15592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3898, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8772, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1679}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:43.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:47:43.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8782", 'reported_seq': 8864, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:41.628749+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:47:41.628749+0000', 'last_peered': '2024-09-16T04:47:41.628749+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:47:41.628749+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8782, 'log_dups_size': 0, 'ondisk_log_size': 8782, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31847692, 'num_objects': 3888, 'num_object_clones': 0, 'num_object_copies': 15552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3888, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8780, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1667}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:43.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:47:43.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8861", 'reported_seq': 8989, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:41.823500+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:47:41.823500+0000', 'last_peered': '2024-09-16T04:47:41.823500+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:47:41.823500+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8861, 'log_dups_size': 0, 'ondisk_log_size': 8861, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31782156, 'num_objects': 3880, 'num_object_clones': 0, 'num_object_copies': 15520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8859, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1632}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:43.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:47:43.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8556", 'reported_seq': 8662, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:41.691101+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:47:41.691101+0000', 'last_peered': '2024-09-16T04:47:41.691101+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:47:41.691101+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8556, 'log_dups_size': 0, 'ondisk_log_size': 8556, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30769152, 'num_objects': 3756, 'num_object_clones': 0, 'num_object_copies': 15024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8556, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1578}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:43.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:47:43.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8754", 'reported_seq': 8846, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:41.541058+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:47:41.541058+0000', 'last_peered': '2024-09-16T04:47:41.541058+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:47:41.541058+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8754, 'log_dups_size': 0, 'ondisk_log_size': 8754, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31503628, 'num_objects': 3846, 'num_object_clones': 0, 'num_object_copies': 15384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3846, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8752, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1633}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:43.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:47:43.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8930", 'reported_seq': 13434, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:41.869955+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:47:41.869955+0000', 'last_peered': '2024-09-16T04:47:41.869955+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:47:41.869955+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8930, 'log_dups_size': 0, 'ondisk_log_size': 8930, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31336984, 'num_objects': 3826, 'num_object_clones': 0, 'num_object_copies': 15304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 121, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8927, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2225, 'num_bytes_recovered': 15925248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3705}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:43.211 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:47:43.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8783", 'reported_seq': 8885, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:41.110192+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:47:41.110192+0000', 'last_peered': '2024-09-16T04:47:41.110192+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:47:41.110192+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8783, 'log_dups_size': 0, 'ondisk_log_size': 8783, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32560396, 'num_objects': 3975, 'num_object_clones': 0, 'num_object_copies': 15900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8781, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1786}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:43.211 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:47:43.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8639", 'reported_seq': 8766, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:41.845703+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:47:41.845703+0000', 'last_peered': '2024-09-16T04:47:41.845703+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:47:41.845703+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8639, 'log_dups_size': 0, 'ondisk_log_size': 8639, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31551378, 'num_objects': 3852, 'num_object_clones': 0, 'num_object_copies': 15408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8636, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1697}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:43.212 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:47:43.212 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8488", 'reported_seq': 8576, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:41.667795+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:47:41.667795+0000', 'last_peered': '2024-09-16T04:47:41.667795+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:47:41.667795+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8488, 'log_dups_size': 0, 'ondisk_log_size': 8488, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31480454, 'num_objects': 3843, 'num_object_clones': 0, 'num_object_copies': 15372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8487, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1738}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:43.212 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:47:43.212 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8696", 'reported_seq': 8780, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:41.008871+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:47:41.008871+0000', 'last_peered': '2024-09-16T04:47:41.008871+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:47:41.008871+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8696, 'log_dups_size': 0, 'ondisk_log_size': 8696, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31431302, 'num_objects': 3837, 'num_object_clones': 0, 'num_object_copies': 15348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3837, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8695, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1616}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:43.213 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:47:43.213 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8836", 'reported_seq': 8964, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:41.822316+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:47:41.822316+0000', 'last_peered': '2024-09-16T04:47:41.822316+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:47:41.822316+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8836, 'log_dups_size': 0, 'ondisk_log_size': 8836, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32322828, 'num_objects': 3946, 'num_object_clones': 0, 'num_object_copies': 15784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3946, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8834, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1748}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:43.213 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:47:43.213 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8531", 'reported_seq': 8629, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:40.977968+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:47:40.977968+0000', 'last_peered': '2024-09-16T04:47:40.977968+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:47:40.977968+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8531, 'log_dups_size': 0, 'ondisk_log_size': 8531, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32093452, 'num_objects': 3918, 'num_object_clones': 0, 'num_object_copies': 15672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3918, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8529, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1795}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:43.213 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:47:43.213 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8585", 'reported_seq': 8674, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:41.633898+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:47:41.633898+0000', 'last_peered': '2024-09-16T04:47:41.633898+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:47:41.633898+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8585, 'log_dups_size': 0, 'ondisk_log_size': 8585, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30487820, 'num_objects': 3722, 'num_object_clones': 0, 'num_object_copies': 14888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8583, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1494}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:43.213 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:47:43.965 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:43.965 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30830/248036 objects degraded (12.430%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":476,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":62009,"data_bytes":507930304,"bytes_used":2794549248,"bytes_avail":383752507392,"bytes_total":386547056640,"degraded_objects":30830,"degraded_total":248036,"degraded_ratio":0.12429647309261557,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":204,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":268},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T04:47:26.684344+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:47:43.990 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:47:43.990 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:44.589 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:44.590 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:44.621 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2024-09-16T04:47:42.688329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":507930304,"num_objects":62009,"num_object_clones":0,"num_object_copies":248036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30830,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62009,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139478,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4752,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139505,"ondisk_log_size":139505,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2729052,"kb_used_data":997944,"kb_used_omap":14,"kb_used_meta":1730993,"kb_avail":374758308,"statfs":{"total":386547056640,"available":383752507392,"internally_reserved":0,"allocated":1021894656,"data_stored":1021174780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1772537393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,4,15,18,22,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26378240,"num_objects":-3220,"num_object_clones":0,"num_object_copies":-12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3220,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003027"},"pg_stats":[{"pgid":"1.f","version":"33'8742","reported_seq":8854,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.746805+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:41.746805+0000","last_peered":"2024-09-16T04:47:41.746805+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:41.746805+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8742,"log_dups_size":0,"ondisk_log_size":8742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32047104,"num_objects":3912,"num_object_clones":0,"num_object_copies":15648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8742,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8752","reported_seq":8835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.605468+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:41.605468+0000","last_peered":"2024-09-16T04:47:41.605468+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:41.605468+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8752,"log_dups_size":0,"ondisk_log_size":8752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32126220,"num_objects":3922,"num_object_clones":0,"num_object_copies":15688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8750,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1719}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8773","reported_seq":8864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.068607+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:41.068607+0000","last_peered":"2024-09-16T04:47:41.068607+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:41.068607+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8773,"log_dups_size":0,"ondisk_log_size":8773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31922846,"num_objects":3898,"num_object_clones":0,"num_object_copies":15592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8772,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1679}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8782","reported_seq":8864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.628749+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:41.628749+0000","last_peered":"2024-09-16T04:47:41.628749+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:41.628749+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8782,"log_dups_size":0,"ondisk_log_size":8782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31847692,"num_objects":3888,"num_object_clones":0,"num_object_copies":15552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8780,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8861","reported_seq":8989,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.823500+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:41.823500+0000","last_peered":"2024-09-16T04:47:41.823500+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:41.823500+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8861,"log_dups_size":0,"ondisk_log_size":8861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31782156,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8859,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8556","reported_seq":8662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.691101+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:41.691101+0000","last_peered":"2024-09-16T04:47:41.691101+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:41.691101+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8556,"log_dups_size":0,"ondisk_log_size":8556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30769152,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8556,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1578}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8754","reported_seq":8846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.541058+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:41.541058+0000","last_peered":"2024-09-16T04:47:41.541058+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:41.541058+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8754,"log_dups_size":0,"ondisk_log_size":8754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31503628,"num_objects":3846,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8752,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1633}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8797","reported_seq":13354,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:38.569750+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:38.569750+0000","last_peered":"2024-09-16T04:47:38.569750+0000","last_clean":"2024-09-16T04:47:38.569750+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:38.569750+0000","last_undegraded":"2024-09-16T04:47:38.569750+0000","last_fullsized":"2024-09-16T04:47:38.569750+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8797,"log_dups_size":0,"ondisk_log_size":8797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:39.722133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32666892,"num_objects":3988,"num_object_clones":0,"num_object_copies":15952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8795,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8930","reported_seq":13434,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.869955+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:41.869955+0000","last_peered":"2024-09-16T04:47:41.869955+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:41.869955+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8930,"log_dups_size":0,"ondisk_log_size":8930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31336984,"num_objects":3826,"num_object_clones":0,"num_object_copies":15304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8927,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2225,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3705}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8783","reported_seq":8885,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.110192+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:41.110192+0000","last_peered":"2024-09-16T04:47:41.110192+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:41.110192+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8783,"log_dups_size":0,"ondisk_log_size":8783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32560396,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8781,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1786}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8639","reported_seq":8766,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.845703+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:41.845703+0000","last_peered":"2024-09-16T04:47:41.845703+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:41.845703+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8639,"log_dups_size":0,"ondisk_log_size":8639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31551378,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8636,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8488","reported_seq":8576,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.667795+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:41.667795+0000","last_peered":"2024-09-16T04:47:41.667795+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:41.667795+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8488,"log_dups_size":0,"ondisk_log_size":8488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31480454,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8487,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1738}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8696","reported_seq":8780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.008871+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:41.008871+0000","last_peered":"2024-09-16T04:47:41.008871+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:41.008871+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8696,"log_dups_size":0,"ondisk_log_size":8696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31431302,"num_objects":3837,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8695,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1616}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8836","reported_seq":8964,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.822316+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:41.822316+0000","last_peered":"2024-09-16T04:47:41.822316+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:41.822316+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8836,"log_dups_size":0,"ondisk_log_size":8836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32322828,"num_objects":3946,"num_object_clones":0,"num_object_copies":15784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1748}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8531","reported_seq":8629,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:40.977968+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:40.977968+0000","last_peered":"2024-09-16T04:47:40.977968+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:40.977968+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8531,"log_dups_size":0,"ondisk_log_size":8531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32093452,"num_objects":3918,"num_object_clones":0,"num_object_copies":15672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8529,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1795}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8585","reported_seq":8674,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.633898+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:41.633898+0000","last_peered":"2024-09-16T04:47:41.633898+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:41.633898+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8585,"log_dups_size":0,"ondisk_log_size":8585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30487820,"num_objects":3722,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8583,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1494}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":507930304,"num_objects":62009,"num_object_clones":0,"num_object_copies":248036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30830,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62009,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139478,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4752,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1018482688,"data_stored":1018482688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139505,"ondisk_log_size":139505,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721872,"kb_used_data":248700,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93649968,"statfs":{"total":96636764160,"available":95897567232,"internally_reserved":0,"allocated":254668800,"data_stored":254488831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713440,"kb_used_data":249092,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93658400,"statfs":{"total":96636764160,"available":95906201600,"internally_reserved":0,"allocated":255070208,"data_stored":254890239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,2,14,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572540,"kb_used_data":252060,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93799300,"statfs":{"total":96636764160,"available":96050483200,"internally_reserved":0,"allocated":258109440,"data_stored":257929471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328134351},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721200,"kb_used_data":248092,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93650640,"statfs":{"total":96636764160,"available":95898255360,"internally_reserved":0,"allocated":254046208,"data_stored":253866239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":484437710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,5,6,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":253747200,"data_stored":253747200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257073152,"data_stored":257073152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254173184,"data_stored":254173184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253489152,"data_stored":253489152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:47.624 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:48.248 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:48.307 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:48.308 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-09-16T04:47:46.689339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":505095872,"num_objects":61663,"num_object_clones":0,"num_object_copies":246652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30830,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61663,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139824,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4752,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139851,"ondisk_log_size":139851,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2746668,"kb_used_data":989220,"kb_used_omap":14,"kb_used_meta":1757361,"kb_avail":374740692,"statfs":{"total":386547056640,"available":383734468608,"internally_reserved":0,"allocated":1012961280,"data_stored":1012241404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1799538225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,1,10,7,18,15,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20447232,"num_objects":-2496,"num_object_clones":0,"num_object_copies":-9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2496,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003021"},"pg_stats":[{"pgid":"1.f","version":"33'8742","reported_seq":8854,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.746805+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:41.746805+0000","last_peered":"2024-09-16T04:47:41.746805+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:41.746805+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8742,"log_dups_size":0,"ondisk_log_size":8742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32047104,"num_objects":3912,"num_object_clones":0,"num_object_copies":15648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8742,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8752","reported_seq":8835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.605468+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:41.605468+0000","last_peered":"2024-09-16T04:47:41.605468+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:41.605468+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8752,"log_dups_size":0,"ondisk_log_size":8752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32126220,"num_objects":3922,"num_object_clones":0,"num_object_copies":15688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8750,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1719}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8843","reported_seq":8937,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.048889+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:46.048889+0000","last_peered":"2024-09-16T04:47:46.048889+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:46.048889+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8843,"log_dups_size":0,"ondisk_log_size":8843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31349406,"num_objects":3828,"num_object_clones":0,"num_object_copies":15312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3828,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8842,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1609}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8782","reported_seq":8864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.628749+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:41.628749+0000","last_peered":"2024-09-16T04:47:41.628749+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:41.628749+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8782,"log_dups_size":0,"ondisk_log_size":8782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31847692,"num_objects":3888,"num_object_clones":0,"num_object_copies":15552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8780,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8861","reported_seq":8989,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.823500+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:41.823500+0000","last_peered":"2024-09-16T04:47:41.823500+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:41.823500+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8861,"log_dups_size":0,"ondisk_log_size":8861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31782156,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8859,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8556","reported_seq":8662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.691101+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:41.691101+0000","last_peered":"2024-09-16T04:47:41.691101+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:41.691101+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8556,"log_dups_size":0,"ondisk_log_size":8556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30769152,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8556,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1578}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8754","reported_seq":8846,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.541058+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:41.541058+0000","last_peered":"2024-09-16T04:47:41.541058+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:41.541058+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8754,"log_dups_size":0,"ondisk_log_size":8754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31503628,"num_objects":3846,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8752,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1633}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8869","reported_seq":13426,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:43.490011+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:43.490011+0000","last_peered":"2024-09-16T04:47:43.490011+0000","last_clean":"2024-09-16T04:47:43.490011+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:43.490011+0000","last_undegraded":"2024-09-16T04:47:43.490011+0000","last_fullsized":"2024-09-16T04:47:43.490011+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8869,"log_dups_size":0,"ondisk_log_size":8869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:44.859841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32077068,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8867,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8930","reported_seq":13434,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.869955+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:41.869955+0000","last_peered":"2024-09-16T04:47:41.869955+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:41.869955+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8930,"log_dups_size":0,"ondisk_log_size":8930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31336984,"num_objects":3826,"num_object_clones":0,"num_object_copies":15304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8927,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2225,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3705}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8853","reported_seq":8955,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.100647+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:46.100647+0000","last_peered":"2024-09-16T04:47:46.100647+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:46.100647+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8853,"log_dups_size":0,"ondisk_log_size":8853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31986956,"num_objects":3905,"num_object_clones":0,"num_object_copies":15620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8851,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1716}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8639","reported_seq":8766,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.845703+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:41.845703+0000","last_peered":"2024-09-16T04:47:41.845703+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:41.845703+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8639,"log_dups_size":0,"ondisk_log_size":8639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31551378,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8636,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8488","reported_seq":8576,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.667795+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:41.667795+0000","last_peered":"2024-09-16T04:47:41.667795+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:41.667795+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8488,"log_dups_size":0,"ondisk_log_size":8488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31480454,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8487,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1738}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8756","reported_seq":8839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.100088+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:46.100088+0000","last_peered":"2024-09-16T04:47:46.100088+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:46.100088+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8756,"log_dups_size":0,"ondisk_log_size":8756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30939782,"num_objects":3777,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8755,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1556}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8836","reported_seq":8964,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.822316+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:41.822316+0000","last_peered":"2024-09-16T04:47:41.822316+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:41.822316+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8836,"log_dups_size":0,"ondisk_log_size":8836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32322828,"num_objects":3946,"num_object_clones":0,"num_object_copies":15784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1748}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8605","reported_seq":8704,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.209166+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:46.209166+0000","last_peered":"2024-09-16T04:47:46.209166+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:46.209166+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8605,"log_dups_size":0,"ondisk_log_size":8605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31487244,"num_objects":3844,"num_object_clones":0,"num_object_copies":15376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8603,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8585","reported_seq":8674,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:41.633898+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:41.633898+0000","last_peered":"2024-09-16T04:47:41.633898+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:41.633898+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8585,"log_dups_size":0,"ondisk_log_size":8585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30487820,"num_objects":3722,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8583,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1494}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":505095872,"num_objects":61663,"num_object_clones":0,"num_object_copies":246652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30830,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61663,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139824,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4752,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1009635328,"data_stored":1009635328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139851,"ondisk_log_size":139851,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721872,"kb_used_data":248700,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93649968,"statfs":{"total":96636764160,"available":95897567232,"internally_reserved":0,"allocated":254668800,"data_stored":254488831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731332,"kb_used_data":244788,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93640508,"statfs":{"total":96636764160,"available":95887880192,"internally_reserved":0,"allocated":250662912,"data_stored":250482943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,0,6,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051660,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572264,"kb_used_data":247640,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93799576,"statfs":{"total":96636764160,"available":96050765824,"internally_reserved":0,"allocated":253583360,"data_stored":253403391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332394191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721200,"kb_used_data":248092,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93650640,"statfs":{"total":96636764160,"available":95898255360,"internally_reserved":0,"allocated":254046208,"data_stored":253866239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":484437710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,5,6,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":253747200,"data_stored":253747200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252678144,"data_stored":252678144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":249720832,"data_stored":249720832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253489152,"data_stored":253489152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:48.309 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:48.915 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:48.916 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:48.950 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-09-16T04:47:48.691046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":499115712,"num_objects":60933,"num_object_clones":0,"num_object_copies":243732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60933,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140554,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4835,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140581,"ondisk_log_size":140581,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2761716,"kb_used_data":980660,"kb_used_omap":14,"kb_used_meta":1780977,"kb_avail":374725644,"statfs":{"total":386547056640,"available":383719059456,"internally_reserved":0,"allocated":1004195840,"data_stored":1003475964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1823721009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,14,13,5,16,16,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21741568,"num_objects":-2654,"num_object_clones":0,"num_object_copies":-10616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2654,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004268"},"pg_stats":[{"pgid":"1.f","version":"33'8818","reported_seq":8930,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.931028+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:46.931028+0000","last_peered":"2024-09-16T04:47:46.931028+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:46.931028+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8818,"log_dups_size":0,"ondisk_log_size":8818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31424512,"num_objects":3836,"num_object_clones":0,"num_object_copies":15344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8818,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8822","reported_seq":8904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.607950+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:46.607950+0000","last_peered":"2024-09-16T04:47:46.607950+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:46.607950+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8822,"log_dups_size":0,"ondisk_log_size":8822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31552780,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8820,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1649}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8843","reported_seq":8937,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.048889+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:46.048889+0000","last_peered":"2024-09-16T04:47:46.048889+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:46.048889+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8843,"log_dups_size":0,"ondisk_log_size":8843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31349406,"num_objects":3828,"num_object_clones":0,"num_object_copies":15312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3828,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8842,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1609}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8846","reported_seq":8930,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.652734+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:46.652734+0000","last_peered":"2024-09-16T04:47:46.652734+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:46.652734+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8846,"log_dups_size":0,"ondisk_log_size":8846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31323404,"num_objects":3824,"num_object_clones":0,"num_object_copies":15296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8844,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1603}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8922","reported_seq":9050,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.943113+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:46.943113+0000","last_peered":"2024-09-16T04:47:46.943113+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:46.943113+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31282444,"num_objects":3819,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8920,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1571}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8618","reported_seq":8723,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.928601+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:46.928601+0000","last_peered":"2024-09-16T04:47:46.928601+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:46.928601+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8618,"log_dups_size":0,"ondisk_log_size":8618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30261248,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8618,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8834","reported_seq":8927,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.660596+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:46.660596+0000","last_peered":"2024-09-16T04:47:46.660596+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:46.660596+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8834,"log_dups_size":0,"ondisk_log_size":8834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30848268,"num_objects":3766,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8832,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1553}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8869","reported_seq":13426,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:43.490011+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:43.490011+0000","last_peered":"2024-09-16T04:47:43.490011+0000","last_clean":"2024-09-16T04:47:43.490011+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:43.490011+0000","last_undegraded":"2024-09-16T04:47:43.490011+0000","last_fullsized":"2024-09-16T04:47:43.490011+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8869,"log_dups_size":0,"ondisk_log_size":8869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:44.859841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32077068,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8867,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8993","reported_seq":13662,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.954836+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:46.954836+0000","last_peered":"2024-09-16T04:47:46.954836+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:46.954836+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8993,"log_dups_size":0,"ondisk_log_size":8993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30820888,"num_objects":3763,"num_object_clones":0,"num_object_copies":15052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2308,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3725}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8853","reported_seq":8955,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.100647+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:46.100647+0000","last_peered":"2024-09-16T04:47:46.100647+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:46.100647+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8853,"log_dups_size":0,"ondisk_log_size":8853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31986956,"num_objects":3905,"num_object_clones":0,"num_object_copies":15620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8851,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1716}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8709","reported_seq":8837,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.838890+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:46.838890+0000","last_peered":"2024-09-16T04:47:46.838890+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:46.838890+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8709,"log_dups_size":0,"ondisk_log_size":8709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30977938,"num_objects":3782,"num_object_clones":0,"num_object_copies":15128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8706,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8547","reported_seq":8635,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.610924+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:46.610924+0000","last_peered":"2024-09-16T04:47:46.610924+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:46.610924+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8547,"log_dups_size":0,"ondisk_log_size":8547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30997126,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1679}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8756","reported_seq":8839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.100088+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:46.100088+0000","last_peered":"2024-09-16T04:47:46.100088+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:46.100088+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8756,"log_dups_size":0,"ondisk_log_size":8756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30939782,"num_objects":3777,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8755,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1556}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8898","reported_seq":9026,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.658179+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:46.658179+0000","last_peered":"2024-09-16T04:47:46.658179+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:46.658179+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8898,"log_dups_size":0,"ondisk_log_size":8898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31814924,"num_objects":3884,"num_object_clones":0,"num_object_copies":15536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8896,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1686}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8605","reported_seq":8704,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.209166+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:46.209166+0000","last_peered":"2024-09-16T04:47:46.209166+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:46.209166+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8605,"log_dups_size":0,"ondisk_log_size":8605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31487244,"num_objects":3844,"num_object_clones":0,"num_object_copies":15376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8603,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8648","reported_seq":8738,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.448499+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:46.448499+0000","last_peered":"2024-09-16T04:47:46.448499+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:46.448499+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8648,"log_dups_size":0,"ondisk_log_size":8648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29971724,"num_objects":3659,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8646,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1431}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":499115712,"num_objects":60933,"num_object_clones":0,"num_object_copies":243732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60933,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140554,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4835,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1000734720,"data_stored":1000734720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140581,"ondisk_log_size":140581,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735576,"kb_used_data":244360,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93636264,"statfs":{"total":96636764160,"available":95883534336,"internally_reserved":0,"allocated":250224640,"data_stored":250044671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":502984399},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,2,1,4,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731332,"kb_used_data":244788,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93640508,"statfs":{"total":96636764160,"available":95887880192,"internally_reserved":0,"allocated":250662912,"data_stored":250482943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,0,6,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051660,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572264,"kb_used_data":247640,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93799576,"statfs":{"total":96636764160,"available":96050765824,"internally_reserved":0,"allocated":253583360,"data_stored":253403391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332394191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722544,"kb_used_data":243872,"kb_used_omap":4,"kb_used_meta":478651,"kb_avail":93649296,"statfs":{"total":96636764160,"available":95896879104,"internally_reserved":0,"allocated":249724928,"data_stored":249544959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":490139342},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249266176,"data_stored":249266176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252678144,"data_stored":252678144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":249720832,"data_stored":249720832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":249069568,"data_stored":249069568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:48.951 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:47:48.951 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8818", 'reported_seq': 8930, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:46.931028+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:47:46.931028+0000', 'last_peered': '2024-09-16T04:47:46.931028+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:47:46.931028+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8818, 'log_dups_size': 0, 'ondisk_log_size': 8818, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31424512, 'num_objects': 3836, 'num_object_clones': 0, 'num_object_copies': 15344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8818, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1650}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:48.951 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:47:48.952 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8822", 'reported_seq': 8904, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:46.607950+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:47:46.607950+0000', 'last_peered': '2024-09-16T04:47:46.607950+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:47:46.607950+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8822, 'log_dups_size': 0, 'ondisk_log_size': 8822, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31552780, 'num_objects': 3852, 'num_object_clones': 0, 'num_object_copies': 15408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8820, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1649}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:48.952 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:47:48.952 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8843", 'reported_seq': 8937, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:46.048889+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:47:46.048889+0000', 'last_peered': '2024-09-16T04:47:46.048889+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:47:46.048889+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8843, 'log_dups_size': 0, 'ondisk_log_size': 8843, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31349406, 'num_objects': 3828, 'num_object_clones': 0, 'num_object_copies': 15312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3828, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8842, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1609}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:48.952 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:47:48.952 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8846", 'reported_seq': 8930, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:46.652734+0000', 'last_change': '2024-09-16T04:41:27.744804+0000', 'last_active': '2024-09-16T04:47:46.652734+0000', 'last_peered': '2024-09-16T04:47:46.652734+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:47:46.652734+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8846, 'log_dups_size': 0, 'ondisk_log_size': 8846, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31323404, 'num_objects': 3824, 'num_object_clones': 0, 'num_object_copies': 15296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2223, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3824, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8844, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1603}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:48.952 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:47:48.952 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8922", 'reported_seq': 9050, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:46.943113+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:47:46.943113+0000', 'last_peered': '2024-09-16T04:47:46.943113+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:47:46.943113+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8922, 'log_dups_size': 0, 'ondisk_log_size': 8922, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31282444, 'num_objects': 3819, 'num_object_clones': 0, 'num_object_copies': 15276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8920, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1571}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:48.952 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:47:48.952 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8618", 'reported_seq': 8723, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:46.928601+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:47:46.928601+0000', 'last_peered': '2024-09-16T04:47:46.928601+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:47:46.928601+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8618, 'log_dups_size': 0, 'ondisk_log_size': 8618, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30261248, 'num_objects': 3694, 'num_object_clones': 0, 'num_object_copies': 14776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3694, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8618, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1516}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:48.953 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:47:48.953 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8834", 'reported_seq': 8927, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:46.660596+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:47:46.660596+0000', 'last_peered': '2024-09-16T04:47:46.660596+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:47:46.660596+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8834, 'log_dups_size': 0, 'ondisk_log_size': 8834, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30848268, 'num_objects': 3766, 'num_object_clones': 0, 'num_object_copies': 15064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8832, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1553}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:48.953 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:47:48.953 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8993", 'reported_seq': 13662, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:46.954836+0000', 'last_change': '2024-09-16T04:44:10.478859+0000', 'last_active': '2024-09-16T04:47:46.954836+0000', 'last_peered': '2024-09-16T04:47:46.954836+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:47:46.954836+0000', 'last_undegraded': '2024-09-16T04:41:27.529243+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8993, 'log_dups_size': 0, 'ondisk_log_size': 8993, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30820888, 'num_objects': 3763, 'num_object_clones': 0, 'num_object_copies': 15052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 38, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3763, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8990, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2308, 'num_bytes_recovered': 15925248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3725}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:48.953 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:47:48.953 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8853", 'reported_seq': 8955, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:46.100647+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:47:46.100647+0000', 'last_peered': '2024-09-16T04:47:46.100647+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:47:46.100647+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8853, 'log_dups_size': 0, 'ondisk_log_size': 8853, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31986956, 'num_objects': 3905, 'num_object_clones': 0, 'num_object_copies': 15620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3905, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8851, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1716}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:48.953 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:47:48.953 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'8709", 'reported_seq': 8837, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:46.838890+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:47:46.838890+0000', 'last_peered': '2024-09-16T04:47:46.838890+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:47:46.838890+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8709, 'log_dups_size': 0, 'ondisk_log_size': 8709, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30977938, 'num_objects': 3782, 'num_object_clones': 0, 'num_object_copies': 15128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8706, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1627}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:48.953 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:47:48.953 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8547", 'reported_seq': 8635, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:46.610924+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:47:46.610924+0000', 'last_peered': '2024-09-16T04:47:46.610924+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:47:46.610924+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8547, 'log_dups_size': 0, 'ondisk_log_size': 8547, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30997126, 'num_objects': 3784, 'num_object_clones': 0, 'num_object_copies': 15136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8546, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1679}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:48.955 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:47:48.955 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8756", 'reported_seq': 8839, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:46.100088+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:47:46.100088+0000', 'last_peered': '2024-09-16T04:47:46.100088+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:47:46.100088+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8756, 'log_dups_size': 0, 'ondisk_log_size': 8756, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30939782, 'num_objects': 3777, 'num_object_clones': 0, 'num_object_copies': 15108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8755, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1556}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:48.955 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:47:48.955 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8898", 'reported_seq': 9026, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:46.658179+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:47:46.658179+0000', 'last_peered': '2024-09-16T04:47:46.658179+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:47:46.658179+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8898, 'log_dups_size': 0, 'ondisk_log_size': 8898, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31814924, 'num_objects': 3884, 'num_object_clones': 0, 'num_object_copies': 15536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3884, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8896, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1686}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:48.955 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:47:48.955 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8605", 'reported_seq': 8704, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:46.209166+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:47:46.209166+0000', 'last_peered': '2024-09-16T04:47:46.209166+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:47:46.209166+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8605, 'log_dups_size': 0, 'ondisk_log_size': 8605, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31487244, 'num_objects': 3844, 'num_object_clones': 0, 'num_object_copies': 15376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3844, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8603, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:48.955 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:47:48.955 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8648", 'reported_seq': 8738, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:46.448499+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:47:46.448499+0000', 'last_peered': '2024-09-16T04:47:46.448499+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:47:46.448499+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8648, 'log_dups_size': 0, 'ondisk_log_size': 8648, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29971724, 'num_objects': 3659, 'num_object_clones': 0, 'num_object_copies': 14636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3659, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8646, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1431}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:48.956 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:47:49.690 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:49.690 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30747/243732 objects degraded (12.615%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":482,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":60933,"data_bytes":499115712,"bytes_used":2827997184,"bytes_avail":383719059456,"bytes_total":386547056640,"degraded_objects":30747,"degraded_total":243732,"degraded_ratio":0.12615085421692679,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":206,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":221},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T04:47:26.684344+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:47:49.715 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:47:49.715 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:50.352 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:50.352 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:50.384 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-09-16T04:47:48.691046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":499115712,"num_objects":60933,"num_object_clones":0,"num_object_copies":243732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60933,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140554,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4835,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140581,"ondisk_log_size":140581,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2761716,"kb_used_data":980660,"kb_used_omap":14,"kb_used_meta":1780977,"kb_avail":374725644,"statfs":{"total":386547056640,"available":383719059456,"internally_reserved":0,"allocated":1004195840,"data_stored":1003475964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1823721009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,14,13,5,16,16,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21741568,"num_objects":-2654,"num_object_clones":0,"num_object_copies":-10616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2654,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004268"},"pg_stats":[{"pgid":"1.f","version":"33'8818","reported_seq":8930,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.931028+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:46.931028+0000","last_peered":"2024-09-16T04:47:46.931028+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:46.931028+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8818,"log_dups_size":0,"ondisk_log_size":8818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31424512,"num_objects":3836,"num_object_clones":0,"num_object_copies":15344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8818,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8822","reported_seq":8904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.607950+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:46.607950+0000","last_peered":"2024-09-16T04:47:46.607950+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:46.607950+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8822,"log_dups_size":0,"ondisk_log_size":8822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31552780,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8820,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1649}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8843","reported_seq":8937,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.048889+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:46.048889+0000","last_peered":"2024-09-16T04:47:46.048889+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:46.048889+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8843,"log_dups_size":0,"ondisk_log_size":8843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31349406,"num_objects":3828,"num_object_clones":0,"num_object_copies":15312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3828,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8842,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1609}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8846","reported_seq":8930,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.652734+0000","last_change":"2024-09-16T04:41:27.744804+0000","last_active":"2024-09-16T04:47:46.652734+0000","last_peered":"2024-09-16T04:47:46.652734+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:46.652734+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8846,"log_dups_size":0,"ondisk_log_size":8846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31323404,"num_objects":3824,"num_object_clones":0,"num_object_copies":15296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8844,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1603}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8922","reported_seq":9050,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.943113+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:46.943113+0000","last_peered":"2024-09-16T04:47:46.943113+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:46.943113+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31282444,"num_objects":3819,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8920,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1571}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8618","reported_seq":8723,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.928601+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:46.928601+0000","last_peered":"2024-09-16T04:47:46.928601+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:46.928601+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8618,"log_dups_size":0,"ondisk_log_size":8618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30261248,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8618,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8834","reported_seq":8927,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.660596+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:46.660596+0000","last_peered":"2024-09-16T04:47:46.660596+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:46.660596+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8834,"log_dups_size":0,"ondisk_log_size":8834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30848268,"num_objects":3766,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8832,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1553}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8869","reported_seq":13426,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:43.490011+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:43.490011+0000","last_peered":"2024-09-16T04:47:43.490011+0000","last_clean":"2024-09-16T04:47:43.490011+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:43.490011+0000","last_undegraded":"2024-09-16T04:47:43.490011+0000","last_fullsized":"2024-09-16T04:47:43.490011+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8869,"log_dups_size":0,"ondisk_log_size":8869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:44.859841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32077068,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8867,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'8993","reported_seq":13662,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.954836+0000","last_change":"2024-09-16T04:44:10.478859+0000","last_active":"2024-09-16T04:47:46.954836+0000","last_peered":"2024-09-16T04:47:46.954836+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:46.954836+0000","last_undegraded":"2024-09-16T04:41:27.529243+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8993,"log_dups_size":0,"ondisk_log_size":8993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30820888,"num_objects":3763,"num_object_clones":0,"num_object_copies":15052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2308,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3725}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8853","reported_seq":8955,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.100647+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:46.100647+0000","last_peered":"2024-09-16T04:47:46.100647+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:46.100647+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8853,"log_dups_size":0,"ondisk_log_size":8853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31986956,"num_objects":3905,"num_object_clones":0,"num_object_copies":15620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8851,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1716}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8709","reported_seq":8837,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.838890+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:46.838890+0000","last_peered":"2024-09-16T04:47:46.838890+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:46.838890+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8709,"log_dups_size":0,"ondisk_log_size":8709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30977938,"num_objects":3782,"num_object_clones":0,"num_object_copies":15128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8706,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8547","reported_seq":8635,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.610924+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:46.610924+0000","last_peered":"2024-09-16T04:47:46.610924+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:46.610924+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8547,"log_dups_size":0,"ondisk_log_size":8547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30997126,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1679}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8756","reported_seq":8839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.100088+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:46.100088+0000","last_peered":"2024-09-16T04:47:46.100088+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:46.100088+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8756,"log_dups_size":0,"ondisk_log_size":8756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30939782,"num_objects":3777,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8755,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1556}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8898","reported_seq":9026,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.658179+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:46.658179+0000","last_peered":"2024-09-16T04:47:46.658179+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:46.658179+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8898,"log_dups_size":0,"ondisk_log_size":8898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31814924,"num_objects":3884,"num_object_clones":0,"num_object_copies":15536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8896,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1686}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8605","reported_seq":8704,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.209166+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:46.209166+0000","last_peered":"2024-09-16T04:47:46.209166+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:46.209166+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8605,"log_dups_size":0,"ondisk_log_size":8605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31487244,"num_objects":3844,"num_object_clones":0,"num_object_copies":15376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8603,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8648","reported_seq":8738,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:46.448499+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:46.448499+0000","last_peered":"2024-09-16T04:47:46.448499+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:46.448499+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8648,"log_dups_size":0,"ondisk_log_size":8648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29971724,"num_objects":3659,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8646,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1431}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":499115712,"num_objects":60933,"num_object_clones":0,"num_object_copies":243732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60933,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140554,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4835,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1000734720,"data_stored":1000734720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140581,"ondisk_log_size":140581,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735576,"kb_used_data":244360,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93636264,"statfs":{"total":96636764160,"available":95883534336,"internally_reserved":0,"allocated":250224640,"data_stored":250044671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":502984399},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,2,1,4,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731332,"kb_used_data":244788,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93640508,"statfs":{"total":96636764160,"available":95887880192,"internally_reserved":0,"allocated":250662912,"data_stored":250482943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,0,6,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051660,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572264,"kb_used_data":247640,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93799576,"statfs":{"total":96636764160,"available":96050765824,"internally_reserved":0,"allocated":253583360,"data_stored":253403391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332394191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722544,"kb_used_data":243872,"kb_used_omap":4,"kb_used_meta":478651,"kb_avail":93649296,"statfs":{"total":96636764160,"available":95896879104,"internally_reserved":0,"allocated":249724928,"data_stored":249544959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":490139342},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249266176,"data_stored":249266176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252678144,"data_stored":252678144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":249720832,"data_stored":249720832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":249069568,"data_stored":249069568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:53.386 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:54.011 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:54.011 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:54.046 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-09-16T04:47:52.692082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":490292928,"num_objects":59856,"num_object_clones":0,"num_object_copies":239424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141631,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4900,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141658,"ondisk_log_size":141658,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2766020,"kb_used_data":962736,"kb_used_omap":14,"kb_used_meta":1803185,"kb_avail":374721340,"statfs":{"total":386547056640,"available":383714652160,"internally_reserved":0,"allocated":985841664,"data_stored":985112576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1846462001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,6,15,5,19,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17047552,"num_objects":-2081,"num_object_clones":0,"num_object_copies":-8324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2081,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003215"},"pg_stats":[{"pgid":"1.f","version":"34'8894","reported_seq":9008,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.961351+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:51.961351+0000","last_peered":"2024-09-16T04:47:51.961351+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:51.961351+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8894,"log_dups_size":0,"ondisk_log_size":8894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30801920,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8894,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1574}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'8887","reported_seq":8973,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.630282+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:51.630282+0000","last_peered":"2024-09-16T04:47:51.630282+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:51.630282+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31020300,"num_objects":3787,"num_object_clones":0,"num_object_copies":15148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8907","reported_seq":8999,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.258706+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:51.258706+0000","last_peered":"2024-09-16T04:47:51.258706+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:51.258706+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8907,"log_dups_size":0,"ondisk_log_size":8907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30825118,"num_objects":3764,"num_object_clones":0,"num_object_copies":15056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3764,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8906,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1545}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'8913","reported_seq":9054,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.859108+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:47:51.859108+0000","last_peered":"2024-09-16T04:47:51.859108+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:51.859108+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8913,"log_dups_size":0,"ondisk_log_size":8913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30774540,"num_objects":3757,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8911,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1561}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'8986","reported_seq":9116,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.688461+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:51.688461+0000","last_peered":"2024-09-16T04:47:51.688461+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:51.688461+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8986,"log_dups_size":0,"ondisk_log_size":8986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30758156,"num_objects":3755,"num_object_clones":0,"num_object_copies":15020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8984,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'8682","reported_seq":8790,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:52.027347+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:52.027347+0000","last_peered":"2024-09-16T04:47:52.027347+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:52.027347+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8682,"log_dups_size":0,"ondisk_log_size":8682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29736960,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8682,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'8901","reported_seq":8996,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.797277+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:51.797277+0000","last_peered":"2024-09-16T04:47:51.797277+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:51.797277+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8901,"log_dups_size":0,"ondisk_log_size":8901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30299404,"num_objects":3699,"num_object_clones":0,"num_object_copies":14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8899,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1486}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8949","reported_seq":13506,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:48.680351+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:48.680351+0000","last_peered":"2024-09-16T04:47:48.680351+0000","last_clean":"2024-09-16T04:47:48.680351+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:48.680351+0000","last_undegraded":"2024-09-16T04:47:48.680351+0000","last_fullsized":"2024-09-16T04:47:48.680351+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:50.800964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31421708,"num_objects":3836,"num_object_clones":0,"num_object_copies":15344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8947,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'9062","reported_seq":13809,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T04:47:51.340569+0000","last_change":"2024-09-16T04:47:51.386520+0000","last_active":"2024-09-16T04:47:51.386520+0000","last_peered":"2024-09-16T04:47:51.340569+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:51.340569+0000","last_undegraded":"2024-09-16T04:47:51.340569+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9062,"log_dups_size":0,"ondisk_log_size":9062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30255640,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9059,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8922","reported_seq":9024,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.258025+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:51.258025+0000","last_peered":"2024-09-16T04:47:51.258025+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:51.258025+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31421708,"num_objects":3836,"num_object_clones":0,"num_object_copies":15344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8920,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1647}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'8788","reported_seq":8918,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.959317+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:51.959317+0000","last_peered":"2024-09-16T04:47:51.959317+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:51.959317+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8788,"log_dups_size":0,"ondisk_log_size":8788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30330770,"num_objects":3703,"num_object_clones":0,"num_object_copies":14812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8785,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1548}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'8609","reported_seq":8699,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.942904+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:51.942904+0000","last_peered":"2024-09-16T04:47:51.942904+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:51.942904+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8609,"log_dups_size":0,"ondisk_log_size":8609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30489222,"num_objects":3722,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8608,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1617}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8820","reported_seq":8904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.313965+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:51.313965+0000","last_peered":"2024-09-16T04:47:51.313965+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:51.313965+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8820,"log_dups_size":0,"ondisk_log_size":8820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30415494,"num_objects":3713,"num_object_clones":0,"num_object_copies":14852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8819,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1492}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'8955","reported_seq":9085,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.777926+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:51.777926+0000","last_peered":"2024-09-16T04:47:51.777926+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:51.777926+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8955,"log_dups_size":0,"ondisk_log_size":8955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31347980,"num_objects":3827,"num_object_clones":0,"num_object_copies":15308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8953,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1629}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8668","reported_seq":8767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.206137+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:51.206137+0000","last_peered":"2024-09-16T04:47:51.206137+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:51.206137+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8668,"log_dups_size":0,"ondisk_log_size":8668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30971148,"num_objects":3781,"num_object_clones":0,"num_object_copies":15124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8666,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1658}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'8715","reported_seq":8807,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.801924+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:51.801924+0000","last_peered":"2024-09-16T04:47:51.801924+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:51.801924+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8715,"log_dups_size":0,"ondisk_log_size":8715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29422860,"num_objects":3592,"num_object_clones":0,"num_object_copies":14368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":490292928,"num_objects":59856,"num_object_clones":0,"num_object_copies":239424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141631,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4900,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983052288,"data_stored":983052288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141658,"ondisk_log_size":141658,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722192,"kb_used_data":239296,"kb_used_omap":4,"kb_used_meta":482875,"kb_avail":93649648,"statfs":{"total":96636764160,"available":95897239552,"internally_reserved":0,"allocated":245039104,"data_stored":244854529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":494464719},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,3,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731084,"kb_used_data":240380,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93640756,"statfs":{"total":96636764160,"available":95888134144,"internally_reserved":0,"allocated":246149120,"data_stored":245969151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,5,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590216,"kb_used_data":243376,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93781624,"statfs":{"total":96636764160,"available":96032382976,"internally_reserved":0,"allocated":249217024,"data_stored":249037055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355135183},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,10,2,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722528,"kb_used_data":239684,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93649312,"statfs":{"total":96636764160,"available":95896895488,"internally_reserved":0,"allocated":245436416,"data_stored":245251841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":494399182},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":244895744,"data_stored":244895744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248094720,"data_stored":248094720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":245321728,"data_stored":245321728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":244740096,"data_stored":244740096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:54.048 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:54.652 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:54.652 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:54.690 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-09-16T04:47:52.692082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":490292928,"num_objects":59856,"num_object_clones":0,"num_object_copies":239424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141631,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4900,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141658,"ondisk_log_size":141658,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2766020,"kb_used_data":962736,"kb_used_omap":14,"kb_used_meta":1803185,"kb_avail":374721340,"statfs":{"total":386547056640,"available":383714652160,"internally_reserved":0,"allocated":985841664,"data_stored":985112576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1846462001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,6,15,5,19,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17047552,"num_objects":-2081,"num_object_clones":0,"num_object_copies":-8324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2081,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003215"},"pg_stats":[{"pgid":"1.f","version":"34'8894","reported_seq":9008,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.961351+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:51.961351+0000","last_peered":"2024-09-16T04:47:51.961351+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:51.961351+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8894,"log_dups_size":0,"ondisk_log_size":8894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30801920,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8894,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1574}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'8887","reported_seq":8973,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.630282+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:51.630282+0000","last_peered":"2024-09-16T04:47:51.630282+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:51.630282+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31020300,"num_objects":3787,"num_object_clones":0,"num_object_copies":15148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8907","reported_seq":8999,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.258706+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:51.258706+0000","last_peered":"2024-09-16T04:47:51.258706+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:51.258706+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8907,"log_dups_size":0,"ondisk_log_size":8907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30825118,"num_objects":3764,"num_object_clones":0,"num_object_copies":15056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3764,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8906,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1545}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'8913","reported_seq":9054,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.859108+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:47:51.859108+0000","last_peered":"2024-09-16T04:47:51.859108+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:51.859108+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8913,"log_dups_size":0,"ondisk_log_size":8913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30774540,"num_objects":3757,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8911,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1561}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'8986","reported_seq":9116,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.688461+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:51.688461+0000","last_peered":"2024-09-16T04:47:51.688461+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:51.688461+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8986,"log_dups_size":0,"ondisk_log_size":8986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30758156,"num_objects":3755,"num_object_clones":0,"num_object_copies":15020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8984,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'8682","reported_seq":8790,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:52.027347+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:52.027347+0000","last_peered":"2024-09-16T04:47:52.027347+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:52.027347+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8682,"log_dups_size":0,"ondisk_log_size":8682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29736960,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8682,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'8901","reported_seq":8996,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.797277+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:51.797277+0000","last_peered":"2024-09-16T04:47:51.797277+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:51.797277+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8901,"log_dups_size":0,"ondisk_log_size":8901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30299404,"num_objects":3699,"num_object_clones":0,"num_object_copies":14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8899,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1486}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8949","reported_seq":13506,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:47:48.680351+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:48.680351+0000","last_peered":"2024-09-16T04:47:48.680351+0000","last_clean":"2024-09-16T04:47:48.680351+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:48.680351+0000","last_undegraded":"2024-09-16T04:47:48.680351+0000","last_fullsized":"2024-09-16T04:47:48.680351+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:50.800964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31421708,"num_objects":3836,"num_object_clones":0,"num_object_copies":15344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8947,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'9062","reported_seq":13809,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T04:47:51.340569+0000","last_change":"2024-09-16T04:47:51.386520+0000","last_active":"2024-09-16T04:47:51.386520+0000","last_peered":"2024-09-16T04:47:51.340569+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:51.340569+0000","last_undegraded":"2024-09-16T04:47:51.340569+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9062,"log_dups_size":0,"ondisk_log_size":9062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30255640,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9059,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8922","reported_seq":9024,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.258025+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:51.258025+0000","last_peered":"2024-09-16T04:47:51.258025+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:51.258025+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31421708,"num_objects":3836,"num_object_clones":0,"num_object_copies":15344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8920,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1647}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'8788","reported_seq":8918,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.959317+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:51.959317+0000","last_peered":"2024-09-16T04:47:51.959317+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:51.959317+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8788,"log_dups_size":0,"ondisk_log_size":8788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30330770,"num_objects":3703,"num_object_clones":0,"num_object_copies":14812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8785,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1548}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'8609","reported_seq":8699,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.942904+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:51.942904+0000","last_peered":"2024-09-16T04:47:51.942904+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:51.942904+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8609,"log_dups_size":0,"ondisk_log_size":8609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30489222,"num_objects":3722,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8608,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1617}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8820","reported_seq":8904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.313965+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:51.313965+0000","last_peered":"2024-09-16T04:47:51.313965+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:51.313965+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8820,"log_dups_size":0,"ondisk_log_size":8820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30415494,"num_objects":3713,"num_object_clones":0,"num_object_copies":14852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8819,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1492}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'8955","reported_seq":9085,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.777926+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:51.777926+0000","last_peered":"2024-09-16T04:47:51.777926+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:51.777926+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8955,"log_dups_size":0,"ondisk_log_size":8955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31347980,"num_objects":3827,"num_object_clones":0,"num_object_copies":15308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8953,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1629}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8668","reported_seq":8767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.206137+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:51.206137+0000","last_peered":"2024-09-16T04:47:51.206137+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:51.206137+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8668,"log_dups_size":0,"ondisk_log_size":8668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30971148,"num_objects":3781,"num_object_clones":0,"num_object_copies":15124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8666,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1658}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'8715","reported_seq":8807,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.801924+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:51.801924+0000","last_peered":"2024-09-16T04:47:51.801924+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:51.801924+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8715,"log_dups_size":0,"ondisk_log_size":8715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29422860,"num_objects":3592,"num_object_clones":0,"num_object_copies":14368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":490292928,"num_objects":59856,"num_object_clones":0,"num_object_copies":239424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141631,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4900,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983052288,"data_stored":983052288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141658,"ondisk_log_size":141658,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722192,"kb_used_data":239296,"kb_used_omap":4,"kb_used_meta":482875,"kb_avail":93649648,"statfs":{"total":96636764160,"available":95897239552,"internally_reserved":0,"allocated":245039104,"data_stored":244854529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":494464719},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,3,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731084,"kb_used_data":240380,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93640756,"statfs":{"total":96636764160,"available":95888134144,"internally_reserved":0,"allocated":246149120,"data_stored":245969151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,5,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590216,"kb_used_data":243376,"kb_used_omap":4,"kb_used_meta":346811,"kb_avail":93781624,"statfs":{"total":96636764160,"available":96032382976,"internally_reserved":0,"allocated":249217024,"data_stored":249037055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355135183},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,10,2,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722528,"kb_used_data":239684,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93649312,"statfs":{"total":96636764160,"available":95896895488,"internally_reserved":0,"allocated":245436416,"data_stored":245251841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":494399182},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":244895744,"data_stored":244895744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248094720,"data_stored":248094720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":245321728,"data_stored":245321728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":244740096,"data_stored":244740096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:54.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:47:54.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "34'8894", 'reported_seq': 9008, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:51.961351+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:47:51.961351+0000', 'last_peered': '2024-09-16T04:47:51.961351+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:47:51.961351+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8894, 'log_dups_size': 0, 'ondisk_log_size': 8894, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30801920, 'num_objects': 3760, 'num_object_clones': 0, 'num_object_copies': 15040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3760, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8894, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1574}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:54.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:47:54.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "34'8887", 'reported_seq': 8973, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:51.630282+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:47:51.630282+0000', 'last_peered': '2024-09-16T04:47:51.630282+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:47:51.630282+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8887, 'log_dups_size': 0, 'ondisk_log_size': 8887, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31020300, 'num_objects': 3787, 'num_object_clones': 0, 'num_object_copies': 15148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3787, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8885, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1584}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:54.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:47:54.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8907", 'reported_seq': 8999, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:51.258706+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:47:51.258706+0000', 'last_peered': '2024-09-16T04:47:51.258706+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:47:51.258706+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8907, 'log_dups_size': 0, 'ondisk_log_size': 8907, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30825118, 'num_objects': 3764, 'num_object_clones': 0, 'num_object_copies': 15056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3764, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8906, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1545}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:54.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:47:54.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'8913", 'reported_seq': 9054, 'reported_epoch': 34, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:51.859108+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:47:51.859108+0000', 'last_peered': '2024-09-16T04:47:51.859108+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:47:51.859108+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8913, 'log_dups_size': 0, 'ondisk_log_size': 8913, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30774540, 'num_objects': 3757, 'num_object_clones': 0, 'num_object_copies': 15028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2196, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8911, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1561}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:54.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:47:54.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "34'8986", 'reported_seq': 9116, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:51.688461+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:47:51.688461+0000', 'last_peered': '2024-09-16T04:47:51.688461+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:47:51.688461+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8986, 'log_dups_size': 0, 'ondisk_log_size': 8986, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30758156, 'num_objects': 3755, 'num_object_clones': 0, 'num_object_copies': 15020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8984, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1507}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:54.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:47:54.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "34'8682", 'reported_seq': 8790, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:52.027347+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:47:52.027347+0000', 'last_peered': '2024-09-16T04:47:52.027347+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:47:52.027347+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8682, 'log_dups_size': 0, 'ondisk_log_size': 8682, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29736960, 'num_objects': 3630, 'num_object_clones': 0, 'num_object_copies': 14520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8682, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1452}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:54.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:47:54.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "34'8901", 'reported_seq': 8996, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:51.797277+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:47:51.797277+0000', 'last_peered': '2024-09-16T04:47:51.797277+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:47:51.797277+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8901, 'log_dups_size': 0, 'ondisk_log_size': 8901, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30299404, 'num_objects': 3699, 'num_object_clones': 0, 'num_object_copies': 14796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8899, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1486}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:54.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-16T04:47:54.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'9062", 'reported_seq': 13809, 'reported_epoch': 33, 'state': 'peering', 'last_fresh': '2024-09-16T04:47:51.340569+0000', 'last_change': '2024-09-16T04:47:51.386520+0000', 'last_active': '2024-09-16T04:47:51.386520+0000', 'last_peered': '2024-09-16T04:47:51.340569+0000', 'last_clean': '2024-09-16T04:40:23.018364+0000', 'last_became_active': '2024-09-16T04:41:27.761268+0000', 'last_became_peered': '2024-09-16T04:41:27.761268+0000', 'last_unstale': '2024-09-16T04:47:51.340569+0000', 'last_undegraded': '2024-09-16T04:47:51.340569+0000', 'last_fullsized': '2024-09-16T04:41:27.484100+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9062, 'log_dups_size': 0, 'ondisk_log_size': 9062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30255640, 'num_objects': 3694, 'num_object_clones': 0, 'num_object_copies': 14776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3694, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9059, 'num_write_kb': 51009, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2346, 'num_bytes_recovered': 15925248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:54.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:47:54.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8922", 'reported_seq': 9024, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:51.258025+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:47:51.258025+0000', 'last_peered': '2024-09-16T04:47:51.258025+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:47:51.258025+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8922, 'log_dups_size': 0, 'ondisk_log_size': 8922, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31421708, 'num_objects': 3836, 'num_object_clones': 0, 'num_object_copies': 15344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8920, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1647}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:54.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:47:54.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "34'8788", 'reported_seq': 8918, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:51.959317+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:47:51.959317+0000', 'last_peered': '2024-09-16T04:47:51.959317+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:47:51.959317+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8788, 'log_dups_size': 0, 'ondisk_log_size': 8788, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30330770, 'num_objects': 3703, 'num_object_clones': 0, 'num_object_copies': 14812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3703, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8785, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1548}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:54.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:47:54.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "34'8609", 'reported_seq': 8699, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:51.942904+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:47:51.942904+0000', 'last_peered': '2024-09-16T04:47:51.942904+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:47:51.942904+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8609, 'log_dups_size': 0, 'ondisk_log_size': 8609, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30489222, 'num_objects': 3722, 'num_object_clones': 0, 'num_object_copies': 14888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8608, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1617}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:54.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:47:54.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8820", 'reported_seq': 8904, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:51.313965+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:47:51.313965+0000', 'last_peered': '2024-09-16T04:47:51.313965+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:47:51.313965+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8820, 'log_dups_size': 0, 'ondisk_log_size': 8820, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30415494, 'num_objects': 3713, 'num_object_clones': 0, 'num_object_copies': 14852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8819, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1492}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:54.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:47:54.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "34'8955", 'reported_seq': 9085, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:51.777926+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:47:51.777926+0000', 'last_peered': '2024-09-16T04:47:51.777926+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:47:51.777926+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8955, 'log_dups_size': 0, 'ondisk_log_size': 8955, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31347980, 'num_objects': 3827, 'num_object_clones': 0, 'num_object_copies': 15308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3827, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8953, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1629}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:47:54.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:47:54.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8668", 'reported_seq': 8767, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:51.206137+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:47:51.206137+0000', 'last_peered': '2024-09-16T04:47:51.206137+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:47:51.206137+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8668, 'log_dups_size': 0, 'ondisk_log_size': 8668, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30971148, 'num_objects': 3781, 'num_object_clones': 0, 'num_object_copies': 15124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3781, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8666, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1658}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:47:54.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:47:54.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'8715", 'reported_seq': 8807, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:51.801924+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:47:51.801924+0000', 'last_peered': '2024-09-16T04:47:51.801924+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:47:51.801924+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8715, 'log_dups_size': 0, 'ondisk_log_size': 8715, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29422860, 'num_objects': 3592, 'num_object_clones': 0, 'num_object_copies': 14368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3592, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8713, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1364}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:47:54.694 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:47:55.417 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:55.417 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30682/239160 objects degraded (12.829%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":487,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"peering","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":59790,"data_bytes":489752256,"bytes_used":2850942976,"bytes_avail":383696113664,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":30682,"degraded_total":239160,"degraded_ratio":0.12829068406087973,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":234},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T04:47:26.684344+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:47:55.443 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:47:55.443 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:55.995 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:55.996 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:56.024 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-09-16T04:47:54.692543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":489752256,"num_objects":59790,"num_object_clones":0,"num_object_copies":239160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59790,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141697,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4900,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141724,"ondisk_log_size":141724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2784124,"kb_used_data":958636,"kb_used_omap":14,"kb_used_meta":1825393,"kb_avail":374703236,"statfs":{"total":386547056640,"available":383696113664,"internally_reserved":0,"allocated":981643264,"data_stored":980904943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1869202993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,6,15,5,19,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15343616,"num_objects":-1873,"num_object_clones":0,"num_object_copies":-7492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1873,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003203"},"pg_stats":[{"pgid":"1.f","version":"34'8894","reported_seq":9008,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.961351+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:51.961351+0000","last_peered":"2024-09-16T04:47:51.961351+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:51.961351+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8894,"log_dups_size":0,"ondisk_log_size":8894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30801920,"num_objects":3760,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8894,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1574}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'8887","reported_seq":8973,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.630282+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:51.630282+0000","last_peered":"2024-09-16T04:47:51.630282+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:51.630282+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31020300,"num_objects":3787,"num_object_clones":0,"num_object_copies":15148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8907","reported_seq":8999,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.258706+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:51.258706+0000","last_peered":"2024-09-16T04:47:51.258706+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:51.258706+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8907,"log_dups_size":0,"ondisk_log_size":8907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30825118,"num_objects":3764,"num_object_clones":0,"num_object_copies":15056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3764,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8906,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1545}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'8913","reported_seq":9054,"reported_epoch":34,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.859108+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:47:51.859108+0000","last_peered":"2024-09-16T04:47:51.859108+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:51.859108+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8913,"log_dups_size":0,"ondisk_log_size":8913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30774540,"num_objects":3757,"num_object_clones":0,"num_object_copies":15028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8911,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1561}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"34'8986","reported_seq":9116,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.688461+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:51.688461+0000","last_peered":"2024-09-16T04:47:51.688461+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:51.688461+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8986,"log_dups_size":0,"ondisk_log_size":8986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30758156,"num_objects":3755,"num_object_clones":0,"num_object_copies":15020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8984,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"34'8682","reported_seq":8790,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:52.027347+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:52.027347+0000","last_peered":"2024-09-16T04:47:52.027347+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:52.027347+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8682,"log_dups_size":0,"ondisk_log_size":8682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29736960,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8682,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'8901","reported_seq":8996,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.797277+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:51.797277+0000","last_peered":"2024-09-16T04:47:51.797277+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:51.797277+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8901,"log_dups_size":0,"ondisk_log_size":8901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30299404,"num_objects":3699,"num_object_clones":0,"num_object_copies":14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8899,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1486}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9015","reported_seq":13576,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:47:53.665854+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:53.665854+0000","last_peered":"2024-09-16T04:47:53.665854+0000","last_clean":"2024-09-16T04:47:53.665854+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:53.665854+0000","last_undegraded":"2024-09-16T04:47:53.665854+0000","last_fullsized":"2024-09-16T04:47:53.665854+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9015,"log_dups_size":0,"ondisk_log_size":9015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:56.755759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30881036,"num_objects":3770,"num_object_clones":0,"num_object_copies":15080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"33'9062","reported_seq":13809,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T04:47:51.340569+0000","last_change":"2024-09-16T04:47:51.386520+0000","last_active":"2024-09-16T04:47:51.386520+0000","last_peered":"2024-09-16T04:47:51.340569+0000","last_clean":"2024-09-16T04:40:23.018364+0000","last_became_active":"2024-09-16T04:41:27.761268+0000","last_became_peered":"2024-09-16T04:41:27.761268+0000","last_unstale":"2024-09-16T04:47:51.340569+0000","last_undegraded":"2024-09-16T04:47:51.340569+0000","last_fullsized":"2024-09-16T04:41:27.484100+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9062,"log_dups_size":0,"ondisk_log_size":9062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30255640,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9059,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8922","reported_seq":9024,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.258025+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:51.258025+0000","last_peered":"2024-09-16T04:47:51.258025+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:51.258025+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31421708,"num_objects":3836,"num_object_clones":0,"num_object_copies":15344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8920,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1647}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"34'8788","reported_seq":8918,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.959317+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:51.959317+0000","last_peered":"2024-09-16T04:47:51.959317+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:51.959317+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8788,"log_dups_size":0,"ondisk_log_size":8788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30330770,"num_objects":3703,"num_object_clones":0,"num_object_copies":14812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8785,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1548}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"34'8609","reported_seq":8699,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.942904+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:51.942904+0000","last_peered":"2024-09-16T04:47:51.942904+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:51.942904+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8609,"log_dups_size":0,"ondisk_log_size":8609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30489222,"num_objects":3722,"num_object_clones":0,"num_object_copies":14888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8608,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1617}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8820","reported_seq":8904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.313965+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:51.313965+0000","last_peered":"2024-09-16T04:47:51.313965+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:51.313965+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8820,"log_dups_size":0,"ondisk_log_size":8820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30415494,"num_objects":3713,"num_object_clones":0,"num_object_copies":14852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8819,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1492}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"34'8955","reported_seq":9085,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.777926+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:51.777926+0000","last_peered":"2024-09-16T04:47:51.777926+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:51.777926+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8955,"log_dups_size":0,"ondisk_log_size":8955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31347980,"num_objects":3827,"num_object_clones":0,"num_object_copies":15308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8953,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1629}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8668","reported_seq":8767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.206137+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:51.206137+0000","last_peered":"2024-09-16T04:47:51.206137+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:51.206137+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8668,"log_dups_size":0,"ondisk_log_size":8668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30971148,"num_objects":3781,"num_object_clones":0,"num_object_copies":15124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8666,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1658}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'8715","reported_seq":8807,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:51.801924+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:51.801924+0000","last_peered":"2024-09-16T04:47:51.801924+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:51.801924+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8715,"log_dups_size":0,"ondisk_log_size":8715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29422860,"num_objects":3592,"num_object_clones":0,"num_object_copies":14368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":489752256,"num_objects":59790,"num_object_clones":0,"num_object_copies":239160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59790,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141697,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4900,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":978808832,"data_stored":978808832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141724,"ondisk_log_size":141724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722192,"kb_used_data":239296,"kb_used_omap":4,"kb_used_meta":482875,"kb_avail":93649648,"statfs":{"total":96636764160,"available":95897239552,"internally_reserved":0,"allocated":245039104,"data_stored":244854529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":494464719},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,3,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731084,"kb_used_data":240380,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93640756,"statfs":{"total":96636764160,"available":95888134144,"internally_reserved":0,"allocated":246149120,"data_stored":245969151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,5,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051662,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608320,"kb_used_data":239276,"kb_used_omap":4,"kb_used_meta":369019,"kb_avail":93763520,"statfs":{"total":96636764160,"available":96013844480,"internally_reserved":0,"allocated":245018624,"data_stored":244829422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":377876175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,10,2,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722528,"kb_used_data":239684,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93649312,"statfs":{"total":96636764160,"available":95896895488,"internally_reserved":0,"allocated":245436416,"data_stored":245251841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":494399182},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":244895744,"data_stored":244895744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":243851264,"data_stored":243851264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":245321728,"data_stored":245321728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":244740096,"data_stored":244740096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:59.026 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:59.625 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:47:59.628 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:47:59.659 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-09-16T04:47:58.693534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":481937088,"num_objects":58836,"num_object_clones":0,"num_object_copies":235344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58836,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142651,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4965,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142678,"ondisk_log_size":142678,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2822008,"kb_used_data":946540,"kb_used_omap":14,"kb_used_meta":1875377,"kb_avail":374665352,"statfs":{"total":386547056640,"available":383657320448,"internally_reserved":0,"allocated":969256960,"data_stored":968500152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1920386609},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,22,15,17,19,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16523264,"num_objects":-2017,"num_object_clones":0,"num_object_copies":-8068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2017,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001928"},"pg_stats":[{"pgid":"1.f","version":"35'8960","reported_seq":9076,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:57.017929+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:57.017929+0000","last_peered":"2024-09-16T04:47:57.017929+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:57.017929+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8960,"log_dups_size":0,"ondisk_log_size":8960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30261248,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8960,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8951","reported_seq":9039,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.797790+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:56.797790+0000","last_peered":"2024-09-16T04:47:56.797790+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:56.797790+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8951,"log_dups_size":0,"ondisk_log_size":8951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30496012,"num_objects":3723,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8949,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8969","reported_seq":9066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.280591+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:56.280591+0000","last_peered":"2024-09-16T04:47:56.280591+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:56.280591+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30317214,"num_objects":3702,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8968,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8969","reported_seq":9242,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.924736+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:47:56.924736+0000","last_peered":"2024-09-16T04:47:56.924736+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:56.924736+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30315788,"num_objects":3701,"num_object_clones":0,"num_object_copies":14804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8967,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9053","reported_seq":9185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.861702+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:56.861702+0000","last_peered":"2024-09-16T04:47:56.861702+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:56.861702+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9053,"log_dups_size":0,"ondisk_log_size":9053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30209292,"num_objects":3688,"num_object_clones":0,"num_object_copies":14752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9051,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1440}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8748","reported_seq":8858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.893999+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:56.893999+0000","last_peered":"2024-09-16T04:47:56.893999+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:56.893999+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8748,"log_dups_size":0,"ondisk_log_size":8748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29196288,"num_objects":3564,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8748,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8972","reported_seq":9069,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.507514+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:56.507514+0000","last_peered":"2024-09-16T04:47:56.507514+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:56.507514+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8972,"log_dups_size":0,"ondisk_log_size":8972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29717772,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8970,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1415}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9015","reported_seq":13576,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:47:53.665854+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:53.665854+0000","last_peered":"2024-09-16T04:47:53.665854+0000","last_clean":"2024-09-16T04:47:53.665854+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:53.665854+0000","last_undegraded":"2024-09-16T04:47:53.665854+0000","last_fullsized":"2024-09-16T04:47:53.665854+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9015,"log_dups_size":0,"ondisk_log_size":9015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:56.755759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30881036,"num_objects":3770,"num_object_clones":0,"num_object_copies":15080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9137","reported_seq":13894,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:47:57.026136+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:47:57.026136+0000","last_peered":"2024-09-16T04:47:57.026136+0000","last_clean":"2024-09-16T04:47:57.026136+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:47:57.026136+0000","last_undegraded":"2024-09-16T04:47:57.026136+0000","last_fullsized":"2024-09-16T04:47:57.026136+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9137,"log_dups_size":0,"ondisk_log_size":9137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:58.093812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29641240,"num_objects":3619,"num_object_clones":0,"num_object_copies":14476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9134,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8987","reported_seq":9093,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.344914+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:56.344914+0000","last_peered":"2024-09-16T04:47:56.344914+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:56.344914+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8987,"log_dups_size":0,"ondisk_log_size":8987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30889228,"num_objects":3771,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8985,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1582}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8855","reported_seq":8986,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:57.037052+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:57.037052+0000","last_peered":"2024-09-16T04:47:57.037052+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:57.037052+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8855,"log_dups_size":0,"ondisk_log_size":8855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29781906,"num_objects":3636,"num_object_clones":0,"num_object_copies":14544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8852,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1481}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8678","reported_seq":8770,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.842965+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:56.842965+0000","last_peered":"2024-09-16T04:47:56.842965+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:56.842965+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8678,"log_dups_size":0,"ondisk_log_size":8678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29923974,"num_objects":3653,"num_object_clones":0,"num_object_copies":14612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8677,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1548}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8874","reported_seq":8962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.308305+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:56.308305+0000","last_peered":"2024-09-16T04:47:56.308305+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:56.308305+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8874,"log_dups_size":0,"ondisk_log_size":8874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29973126,"num_objects":3659,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8873,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1438}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9010","reported_seq":9142,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.635648+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:56.635648+0000","last_peered":"2024-09-16T04:47:56.635648+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:56.635648+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9010,"log_dups_size":0,"ondisk_log_size":9010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30897420,"num_objects":3772,"num_object_clones":0,"num_object_copies":15088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9008,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1574}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8728","reported_seq":8831,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.340061+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:56.340061+0000","last_peered":"2024-09-16T04:47:56.340061+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:56.340061+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8728,"log_dups_size":0,"ondisk_log_size":8728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30479628,"num_objects":3721,"num_object_clones":0,"num_object_copies":14884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8726,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1598}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8772","reported_seq":8866,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.636741+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:56.636741+0000","last_peered":"2024-09-16T04:47:56.636741+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:56.636741+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8772,"log_dups_size":0,"ondisk_log_size":8772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28955916,"num_objects":3535,"num_object_clones":0,"num_object_copies":14140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8770,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1307}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":481937088,"num_objects":58836,"num_object_clones":0,"num_object_copies":235344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58836,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142651,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4965,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":966193152,"data_stored":966193152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142678,"ondisk_log_size":142678,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740352,"kb_used_data":235248,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93631488,"statfs":{"total":96636764160,"available":95878643712,"internally_reserved":0,"allocated":240893952,"data_stored":240704750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517205711},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,21,0,9,9,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732660,"kb_used_data":236380,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93639180,"statfs":{"total":96636764160,"available":95886520320,"internally_reserved":0,"allocated":242053120,"data_stored":241863918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051662,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608320,"kb_used_data":239276,"kb_used_omap":4,"kb_used_meta":369019,"kb_avail":93763520,"statfs":{"total":96636764160,"available":96013844480,"internally_reserved":0,"allocated":245018624,"data_stored":244829422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":377876175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,10,2,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740676,"kb_used_data":235636,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93631164,"statfs":{"total":96636764160,"available":95878311936,"internally_reserved":0,"allocated":241291264,"data_stored":241102062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":240664576,"data_stored":240664576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":243851264,"data_stored":243851264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":241127424,"data_stored":241127424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":240549888,"data_stored":240549888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:59.660 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:00.265 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:00.266 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:00.292 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-09-16T04:47:58.693534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":481937088,"num_objects":58836,"num_object_clones":0,"num_object_copies":235344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58836,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142651,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4965,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142678,"ondisk_log_size":142678,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2822008,"kb_used_data":946540,"kb_used_omap":14,"kb_used_meta":1875377,"kb_avail":374665352,"statfs":{"total":386547056640,"available":383657320448,"internally_reserved":0,"allocated":969256960,"data_stored":968500152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1920386609},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,22,15,17,19,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16523264,"num_objects":-2017,"num_object_clones":0,"num_object_copies":-8068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2017,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001928"},"pg_stats":[{"pgid":"1.f","version":"35'8960","reported_seq":9076,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:57.017929+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:57.017929+0000","last_peered":"2024-09-16T04:47:57.017929+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:57.017929+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8960,"log_dups_size":0,"ondisk_log_size":8960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30261248,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8960,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8951","reported_seq":9039,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.797790+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:56.797790+0000","last_peered":"2024-09-16T04:47:56.797790+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:56.797790+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8951,"log_dups_size":0,"ondisk_log_size":8951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30496012,"num_objects":3723,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8949,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8969","reported_seq":9066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.280591+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:56.280591+0000","last_peered":"2024-09-16T04:47:56.280591+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:56.280591+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30317214,"num_objects":3702,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8968,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8969","reported_seq":9242,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.924736+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:47:56.924736+0000","last_peered":"2024-09-16T04:47:56.924736+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:56.924736+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30315788,"num_objects":3701,"num_object_clones":0,"num_object_copies":14804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8967,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9053","reported_seq":9185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.861702+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:56.861702+0000","last_peered":"2024-09-16T04:47:56.861702+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:56.861702+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9053,"log_dups_size":0,"ondisk_log_size":9053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30209292,"num_objects":3688,"num_object_clones":0,"num_object_copies":14752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9051,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1440}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8748","reported_seq":8858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.893999+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:56.893999+0000","last_peered":"2024-09-16T04:47:56.893999+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:56.893999+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8748,"log_dups_size":0,"ondisk_log_size":8748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29196288,"num_objects":3564,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8748,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8972","reported_seq":9069,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.507514+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:56.507514+0000","last_peered":"2024-09-16T04:47:56.507514+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:56.507514+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8972,"log_dups_size":0,"ondisk_log_size":8972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29717772,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8970,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1415}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9015","reported_seq":13576,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:47:53.665854+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:53.665854+0000","last_peered":"2024-09-16T04:47:53.665854+0000","last_clean":"2024-09-16T04:47:53.665854+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:53.665854+0000","last_undegraded":"2024-09-16T04:47:53.665854+0000","last_fullsized":"2024-09-16T04:47:53.665854+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9015,"log_dups_size":0,"ondisk_log_size":9015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:56.755759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30881036,"num_objects":3770,"num_object_clones":0,"num_object_copies":15080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9137","reported_seq":13894,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:47:57.026136+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:47:57.026136+0000","last_peered":"2024-09-16T04:47:57.026136+0000","last_clean":"2024-09-16T04:47:57.026136+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:47:57.026136+0000","last_undegraded":"2024-09-16T04:47:57.026136+0000","last_fullsized":"2024-09-16T04:47:57.026136+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9137,"log_dups_size":0,"ondisk_log_size":9137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:58.093812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29641240,"num_objects":3619,"num_object_clones":0,"num_object_copies":14476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9134,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8987","reported_seq":9093,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.344914+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:56.344914+0000","last_peered":"2024-09-16T04:47:56.344914+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:56.344914+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8987,"log_dups_size":0,"ondisk_log_size":8987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30889228,"num_objects":3771,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8985,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1582}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8855","reported_seq":8986,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:57.037052+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:57.037052+0000","last_peered":"2024-09-16T04:47:57.037052+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:57.037052+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8855,"log_dups_size":0,"ondisk_log_size":8855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29781906,"num_objects":3636,"num_object_clones":0,"num_object_copies":14544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8852,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1481}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8678","reported_seq":8770,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.842965+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:56.842965+0000","last_peered":"2024-09-16T04:47:56.842965+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:56.842965+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8678,"log_dups_size":0,"ondisk_log_size":8678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29923974,"num_objects":3653,"num_object_clones":0,"num_object_copies":14612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8677,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1548}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8874","reported_seq":8962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.308305+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:56.308305+0000","last_peered":"2024-09-16T04:47:56.308305+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:56.308305+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8874,"log_dups_size":0,"ondisk_log_size":8874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29973126,"num_objects":3659,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8873,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1438}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9010","reported_seq":9142,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.635648+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:56.635648+0000","last_peered":"2024-09-16T04:47:56.635648+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:56.635648+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9010,"log_dups_size":0,"ondisk_log_size":9010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30897420,"num_objects":3772,"num_object_clones":0,"num_object_copies":15088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9008,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1574}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8728","reported_seq":8831,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.340061+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:56.340061+0000","last_peered":"2024-09-16T04:47:56.340061+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:56.340061+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8728,"log_dups_size":0,"ondisk_log_size":8728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30479628,"num_objects":3721,"num_object_clones":0,"num_object_copies":14884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8726,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1598}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8772","reported_seq":8866,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.636741+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:56.636741+0000","last_peered":"2024-09-16T04:47:56.636741+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:56.636741+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8772,"log_dups_size":0,"ondisk_log_size":8772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28955916,"num_objects":3535,"num_object_clones":0,"num_object_copies":14140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8770,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1307}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":481937088,"num_objects":58836,"num_object_clones":0,"num_object_copies":235344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58836,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142651,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4965,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":966193152,"data_stored":966193152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142678,"ondisk_log_size":142678,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740352,"kb_used_data":235248,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93631488,"statfs":{"total":96636764160,"available":95878643712,"internally_reserved":0,"allocated":240893952,"data_stored":240704750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517205711},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,21,0,9,9,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732660,"kb_used_data":236380,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93639180,"statfs":{"total":96636764160,"available":95886520320,"internally_reserved":0,"allocated":242053120,"data_stored":241863918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051662,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608320,"kb_used_data":239276,"kb_used_omap":4,"kb_used_meta":369019,"kb_avail":93763520,"statfs":{"total":96636764160,"available":96013844480,"internally_reserved":0,"allocated":245018624,"data_stored":244829422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":377876175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,10,2,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740676,"kb_used_data":235636,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93631164,"statfs":{"total":96636764160,"available":95878311936,"internally_reserved":0,"allocated":241291264,"data_stored":241102062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":240664576,"data_stored":240664576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":243851264,"data_stored":243851264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":241127424,"data_stored":241127424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":240549888,"data_stored":240549888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:00.293 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:48:00.293 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8960", 'reported_seq': 9076, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:57.017929+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:47:57.017929+0000', 'last_peered': '2024-09-16T04:47:57.017929+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:47:57.017929+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8960, 'log_dups_size': 0, 'ondisk_log_size': 8960, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30261248, 'num_objects': 3694, 'num_object_clones': 0, 'num_object_copies': 14776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3694, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8960, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1508}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:00.293 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:48:00.293 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'8951", 'reported_seq': 9039, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:56.797790+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:47:56.797790+0000', 'last_peered': '2024-09-16T04:47:56.797790+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:47:56.797790+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8951, 'log_dups_size': 0, 'ondisk_log_size': 8951, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30496012, 'num_objects': 3723, 'num_object_clones': 0, 'num_object_copies': 14892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3723, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8949, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1520}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:00.293 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:48:00.293 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8969", 'reported_seq': 9066, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:56.280591+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:47:56.280591+0000', 'last_peered': '2024-09-16T04:47:56.280591+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:47:56.280591+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8969, 'log_dups_size': 0, 'ondisk_log_size': 8969, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30317214, 'num_objects': 3702, 'num_object_clones': 0, 'num_object_copies': 14808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3702, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8968, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1483}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:00.293 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:48:00.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'8969", 'reported_seq': 9242, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:56.924736+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:47:56.924736+0000', 'last_peered': '2024-09-16T04:47:56.924736+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:47:56.924736+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8969, 'log_dups_size': 0, 'ondisk_log_size': 8969, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30315788, 'num_objects': 3701, 'num_object_clones': 0, 'num_object_copies': 14804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2131, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3701, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8967, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1570}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:00.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:48:00.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9053", 'reported_seq': 9185, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:56.861702+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:47:56.861702+0000', 'last_peered': '2024-09-16T04:47:56.861702+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:47:56.861702+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9053, 'log_dups_size': 0, 'ondisk_log_size': 9053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30209292, 'num_objects': 3688, 'num_object_clones': 0, 'num_object_copies': 14752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9051, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1440}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:00.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:48:00.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8748", 'reported_seq': 8858, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:56.893999+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:47:56.893999+0000', 'last_peered': '2024-09-16T04:47:56.893999+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:47:56.893999+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8748, 'log_dups_size': 0, 'ondisk_log_size': 8748, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29196288, 'num_objects': 3564, 'num_object_clones': 0, 'num_object_copies': 14256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8748, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1386}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:00.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:48:00.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8972", 'reported_seq': 9069, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:56.507514+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:47:56.507514+0000', 'last_peered': '2024-09-16T04:47:56.507514+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:47:56.507514+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8972, 'log_dups_size': 0, 'ondisk_log_size': 8972, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29717772, 'num_objects': 3628, 'num_object_clones': 0, 'num_object_copies': 14512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8970, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1415}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:00.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:48:00.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8987", 'reported_seq': 9093, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:56.344914+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:47:56.344914+0000', 'last_peered': '2024-09-16T04:47:56.344914+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:47:56.344914+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8987, 'log_dups_size': 0, 'ondisk_log_size': 8987, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30889228, 'num_objects': 3771, 'num_object_clones': 0, 'num_object_copies': 15084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3771, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8985, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1582}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:00.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:48:00.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8855", 'reported_seq': 8986, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:57.037052+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:47:57.037052+0000', 'last_peered': '2024-09-16T04:47:57.037052+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:47:57.037052+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8855, 'log_dups_size': 0, 'ondisk_log_size': 8855, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29781906, 'num_objects': 3636, 'num_object_clones': 0, 'num_object_copies': 14544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8852, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1481}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:00.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:48:00.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8678", 'reported_seq': 8770, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:56.842965+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:47:56.842965+0000', 'last_peered': '2024-09-16T04:47:56.842965+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:47:56.842965+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8678, 'log_dups_size': 0, 'ondisk_log_size': 8678, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29923974, 'num_objects': 3653, 'num_object_clones': 0, 'num_object_copies': 14612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8677, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1548}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:00.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:48:00.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8874", 'reported_seq': 8962, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:56.308305+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:47:56.308305+0000', 'last_peered': '2024-09-16T04:47:56.308305+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:47:56.308305+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8874, 'log_dups_size': 0, 'ondisk_log_size': 8874, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29973126, 'num_objects': 3659, 'num_object_clones': 0, 'num_object_copies': 14636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3659, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8873, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1438}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:00.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:48:00.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9010", 'reported_seq': 9142, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:56.635648+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:47:56.635648+0000', 'last_peered': '2024-09-16T04:47:56.635648+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:47:56.635648+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9010, 'log_dups_size': 0, 'ondisk_log_size': 9010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30897420, 'num_objects': 3772, 'num_object_clones': 0, 'num_object_copies': 15088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9008, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1574}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:00.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:48:00.297 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'8728", 'reported_seq': 8831, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:56.340061+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:47:56.340061+0000', 'last_peered': '2024-09-16T04:47:56.340061+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:47:56.340061+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8728, 'log_dups_size': 0, 'ondisk_log_size': 8728, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30479628, 'num_objects': 3721, 'num_object_clones': 0, 'num_object_copies': 14884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3721, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8726, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1598}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:00.297 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:48:00.297 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8772", 'reported_seq': 8866, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:47:56.636741+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:47:56.636741+0000', 'last_peered': '2024-09-16T04:47:56.636741+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:47:56.636741+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8772, 'log_dups_size': 0, 'ondisk_log_size': 8772, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28955916, 'num_objects': 3535, 'num_object_clones': 0, 'num_object_copies': 14140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3535, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8770, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1307}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:00.297 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:48:01.041 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:01.041 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30617/235008 objects degraded (13.028%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":493,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":58752,"data_bytes":481248960,"bytes_used":2871742464,"bytes_avail":383675314176,"bytes_total":386547056640,"degraded_objects":30617,"degraded_total":235008,"degraded_ratio":0.13028067129629631,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":225},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T04:47:26.684344+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:48:01.066 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:48:01.067 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:01.746 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:01.746 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:01.771 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-09-16T04:48:00.694035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":481248960,"num_objects":58752,"num_object_clones":0,"num_object_copies":235008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58752,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142735,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4965,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142762,"ondisk_log_size":142762,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2804436,"kb_used_data":941452,"kb_used_omap":14,"kb_used_meta":1862897,"kb_avail":374682924,"statfs":{"total":386547056640,"available":383675314176,"internally_reserved":0,"allocated":964046848,"data_stored":963290040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1907607089},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,29,6,17,14,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17211392,"num_objects":-2101,"num_object_clones":0,"num_object_copies":-8404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2101,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.332692"},"pg_stats":[{"pgid":"1.f","version":"35'8960","reported_seq":9076,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:57.017929+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:47:57.017929+0000","last_peered":"2024-09-16T04:47:57.017929+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:47:57.017929+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8960,"log_dups_size":0,"ondisk_log_size":8960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30261248,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8960,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8951","reported_seq":9039,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.797790+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:47:56.797790+0000","last_peered":"2024-09-16T04:47:56.797790+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:47:56.797790+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8951,"log_dups_size":0,"ondisk_log_size":8951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30496012,"num_objects":3723,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8949,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8969","reported_seq":9066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.280591+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:47:56.280591+0000","last_peered":"2024-09-16T04:47:56.280591+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:47:56.280591+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30317214,"num_objects":3702,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8968,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8969","reported_seq":9242,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.924736+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:47:56.924736+0000","last_peered":"2024-09-16T04:47:56.924736+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:47:56.924736+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30315788,"num_objects":3701,"num_object_clones":0,"num_object_copies":14804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8967,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9053","reported_seq":9185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.861702+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:47:56.861702+0000","last_peered":"2024-09-16T04:47:56.861702+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:47:56.861702+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9053,"log_dups_size":0,"ondisk_log_size":9053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30209292,"num_objects":3688,"num_object_clones":0,"num_object_copies":14752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9051,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1440}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8748","reported_seq":8858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.893999+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:47:56.893999+0000","last_peered":"2024-09-16T04:47:56.893999+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:47:56.893999+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8748,"log_dups_size":0,"ondisk_log_size":8748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29196288,"num_objects":3564,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8748,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8972","reported_seq":9069,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.507514+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:47:56.507514+0000","last_peered":"2024-09-16T04:47:56.507514+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:47:56.507514+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8972,"log_dups_size":0,"ondisk_log_size":8972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29717772,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8970,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1415}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9099","reported_seq":13660,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:47:58.915043+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:47:58.915043+0000","last_peered":"2024-09-16T04:47:58.915043+0000","last_clean":"2024-09-16T04:47:58.915043+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:47:58.915043+0000","last_undegraded":"2024-09-16T04:47:58.915043+0000","last_fullsized":"2024-09-16T04:47:58.915043+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:01.812884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30192908,"num_objects":3686,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9097,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9137","reported_seq":13894,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:47:57.026136+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:47:57.026136+0000","last_peered":"2024-09-16T04:47:57.026136+0000","last_clean":"2024-09-16T04:47:57.026136+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:47:57.026136+0000","last_undegraded":"2024-09-16T04:47:57.026136+0000","last_fullsized":"2024-09-16T04:47:57.026136+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9137,"log_dups_size":0,"ondisk_log_size":9137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:47:58.093812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29641240,"num_objects":3619,"num_object_clones":0,"num_object_copies":14476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9134,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8987","reported_seq":9093,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.344914+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:47:56.344914+0000","last_peered":"2024-09-16T04:47:56.344914+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:47:56.344914+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8987,"log_dups_size":0,"ondisk_log_size":8987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30889228,"num_objects":3771,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8985,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1582}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8855","reported_seq":8986,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:57.037052+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:47:57.037052+0000","last_peered":"2024-09-16T04:47:57.037052+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:47:57.037052+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8855,"log_dups_size":0,"ondisk_log_size":8855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29781906,"num_objects":3636,"num_object_clones":0,"num_object_copies":14544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8852,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1481}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8678","reported_seq":8770,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.842965+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:47:56.842965+0000","last_peered":"2024-09-16T04:47:56.842965+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:47:56.842965+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8678,"log_dups_size":0,"ondisk_log_size":8678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29923974,"num_objects":3653,"num_object_clones":0,"num_object_copies":14612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8677,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1548}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8874","reported_seq":8962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.308305+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:47:56.308305+0000","last_peered":"2024-09-16T04:47:56.308305+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:47:56.308305+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8874,"log_dups_size":0,"ondisk_log_size":8874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29973126,"num_objects":3659,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8873,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1438}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9010","reported_seq":9142,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.635648+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:47:56.635648+0000","last_peered":"2024-09-16T04:47:56.635648+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:47:56.635648+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9010,"log_dups_size":0,"ondisk_log_size":9010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30897420,"num_objects":3772,"num_object_clones":0,"num_object_copies":15088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9008,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1574}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8728","reported_seq":8831,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.340061+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:47:56.340061+0000","last_peered":"2024-09-16T04:47:56.340061+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:47:56.340061+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8728,"log_dups_size":0,"ondisk_log_size":8728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30479628,"num_objects":3721,"num_object_clones":0,"num_object_copies":14884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8726,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1598}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8772","reported_seq":8866,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:47:56.636741+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:47:56.636741+0000","last_peered":"2024-09-16T04:47:56.636741+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:47:56.636741+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8772,"log_dups_size":0,"ondisk_log_size":8772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28955916,"num_objects":3535,"num_object_clones":0,"num_object_copies":14140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8770,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1307}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":481248960,"num_objects":58752,"num_object_clones":0,"num_object_copies":235008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58752,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142735,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4965,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":961802240,"data_stored":961802240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142762,"ondisk_log_size":142762,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740352,"kb_used_data":235248,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93631488,"statfs":{"total":96636764160,"available":95878643712,"internally_reserved":0,"allocated":240893952,"data_stored":240704750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517205711},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,21,0,9,9,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732660,"kb_used_data":236380,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93639180,"statfs":{"total":96636764160,"available":95886520320,"internally_reserved":0,"allocated":242053120,"data_stored":241863918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051663,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590748,"kb_used_data":234188,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93781092,"statfs":{"total":96636764160,"available":96031838208,"internally_reserved":0,"allocated":239808512,"data_stored":239619310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365096655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,7,1,2,2,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740676,"kb_used_data":235636,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93631164,"statfs":{"total":96636764160,"available":95878311936,"internally_reserved":0,"allocated":241291264,"data_stored":241102062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":240664576,"data_stored":240664576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":239460352,"data_stored":239460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":241127424,"data_stored":241127424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":240549888,"data_stored":240549888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:04.773 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:05.334 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:05.334 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:05.356 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-09-16T04:48:04.695020+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":472712896,"num_objects":57710,"num_object_clones":0,"num_object_copies":230840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57710,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143777,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5011,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143804,"ondisk_log_size":143804,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2804360,"kb_used_data":924736,"kb_used_omap":14,"kb_used_meta":1879537,"kb_avail":374683000,"statfs":{"total":386547056640,"available":383675392000,"internally_reserved":0,"allocated":946929664,"data_stored":946172856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1924646449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,23,3,22,25,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17580032,"num_objects":-2146,"num_object_clones":0,"num_object_copies":-8584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2146,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002938"},"pg_stats":[{"pgid":"1.f","version":"35'9027","reported_seq":9143,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:02.004166+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:02.004166+0000","last_peered":"2024-09-16T04:48:02.004166+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:02.004166+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9027,"log_dups_size":0,"ondisk_log_size":9027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29712384,"num_objects":3627,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9027,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1441}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9015","reported_seq":9103,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.690266+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:01.690266+0000","last_peered":"2024-09-16T04:48:01.690266+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:01.690266+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9015,"log_dups_size":0,"ondisk_log_size":9015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29971724,"num_objects":3659,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1456}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9039","reported_seq":9136,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.294759+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:01.294759+0000","last_peered":"2024-09-16T04:48:01.294759+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:01.294759+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9039,"log_dups_size":0,"ondisk_log_size":9039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29743774,"num_objects":3632,"num_object_clones":0,"num_object_copies":14528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3632,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9038,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1413}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9042","reported_seq":9407,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.893208+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:01.893208+0000","last_peered":"2024-09-16T04:48:01.893208+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:01.893208+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9042,"log_dups_size":0,"ondisk_log_size":9042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29717772,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9040,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1543}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9124","reported_seq":9256,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:02.032005+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:02.032005+0000","last_peered":"2024-09-16T04:48:02.032005+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:02.032005+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9124,"log_dups_size":0,"ondisk_log_size":9124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29627660,"num_objects":3617,"num_object_clones":0,"num_object_copies":14468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9122,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8811","reported_seq":8921,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:02.022069+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:02.022069+0000","last_peered":"2024-09-16T04:48:02.022069+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:02.022069+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8811,"log_dups_size":0,"ondisk_log_size":8811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28680192,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8811,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9034","reported_seq":9131,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.900851+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:01.900851+0000","last_peered":"2024-09-16T04:48:01.900851+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:01.900851+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9034,"log_dups_size":0,"ondisk_log_size":9034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29209868,"num_objects":3566,"num_object_clones":0,"num_object_copies":14264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9032,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1353}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9165","reported_seq":13725,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:03.831053+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:03.831053+0000","last_peered":"2024-09-16T04:48:03.831053+0000","last_clean":"2024-09-16T04:48:03.831053+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:03.831053+0000","last_undegraded":"2024-09-16T04:48:03.831053+0000","last_fullsized":"2024-09-16T04:48:03.831053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9165,"log_dups_size":0,"ondisk_log_size":9165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:06.967438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29652236,"num_objects":3620,"num_object_clones":0,"num_object_copies":14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9163,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9194","reported_seq":13951,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:01.954798+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:01.954798+0000","last_peered":"2024-09-16T04:48:01.954798+0000","last_clean":"2024-09-16T04:48:01.954798+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:01.954798+0000","last_undegraded":"2024-09-16T04:48:01.954798+0000","last_fullsized":"2024-09-16T04:48:01.954798+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9194,"log_dups_size":0,"ondisk_log_size":9194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:03.953796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29174296,"num_objects":3562,"num_object_clones":0,"num_object_copies":14248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9191,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9045","reported_seq":9151,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.166580+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:01.166580+0000","last_peered":"2024-09-16T04:48:01.166580+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:01.166580+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9045,"log_dups_size":0,"ondisk_log_size":9045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30414092,"num_objects":3713,"num_object_clones":0,"num_object_copies":14852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9043,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1524}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8915","reported_seq":9047,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.951254+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:01.951254+0000","last_peered":"2024-09-16T04:48:01.951254+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:01.951254+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8915,"log_dups_size":0,"ondisk_log_size":8915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29290386,"num_objects":3576,"num_object_clones":0,"num_object_copies":14304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8912,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8755","reported_seq":8846,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.997351+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:01.997351+0000","last_peered":"2024-09-16T04:48:01.997351+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:01.997351+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8755,"log_dups_size":0,"ondisk_log_size":8755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29293190,"num_objects":3576,"num_object_clones":0,"num_object_copies":14304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8754,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1471}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8932","reported_seq":9020,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.363859+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:01.363859+0000","last_peered":"2024-09-16T04:48:01.363859+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:01.363859+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8932,"log_dups_size":0,"ondisk_log_size":8932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29497990,"num_objects":3601,"num_object_clones":0,"num_object_copies":14404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8931,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1380}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9076","reported_seq":9208,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:02.074850+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:02.074850+0000","last_peered":"2024-09-16T04:48:02.074850+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:02.074850+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9076,"log_dups_size":0,"ondisk_log_size":9076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30356748,"num_objects":3706,"num_object_clones":0,"num_object_copies":14824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9074,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1508}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8792","reported_seq":8895,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.223508+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:01.223508+0000","last_peered":"2024-09-16T04:48:01.223508+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:01.223508+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29955340,"num_objects":3657,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8790,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1534}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8838","reported_seq":8932,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.718220+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:01.718220+0000","last_peered":"2024-09-16T04:48:01.718220+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:01.718220+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8838,"log_dups_size":0,"ondisk_log_size":8838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28415244,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8836,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1241}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":472712896,"num_objects":57710,"num_object_clones":0,"num_object_copies":230840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57710,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143777,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5011,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":944533504,"data_stored":944533504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143804,"ondisk_log_size":143804,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740400,"kb_used_data":231136,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93631440,"statfs":{"total":96636764160,"available":95878594560,"internally_reserved":0,"allocated":236683264,"data_stored":236494062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521465551},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,21,0,9,9,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732568,"kb_used_data":232136,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93639272,"statfs":{"total":96636764160,"available":95886614528,"internally_reserved":0,"allocated":237707264,"data_stored":237518062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051664,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590680,"kb_used_data":229952,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93781160,"statfs":{"total":96636764160,"available":96031907840,"internally_reserved":0,"allocated":235470848,"data_stored":235281646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369356495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,5,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247902,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740712,"kb_used_data":231512,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93631128,"statfs":{"total":96636764160,"available":95878275072,"internally_reserved":0,"allocated":237068288,"data_stored":236879086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":521400014},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":236388352,"data_stored":236388352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":235139072,"data_stored":235139072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":236802048,"data_stored":236802048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":236204032,"data_stored":236204032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:05.357 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:05.992 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:05.993 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:06.016 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-09-16T04:48:04.695020+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":472712896,"num_objects":57710,"num_object_clones":0,"num_object_copies":230840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57710,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143777,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5011,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143804,"ondisk_log_size":143804,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2804360,"kb_used_data":924736,"kb_used_omap":14,"kb_used_meta":1879537,"kb_avail":374683000,"statfs":{"total":386547056640,"available":383675392000,"internally_reserved":0,"allocated":946929664,"data_stored":946172856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1924646449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,23,3,22,25,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17580032,"num_objects":-2146,"num_object_clones":0,"num_object_copies":-8584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2146,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002938"},"pg_stats":[{"pgid":"1.f","version":"35'9027","reported_seq":9143,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:02.004166+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:02.004166+0000","last_peered":"2024-09-16T04:48:02.004166+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:02.004166+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9027,"log_dups_size":0,"ondisk_log_size":9027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29712384,"num_objects":3627,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9027,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1441}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9015","reported_seq":9103,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.690266+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:01.690266+0000","last_peered":"2024-09-16T04:48:01.690266+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:01.690266+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9015,"log_dups_size":0,"ondisk_log_size":9015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29971724,"num_objects":3659,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1456}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9039","reported_seq":9136,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.294759+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:01.294759+0000","last_peered":"2024-09-16T04:48:01.294759+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:01.294759+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9039,"log_dups_size":0,"ondisk_log_size":9039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29743774,"num_objects":3632,"num_object_clones":0,"num_object_copies":14528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3632,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9038,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1413}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9042","reported_seq":9407,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.893208+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:01.893208+0000","last_peered":"2024-09-16T04:48:01.893208+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:01.893208+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9042,"log_dups_size":0,"ondisk_log_size":9042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29717772,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9040,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1543}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9124","reported_seq":9256,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:02.032005+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:02.032005+0000","last_peered":"2024-09-16T04:48:02.032005+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:02.032005+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9124,"log_dups_size":0,"ondisk_log_size":9124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29627660,"num_objects":3617,"num_object_clones":0,"num_object_copies":14468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9122,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8811","reported_seq":8921,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:02.022069+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:02.022069+0000","last_peered":"2024-09-16T04:48:02.022069+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:02.022069+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8811,"log_dups_size":0,"ondisk_log_size":8811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28680192,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8811,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9034","reported_seq":9131,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.900851+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:01.900851+0000","last_peered":"2024-09-16T04:48:01.900851+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:01.900851+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9034,"log_dups_size":0,"ondisk_log_size":9034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29209868,"num_objects":3566,"num_object_clones":0,"num_object_copies":14264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9032,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1353}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9165","reported_seq":13725,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:03.831053+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:03.831053+0000","last_peered":"2024-09-16T04:48:03.831053+0000","last_clean":"2024-09-16T04:48:03.831053+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:03.831053+0000","last_undegraded":"2024-09-16T04:48:03.831053+0000","last_fullsized":"2024-09-16T04:48:03.831053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9165,"log_dups_size":0,"ondisk_log_size":9165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:06.967438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29652236,"num_objects":3620,"num_object_clones":0,"num_object_copies":14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9163,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9194","reported_seq":13951,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:01.954798+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:01.954798+0000","last_peered":"2024-09-16T04:48:01.954798+0000","last_clean":"2024-09-16T04:48:01.954798+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:01.954798+0000","last_undegraded":"2024-09-16T04:48:01.954798+0000","last_fullsized":"2024-09-16T04:48:01.954798+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9194,"log_dups_size":0,"ondisk_log_size":9194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:03.953796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29174296,"num_objects":3562,"num_object_clones":0,"num_object_copies":14248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9191,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9045","reported_seq":9151,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.166580+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:01.166580+0000","last_peered":"2024-09-16T04:48:01.166580+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:01.166580+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9045,"log_dups_size":0,"ondisk_log_size":9045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30414092,"num_objects":3713,"num_object_clones":0,"num_object_copies":14852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9043,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1524}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8915","reported_seq":9047,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.951254+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:01.951254+0000","last_peered":"2024-09-16T04:48:01.951254+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:01.951254+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8915,"log_dups_size":0,"ondisk_log_size":8915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29290386,"num_objects":3576,"num_object_clones":0,"num_object_copies":14304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8912,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8755","reported_seq":8846,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.997351+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:01.997351+0000","last_peered":"2024-09-16T04:48:01.997351+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:01.997351+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8755,"log_dups_size":0,"ondisk_log_size":8755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29293190,"num_objects":3576,"num_object_clones":0,"num_object_copies":14304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8754,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1471}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'8932","reported_seq":9020,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.363859+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:01.363859+0000","last_peered":"2024-09-16T04:48:01.363859+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:01.363859+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8932,"log_dups_size":0,"ondisk_log_size":8932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29497990,"num_objects":3601,"num_object_clones":0,"num_object_copies":14404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8931,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1380}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9076","reported_seq":9208,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:02.074850+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:02.074850+0000","last_peered":"2024-09-16T04:48:02.074850+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:02.074850+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9076,"log_dups_size":0,"ondisk_log_size":9076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30356748,"num_objects":3706,"num_object_clones":0,"num_object_copies":14824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9074,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1508}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8792","reported_seq":8895,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.223508+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:01.223508+0000","last_peered":"2024-09-16T04:48:01.223508+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:01.223508+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29955340,"num_objects":3657,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8790,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1534}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8838","reported_seq":8932,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.718220+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:01.718220+0000","last_peered":"2024-09-16T04:48:01.718220+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:01.718220+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8838,"log_dups_size":0,"ondisk_log_size":8838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28415244,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8836,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1241}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":472712896,"num_objects":57710,"num_object_clones":0,"num_object_copies":230840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57710,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143777,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5011,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":944533504,"data_stored":944533504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143804,"ondisk_log_size":143804,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740400,"kb_used_data":231136,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93631440,"statfs":{"total":96636764160,"available":95878594560,"internally_reserved":0,"allocated":236683264,"data_stored":236494062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521465551},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,21,0,9,9,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732568,"kb_used_data":232136,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93639272,"statfs":{"total":96636764160,"available":95886614528,"internally_reserved":0,"allocated":237707264,"data_stored":237518062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051664,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590680,"kb_used_data":229952,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93781160,"statfs":{"total":96636764160,"available":96031907840,"internally_reserved":0,"allocated":235470848,"data_stored":235281646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369356495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,5,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247902,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740712,"kb_used_data":231512,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93631128,"statfs":{"total":96636764160,"available":95878275072,"internally_reserved":0,"allocated":237068288,"data_stored":236879086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":521400014},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":236388352,"data_stored":236388352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":235139072,"data_stored":235139072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":236802048,"data_stored":236802048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":236204032,"data_stored":236204032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:06.017 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:48:06.018 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9027", 'reported_seq': 9143, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:02.004166+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:48:02.004166+0000', 'last_peered': '2024-09-16T04:48:02.004166+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:48:02.004166+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9027, 'log_dups_size': 0, 'ondisk_log_size': 9027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29712384, 'num_objects': 3627, 'num_object_clones': 0, 'num_object_copies': 14508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9027, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1441}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:06.018 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:48:06.018 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9015", 'reported_seq': 9103, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:01.690266+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:48:01.690266+0000', 'last_peered': '2024-09-16T04:48:01.690266+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:48:01.690266+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9015, 'log_dups_size': 0, 'ondisk_log_size': 9015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29971724, 'num_objects': 3659, 'num_object_clones': 0, 'num_object_copies': 14636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3659, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9013, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1456}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:06.018 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:48:06.018 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9039", 'reported_seq': 9136, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:01.294759+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:48:01.294759+0000', 'last_peered': '2024-09-16T04:48:01.294759+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:48:01.294759+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9039, 'log_dups_size': 0, 'ondisk_log_size': 9039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29743774, 'num_objects': 3632, 'num_object_clones': 0, 'num_object_copies': 14528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3632, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9038, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1413}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:06.019 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:48:06.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9042", 'reported_seq': 9407, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:01.893208+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:48:01.893208+0000', 'last_peered': '2024-09-16T04:48:01.893208+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:48:01.893208+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9042, 'log_dups_size': 0, 'ondisk_log_size': 9042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29717772, 'num_objects': 3628, 'num_object_clones': 0, 'num_object_copies': 14512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2085, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9040, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 148, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1543}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:06.019 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:48:06.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9124", 'reported_seq': 9256, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:02.032005+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:48:02.032005+0000', 'last_peered': '2024-09-16T04:48:02.032005+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:48:02.032005+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9124, 'log_dups_size': 0, 'ondisk_log_size': 9124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29627660, 'num_objects': 3617, 'num_object_clones': 0, 'num_object_copies': 14468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3617, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9122, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1369}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:06.019 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:48:06.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8811", 'reported_seq': 8921, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:02.022069+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:48:02.022069+0000', 'last_peered': '2024-09-16T04:48:02.022069+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:48:02.022069+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8811, 'log_dups_size': 0, 'ondisk_log_size': 8811, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28680192, 'num_objects': 3501, 'num_object_clones': 0, 'num_object_copies': 14004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8811, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1323}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:06.019 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:48:06.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9034", 'reported_seq': 9131, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:01.900851+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:48:01.900851+0000', 'last_peered': '2024-09-16T04:48:01.900851+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:48:01.900851+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9034, 'log_dups_size': 0, 'ondisk_log_size': 9034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29209868, 'num_objects': 3566, 'num_object_clones': 0, 'num_object_copies': 14264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3566, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9032, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1353}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:06.019 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:48:06.019 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9045", 'reported_seq': 9151, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:01.166580+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:48:01.166580+0000', 'last_peered': '2024-09-16T04:48:01.166580+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:48:01.166580+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9045, 'log_dups_size': 0, 'ondisk_log_size': 9045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30414092, 'num_objects': 3713, 'num_object_clones': 0, 'num_object_copies': 14852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9043, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1524}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:06.020 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:48:06.020 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8915", 'reported_seq': 9047, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:01.951254+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:48:01.951254+0000', 'last_peered': '2024-09-16T04:48:01.951254+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:48:01.951254+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8915, 'log_dups_size': 0, 'ondisk_log_size': 8915, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29290386, 'num_objects': 3576, 'num_object_clones': 0, 'num_object_copies': 14304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8912, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1421}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:06.020 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:48:06.020 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8755", 'reported_seq': 8846, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:01.997351+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:48:01.997351+0000', 'last_peered': '2024-09-16T04:48:01.997351+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:48:01.997351+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8755, 'log_dups_size': 0, 'ondisk_log_size': 8755, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29293190, 'num_objects': 3576, 'num_object_clones': 0, 'num_object_copies': 14304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8754, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1471}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:06.020 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:48:06.020 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8932", 'reported_seq': 9020, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:01.363859+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:48:01.363859+0000', 'last_peered': '2024-09-16T04:48:01.363859+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:48:01.363859+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8932, 'log_dups_size': 0, 'ondisk_log_size': 8932, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29497990, 'num_objects': 3601, 'num_object_clones': 0, 'num_object_copies': 14404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3601, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8931, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1380}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:06.020 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:48:06.020 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9076", 'reported_seq': 9208, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:02.074850+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:48:02.074850+0000', 'last_peered': '2024-09-16T04:48:02.074850+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:48:02.074850+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9076, 'log_dups_size': 0, 'ondisk_log_size': 9076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30356748, 'num_objects': 3706, 'num_object_clones': 0, 'num_object_copies': 14824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9074, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1508}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:06.020 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:48:06.020 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'8792", 'reported_seq': 8895, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:01.223508+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:48:01.223508+0000', 'last_peered': '2024-09-16T04:48:01.223508+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:48:01.223508+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8792, 'log_dups_size': 0, 'ondisk_log_size': 8792, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29955340, 'num_objects': 3657, 'num_object_clones': 0, 'num_object_copies': 14628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3657, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8790, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:06.020 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:48:06.021 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8838", 'reported_seq': 8932, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:01.718220+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:48:01.718220+0000', 'last_peered': '2024-09-16T04:48:01.718220+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:48:01.718220+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8838, 'log_dups_size': 0, 'ondisk_log_size': 8838, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28415244, 'num_objects': 3469, 'num_object_clones': 0, 'num_object_copies': 13876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8836, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1241}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:06.021 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:48:06.708 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:06.709 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30571/230840 objects degraded (13.243%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":499,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":57710,"data_bytes":472712896,"bytes_used":2871664640,"bytes_avail":383675392000,"bytes_total":386547056640,"degraded_objects":30571,"degraded_total":230840,"degraded_ratio":0.13243372032576675,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":178},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T04:47:26.684344+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:48:06.746 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:48:06.746 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:07.456 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:07.458 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:07.498 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-09-16T04:48:06.695489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":470386368,"num_objects":57426,"num_object_clones":0,"num_object_copies":229704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57426,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144061,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5011,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144088,"ondisk_log_size":144088,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2804264,"kb_used_data":920480,"kb_used_omap":14,"kb_used_meta":1883697,"kb_avail":374683096,"statfs":{"total":386547056640,"available":383675490304,"internally_reserved":0,"allocated":942571520,"data_stored":941814712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1928906289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,25,5,23,20,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19365888,"num_objects":-2364,"num_object_clones":0,"num_object_copies":-9456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2364,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002946"},"pg_stats":[{"pgid":"1.f","version":"35'9027","reported_seq":9143,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:02.004166+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:02.004166+0000","last_peered":"2024-09-16T04:48:02.004166+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:02.004166+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9027,"log_dups_size":0,"ondisk_log_size":9027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29712384,"num_objects":3627,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9027,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1441}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9015","reported_seq":9103,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.690266+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:01.690266+0000","last_peered":"2024-09-16T04:48:01.690266+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:01.690266+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9015,"log_dups_size":0,"ondisk_log_size":9015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29971724,"num_objects":3659,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1456}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9113","reported_seq":9211,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.277817+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:06.277817+0000","last_peered":"2024-09-16T04:48:06.277817+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:06.277817+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9113,"log_dups_size":0,"ondisk_log_size":9113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29137566,"num_objects":3558,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9112,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9042","reported_seq":9407,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.893208+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:01.893208+0000","last_peered":"2024-09-16T04:48:01.893208+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:01.893208+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9042,"log_dups_size":0,"ondisk_log_size":9042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29717772,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9040,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1543}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9124","reported_seq":9256,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:02.032005+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:02.032005+0000","last_peered":"2024-09-16T04:48:02.032005+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:02.032005+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9124,"log_dups_size":0,"ondisk_log_size":9124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29627660,"num_objects":3617,"num_object_clones":0,"num_object_copies":14468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9122,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8811","reported_seq":8921,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:02.022069+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:02.022069+0000","last_peered":"2024-09-16T04:48:02.022069+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:02.022069+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8811,"log_dups_size":0,"ondisk_log_size":8811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28680192,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8811,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9034","reported_seq":9131,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.900851+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:01.900851+0000","last_peered":"2024-09-16T04:48:01.900851+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:01.900851+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9034,"log_dups_size":0,"ondisk_log_size":9034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29209868,"num_objects":3566,"num_object_clones":0,"num_object_copies":14264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9032,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1353}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9165","reported_seq":13725,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:03.831053+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:03.831053+0000","last_peered":"2024-09-16T04:48:03.831053+0000","last_clean":"2024-09-16T04:48:03.831053+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:03.831053+0000","last_undegraded":"2024-09-16T04:48:03.831053+0000","last_fullsized":"2024-09-16T04:48:03.831053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9165,"log_dups_size":0,"ondisk_log_size":9165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:06.967438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29652236,"num_objects":3620,"num_object_clones":0,"num_object_copies":14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9163,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9194","reported_seq":13951,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:01.954798+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:01.954798+0000","last_peered":"2024-09-16T04:48:01.954798+0000","last_clean":"2024-09-16T04:48:01.954798+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:01.954798+0000","last_undegraded":"2024-09-16T04:48:01.954798+0000","last_fullsized":"2024-09-16T04:48:01.954798+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9194,"log_dups_size":0,"ondisk_log_size":9194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:03.953796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29174296,"num_objects":3562,"num_object_clones":0,"num_object_copies":14248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9191,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9118","reported_seq":9224,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.318537+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:06.318537+0000","last_peered":"2024-09-16T04:48:06.318537+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:06.318537+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9118,"log_dups_size":0,"ondisk_log_size":9118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29816076,"num_objects":3640,"num_object_clones":0,"num_object_copies":14560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9116,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1451}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8915","reported_seq":9047,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.951254+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:01.951254+0000","last_peered":"2024-09-16T04:48:01.951254+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:01.951254+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8915,"log_dups_size":0,"ondisk_log_size":8915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29290386,"num_objects":3576,"num_object_clones":0,"num_object_copies":14304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8912,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8755","reported_seq":8846,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.997351+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:01.997351+0000","last_peered":"2024-09-16T04:48:01.997351+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:01.997351+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8755,"log_dups_size":0,"ondisk_log_size":8755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29293190,"num_objects":3576,"num_object_clones":0,"num_object_copies":14304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8754,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1471}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9002","reported_seq":9090,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.124563+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:06.124563+0000","last_peered":"2024-09-16T04:48:06.124563+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:06.124563+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9002,"log_dups_size":0,"ondisk_log_size":9002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28924550,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9001,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1310}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9076","reported_seq":9208,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:02.074850+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:02.074850+0000","last_peered":"2024-09-16T04:48:02.074850+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:02.074850+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9076,"log_dups_size":0,"ondisk_log_size":9076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30356748,"num_objects":3706,"num_object_clones":0,"num_object_copies":14824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9074,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1508}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8859","reported_seq":8962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.402237+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:06.402237+0000","last_peered":"2024-09-16T04:48:06.402237+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:06.402237+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8859,"log_dups_size":0,"ondisk_log_size":8859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29406476,"num_objects":3590,"num_object_clones":0,"num_object_copies":14360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8857,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1467}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8838","reported_seq":8932,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:01.718220+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:01.718220+0000","last_peered":"2024-09-16T04:48:01.718220+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:01.718220+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8838,"log_dups_size":0,"ondisk_log_size":8838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28415244,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8836,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1241}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":470386368,"num_objects":57426,"num_object_clones":0,"num_object_copies":229704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57426,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144061,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5011,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":940183552,"data_stored":940183552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144088,"ondisk_log_size":144088,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740400,"kb_used_data":231136,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93631440,"statfs":{"total":96636764160,"available":95878594560,"internally_reserved":0,"allocated":236683264,"data_stored":236494062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521465551},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,21,0,9,9,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732472,"kb_used_data":227880,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93639368,"statfs":{"total":96636764160,"available":95886712832,"internally_reserved":0,"allocated":233349120,"data_stored":233159918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051664,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590680,"kb_used_data":229952,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93781160,"statfs":{"total":96636764160,"available":96031907840,"internally_reserved":0,"allocated":235470848,"data_stored":235281646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369356495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,5,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247902,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740712,"kb_used_data":231512,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93631128,"statfs":{"total":96636764160,"available":95878275072,"internally_reserved":0,"allocated":237068288,"data_stored":236879086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":521400014},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":236388352,"data_stored":236388352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":235139072,"data_stored":235139072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":232452096,"data_stored":232452096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":236204032,"data_stored":236204032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:10.501 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:11.824 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:11.824 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:11.854 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2024-09-16T04:48:10.700206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":463873728,"num_objects":56631,"num_object_clones":0,"num_object_copies":226524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56631,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144856,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5056,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144883,"ondisk_log_size":144883,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788276,"kb_used_data":908652,"kb_used_omap":14,"kb_used_meta":1879537,"kb_avail":374699084,"statfs":{"total":386547056640,"available":383691862016,"internally_reserved":0,"allocated":930459648,"data_stored":929702840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1924646449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,8,6,8,12,7,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18063360,"num_objects":-2205,"num_object_clones":0,"num_object_copies":-8820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2205,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006671"},"pg_stats":[{"pgid":"1.f","version":"35'9090","reported_seq":9206,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:07.069302+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:07.069302+0000","last_peered":"2024-09-16T04:48:07.069302+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:07.069302+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9090,"log_dups_size":0,"ondisk_log_size":9090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29196288,"num_objects":3564,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9090,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1378}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9076","reported_seq":9164,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.669867+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:06.669867+0000","last_peered":"2024-09-16T04:48:06.669867+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:06.669867+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9076,"log_dups_size":0,"ondisk_log_size":9076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29472012,"num_objects":3598,"num_object_clones":0,"num_object_copies":14392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9074,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9113","reported_seq":9211,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.277817+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:06.277817+0000","last_peered":"2024-09-16T04:48:06.277817+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:06.277817+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9113,"log_dups_size":0,"ondisk_log_size":9113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29137566,"num_objects":3558,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9112,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9115","reported_seq":9570,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.946991+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:06.946991+0000","last_peered":"2024-09-16T04:48:06.946991+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:06.946991+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9115,"log_dups_size":0,"ondisk_log_size":9115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29119756,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9113,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1515}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9181","reported_seq":9312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:07.124991+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:07.124991+0000","last_peered":"2024-09-16T04:48:07.124991+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:07.124991+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9181,"log_dups_size":0,"ondisk_log_size":9181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29160716,"num_objects":3560,"num_object_clones":0,"num_object_copies":14240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9179,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1312}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8871","reported_seq":8980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:07.192714+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:07.192714+0000","last_peered":"2024-09-16T04:48:07.192714+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:07.192714+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8871,"log_dups_size":0,"ondisk_log_size":8871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28188672,"num_objects":3441,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8871,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9099","reported_seq":9194,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.816269+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:06.816269+0000","last_peered":"2024-09-16T04:48:06.816269+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:06.816269+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28677388,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9097,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1288}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9228","reported_seq":13789,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:08.818479+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:08.818479+0000","last_peered":"2024-09-16T04:48:08.818479+0000","last_clean":"2024-09-16T04:48:08.818479+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:08.818479+0000","last_undegraded":"2024-09-16T04:48:08.818479+0000","last_fullsized":"2024-09-16T04:48:08.818479+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9228,"log_dups_size":0,"ondisk_log_size":9228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:12.036764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29136140,"num_objects":3557,"num_object_clones":0,"num_object_copies":14228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9263","reported_seq":14020,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:07.086917+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:07.086917+0000","last_peered":"2024-09-16T04:48:07.086917+0000","last_clean":"2024-09-16T04:48:07.086917+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:07.086917+0000","last_undegraded":"2024-09-16T04:48:07.086917+0000","last_fullsized":"2024-09-16T04:48:07.086917+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9263,"log_dups_size":0,"ondisk_log_size":9263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:09.848806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28609048,"num_objects":3493,"num_object_clones":0,"num_object_copies":13972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9260,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9118","reported_seq":9224,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.318537+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:06.318537+0000","last_peered":"2024-09-16T04:48:06.318537+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:06.318537+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9118,"log_dups_size":0,"ondisk_log_size":9118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29816076,"num_objects":3640,"num_object_clones":0,"num_object_copies":14560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9116,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1451}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8991","reported_seq":9120,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:07.247197+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:07.247197+0000","last_peered":"2024-09-16T04:48:07.247197+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:07.247197+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8991,"log_dups_size":0,"ondisk_log_size":8991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28667794,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8988,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1345}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8830","reported_seq":8922,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:07.183882+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:07.183882+0000","last_peered":"2024-09-16T04:48:07.183882+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:07.183882+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8830,"log_dups_size":0,"ondisk_log_size":8830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28678790,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8829,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1396}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9002","reported_seq":9090,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.124563+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:06.124563+0000","last_peered":"2024-09-16T04:48:06.124563+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:06.124563+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9002,"log_dups_size":0,"ondisk_log_size":9002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28924550,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9001,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1310}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9140","reported_seq":9268,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:07.190142+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:07.190142+0000","last_peered":"2024-09-16T04:48:07.190142+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:07.190142+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9140,"log_dups_size":0,"ondisk_log_size":9140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29832460,"num_objects":3642,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9138,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1444}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8859","reported_seq":8962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.402237+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:06.402237+0000","last_peered":"2024-09-16T04:48:06.402237+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:06.402237+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8859,"log_dups_size":0,"ondisk_log_size":8859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29406476,"num_objects":3590,"num_object_clones":0,"num_object_copies":14360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8857,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1467}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8907","reported_seq":9001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.682999+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:06.682999+0000","last_peered":"2024-09-16T04:48:06.682999+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:06.682999+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8907,"log_dups_size":0,"ondisk_log_size":8907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27849996,"num_objects":3400,"num_object_clones":0,"num_object_copies":13600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8905,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1172}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":463873728,"num_objects":56631,"num_object_clones":0,"num_object_copies":226524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56631,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144856,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5056,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":927080448,"data_stored":927080448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144883,"ondisk_log_size":144883,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740392,"kb_used_data":226968,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93631448,"statfs":{"total":96636764160,"available":95878602752,"internally_reserved":0,"allocated":232415232,"data_stored":232226030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":525725391},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732472,"kb_used_data":227880,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93639368,"statfs":{"total":96636764160,"available":95886712832,"internally_reserved":0,"allocated":233349120,"data_stored":233159918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051665,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591436,"kb_used_data":226556,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93780404,"statfs":{"total":96636764160,"available":96031133696,"internally_reserved":0,"allocated":231993344,"data_stored":231804142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373616335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723976,"kb_used_data":227248,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93647864,"statfs":{"total":96636764160,"available":95895412736,"internally_reserved":0,"allocated":232701952,"data_stored":232512750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":508620494},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,0,0,6,2,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":232034304,"data_stored":232034304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":230805504,"data_stored":230805504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":232452096,"data_stored":232452096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":231788544,"data_stored":231788544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:11.856 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:12.495 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:12.496 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:12.523 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2024-09-16T04:48:10.700206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":463873728,"num_objects":56631,"num_object_clones":0,"num_object_copies":226524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56631,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144856,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5056,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144883,"ondisk_log_size":144883,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788276,"kb_used_data":908652,"kb_used_omap":14,"kb_used_meta":1879537,"kb_avail":374699084,"statfs":{"total":386547056640,"available":383691862016,"internally_reserved":0,"allocated":930459648,"data_stored":929702840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1924646449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,8,6,8,12,7,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18063360,"num_objects":-2205,"num_object_clones":0,"num_object_copies":-8820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2205,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006671"},"pg_stats":[{"pgid":"1.f","version":"35'9090","reported_seq":9206,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:07.069302+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:07.069302+0000","last_peered":"2024-09-16T04:48:07.069302+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:07.069302+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9090,"log_dups_size":0,"ondisk_log_size":9090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29196288,"num_objects":3564,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9090,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1378}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9076","reported_seq":9164,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.669867+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:06.669867+0000","last_peered":"2024-09-16T04:48:06.669867+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:06.669867+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9076,"log_dups_size":0,"ondisk_log_size":9076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29472012,"num_objects":3598,"num_object_clones":0,"num_object_copies":14392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9074,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9113","reported_seq":9211,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.277817+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:06.277817+0000","last_peered":"2024-09-16T04:48:06.277817+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:06.277817+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9113,"log_dups_size":0,"ondisk_log_size":9113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29137566,"num_objects":3558,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9112,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9115","reported_seq":9570,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.946991+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:06.946991+0000","last_peered":"2024-09-16T04:48:06.946991+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:06.946991+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9115,"log_dups_size":0,"ondisk_log_size":9115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29119756,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9113,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1515}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9181","reported_seq":9312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:07.124991+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:07.124991+0000","last_peered":"2024-09-16T04:48:07.124991+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:07.124991+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9181,"log_dups_size":0,"ondisk_log_size":9181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29160716,"num_objects":3560,"num_object_clones":0,"num_object_copies":14240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9179,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1312}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8871","reported_seq":8980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:07.192714+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:07.192714+0000","last_peered":"2024-09-16T04:48:07.192714+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:07.192714+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8871,"log_dups_size":0,"ondisk_log_size":8871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28188672,"num_objects":3441,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8871,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9099","reported_seq":9194,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.816269+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:06.816269+0000","last_peered":"2024-09-16T04:48:06.816269+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:06.816269+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28677388,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9097,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1288}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9228","reported_seq":13789,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:08.818479+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:08.818479+0000","last_peered":"2024-09-16T04:48:08.818479+0000","last_clean":"2024-09-16T04:48:08.818479+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:08.818479+0000","last_undegraded":"2024-09-16T04:48:08.818479+0000","last_fullsized":"2024-09-16T04:48:08.818479+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9228,"log_dups_size":0,"ondisk_log_size":9228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:12.036764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29136140,"num_objects":3557,"num_object_clones":0,"num_object_copies":14228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9263","reported_seq":14020,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:07.086917+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:07.086917+0000","last_peered":"2024-09-16T04:48:07.086917+0000","last_clean":"2024-09-16T04:48:07.086917+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:07.086917+0000","last_undegraded":"2024-09-16T04:48:07.086917+0000","last_fullsized":"2024-09-16T04:48:07.086917+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9263,"log_dups_size":0,"ondisk_log_size":9263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:09.848806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28609048,"num_objects":3493,"num_object_clones":0,"num_object_copies":13972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9260,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9118","reported_seq":9224,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.318537+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:06.318537+0000","last_peered":"2024-09-16T04:48:06.318537+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:06.318537+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9118,"log_dups_size":0,"ondisk_log_size":9118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29816076,"num_objects":3640,"num_object_clones":0,"num_object_copies":14560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9116,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1451}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8991","reported_seq":9120,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:07.247197+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:07.247197+0000","last_peered":"2024-09-16T04:48:07.247197+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:07.247197+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8991,"log_dups_size":0,"ondisk_log_size":8991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28667794,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8988,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1345}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8830","reported_seq":8922,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:07.183882+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:07.183882+0000","last_peered":"2024-09-16T04:48:07.183882+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:07.183882+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8830,"log_dups_size":0,"ondisk_log_size":8830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28678790,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8829,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1396}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9002","reported_seq":9090,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.124563+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:06.124563+0000","last_peered":"2024-09-16T04:48:06.124563+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:06.124563+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9002,"log_dups_size":0,"ondisk_log_size":9002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28924550,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9001,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1310}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9140","reported_seq":9268,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:07.190142+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:07.190142+0000","last_peered":"2024-09-16T04:48:07.190142+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:07.190142+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9140,"log_dups_size":0,"ondisk_log_size":9140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29832460,"num_objects":3642,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9138,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1444}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8859","reported_seq":8962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.402237+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:06.402237+0000","last_peered":"2024-09-16T04:48:06.402237+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:06.402237+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8859,"log_dups_size":0,"ondisk_log_size":8859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29406476,"num_objects":3590,"num_object_clones":0,"num_object_copies":14360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8857,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1467}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8907","reported_seq":9001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:06.682999+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:06.682999+0000","last_peered":"2024-09-16T04:48:06.682999+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:06.682999+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8907,"log_dups_size":0,"ondisk_log_size":8907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27849996,"num_objects":3400,"num_object_clones":0,"num_object_copies":13600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8905,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1172}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":463873728,"num_objects":56631,"num_object_clones":0,"num_object_copies":226524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56631,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144856,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5056,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":927080448,"data_stored":927080448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144883,"ondisk_log_size":144883,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740392,"kb_used_data":226968,"kb_used_omap":4,"kb_used_meta":513403,"kb_avail":93631448,"statfs":{"total":96636764160,"available":95878602752,"internally_reserved":0,"allocated":232415232,"data_stored":232226030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":525725391},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732472,"kb_used_data":227880,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93639368,"statfs":{"total":96636764160,"available":95886712832,"internally_reserved":0,"allocated":233349120,"data_stored":233159918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051665,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591436,"kb_used_data":226556,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93780404,"statfs":{"total":96636764160,"available":96031133696,"internally_reserved":0,"allocated":231993344,"data_stored":231804142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373616335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723976,"kb_used_data":227248,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93647864,"statfs":{"total":96636764160,"available":95895412736,"internally_reserved":0,"allocated":232701952,"data_stored":232512750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":508620494},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,0,0,6,2,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":232034304,"data_stored":232034304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":230805504,"data_stored":230805504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":232452096,"data_stored":232452096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":231788544,"data_stored":231788544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:12.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:48:12.524 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9090", 'reported_seq': 9206, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:07.069302+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:48:07.069302+0000', 'last_peered': '2024-09-16T04:48:07.069302+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:48:07.069302+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9090, 'log_dups_size': 0, 'ondisk_log_size': 9090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29196288, 'num_objects': 3564, 'num_object_clones': 0, 'num_object_copies': 14256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9090, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1378}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:12.524 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:48:12.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9076", 'reported_seq': 9164, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:06.669867+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:48:06.669867+0000', 'last_peered': '2024-09-16T04:48:06.669867+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:48:06.669867+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9076, 'log_dups_size': 0, 'ondisk_log_size': 9076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29472012, 'num_objects': 3598, 'num_object_clones': 0, 'num_object_copies': 14392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9074, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:12.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:48:12.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9113", 'reported_seq': 9211, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:06.277817+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:48:06.277817+0000', 'last_peered': '2024-09-16T04:48:06.277817+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:48:06.277817+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9113, 'log_dups_size': 0, 'ondisk_log_size': 9113, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29137566, 'num_objects': 3558, 'num_object_clones': 0, 'num_object_copies': 14232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3558, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9112, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1339}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:12.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:48:12.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9115", 'reported_seq': 9570, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:06.946991+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:48:06.946991+0000', 'last_peered': '2024-09-16T04:48:06.946991+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:48:06.946991+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9115, 'log_dups_size': 0, 'ondisk_log_size': 9115, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29119756, 'num_objects': 3555, 'num_object_clones': 0, 'num_object_copies': 14220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9113, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 193, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1515}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:12.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:48:12.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9181", 'reported_seq': 9312, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:07.124991+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:48:07.124991+0000', 'last_peered': '2024-09-16T04:48:07.124991+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:48:07.124991+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9181, 'log_dups_size': 0, 'ondisk_log_size': 9181, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29160716, 'num_objects': 3560, 'num_object_clones': 0, 'num_object_copies': 14240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3560, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9179, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1312}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:12.525 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:48:12.525 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8871", 'reported_seq': 8980, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:07.192714+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:48:07.192714+0000', 'last_peered': '2024-09-16T04:48:07.192714+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:48:07.192714+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8871, 'log_dups_size': 0, 'ondisk_log_size': 8871, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28188672, 'num_objects': 3441, 'num_object_clones': 0, 'num_object_copies': 13764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8871, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1263}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:12.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:48:12.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9099", 'reported_seq': 9194, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:06.816269+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:48:06.816269+0000', 'last_peered': '2024-09-16T04:48:06.816269+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:48:06.816269+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9099, 'log_dups_size': 0, 'ondisk_log_size': 9099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28677388, 'num_objects': 3501, 'num_object_clones': 0, 'num_object_copies': 14004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9097, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1288}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:12.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:48:12.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9118", 'reported_seq': 9224, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:06.318537+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:48:06.318537+0000', 'last_peered': '2024-09-16T04:48:06.318537+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:48:06.318537+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9118, 'log_dups_size': 0, 'ondisk_log_size': 9118, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29816076, 'num_objects': 3640, 'num_object_clones': 0, 'num_object_copies': 14560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3640, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9116, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1451}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:12.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:48:12.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8991", 'reported_seq': 9120, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:07.247197+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:48:07.247197+0000', 'last_peered': '2024-09-16T04:48:07.247197+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:48:07.247197+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8991, 'log_dups_size': 0, 'ondisk_log_size': 8991, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28667794, 'num_objects': 3500, 'num_object_clones': 0, 'num_object_copies': 14000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3500, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8988, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1345}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:12.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:48:12.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8830", 'reported_seq': 8922, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:07.183882+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:48:07.183882+0000', 'last_peered': '2024-09-16T04:48:07.183882+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:48:07.183882+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8830, 'log_dups_size': 0, 'ondisk_log_size': 8830, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28678790, 'num_objects': 3501, 'num_object_clones': 0, 'num_object_copies': 14004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8829, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1396}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:12.526 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:48:12.526 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9002", 'reported_seq': 9090, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:06.124563+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:48:06.124563+0000', 'last_peered': '2024-09-16T04:48:06.124563+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:48:06.124563+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9002, 'log_dups_size': 0, 'ondisk_log_size': 9002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28924550, 'num_objects': 3531, 'num_object_clones': 0, 'num_object_copies': 14124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9001, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1310}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:12.527 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:48:12.527 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9140", 'reported_seq': 9268, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:07.190142+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:48:07.190142+0000', 'last_peered': '2024-09-16T04:48:07.190142+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:48:07.190142+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9140, 'log_dups_size': 0, 'ondisk_log_size': 9140, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29832460, 'num_objects': 3642, 'num_object_clones': 0, 'num_object_copies': 14568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9138, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1444}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:12.527 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:48:12.527 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'8859", 'reported_seq': 8962, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:06.402237+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:48:06.402237+0000', 'last_peered': '2024-09-16T04:48:06.402237+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:48:06.402237+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8859, 'log_dups_size': 0, 'ondisk_log_size': 8859, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29406476, 'num_objects': 3590, 'num_object_clones': 0, 'num_object_copies': 14360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8857, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1467}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:12.527 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:48:12.527 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8907", 'reported_seq': 9001, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:06.682999+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:48:06.682999+0000', 'last_peered': '2024-09-16T04:48:06.682999+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:48:06.682999+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8907, 'log_dups_size': 0, 'ondisk_log_size': 8907, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27849996, 'num_objects': 3400, 'num_object_clones': 0, 'num_object_copies': 13600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3400, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8905, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1172}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:12.528 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:48:13.272 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:13.272 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30479/222424 objects degraded (13.703%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":505,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":55606,"data_bytes":455476928,"bytes_used":2837864448,"bytes_avail":383709192192,"bytes_total":386547056640,"degraded_objects":30479,"degraded_total":222424,"degraded_ratio":0.13703107578318888,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":262},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T04:47:26.684344+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T04:48:13.290 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:48:13.290 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:13.927 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:13.927 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:13.968 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-09-16T04:48:12.700687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":455476928,"num_objects":55606,"num_object_clones":0,"num_object_copies":222424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55606,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145881,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5103,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145908,"ondisk_log_size":145908,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771352,"kb_used_data":895816,"kb_used_omap":14,"kb_used_meta":1875377,"kb_avail":374716008,"statfs":{"total":386547056640,"available":383709192192,"internally_reserved":0,"allocated":917315584,"data_stored":916558776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1920386609},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,0,6,11,9,6,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25772032,"num_objects":-3146,"num_object_clones":0,"num_object_copies":-12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3146,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006651"},"pg_stats":[{"pgid":"1.f","version":"35'9143","reported_seq":9259,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.286068+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:12.286068+0000","last_peered":"2024-09-16T04:48:12.286068+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:12.286068+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9143,"log_dups_size":0,"ondisk_log_size":9143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28762112,"num_objects":3511,"num_object_clones":0,"num_object_copies":14044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9143,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9151","reported_seq":9239,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.150114+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:12.150114+0000","last_peered":"2024-09-16T04:48:12.150114+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:12.150114+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9151,"log_dups_size":0,"ondisk_log_size":9151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28857612,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9149,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9176","reported_seq":9274,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:11.283217+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:11.283217+0000","last_peered":"2024-09-16T04:48:11.283217+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:11.283217+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9176,"log_dups_size":0,"ondisk_log_size":9176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28621470,"num_objects":3495,"num_object_clones":0,"num_object_copies":13980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3495,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9175,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1276}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9192","reported_seq":9741,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.134603+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:12.134603+0000","last_peered":"2024-09-16T04:48:12.134603+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:12.134603+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9192,"log_dups_size":0,"ondisk_log_size":9192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28488972,"num_objects":3478,"num_object_clones":0,"num_object_copies":13912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9190,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9236","reported_seq":9368,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.194063+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:12.194063+0000","last_peered":"2024-09-16T04:48:12.194063+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:12.194063+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9236,"log_dups_size":0,"ondisk_log_size":9236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28710156,"num_objects":3505,"num_object_clones":0,"num_object_copies":14020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8940","reported_seq":9047,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.324126+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:12.324126+0000","last_peered":"2024-09-16T04:48:12.324126+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:12.324126+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8940,"log_dups_size":0,"ondisk_log_size":8940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27623424,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8940,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9158","reported_seq":9254,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.070461+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:12.070461+0000","last_peered":"2024-09-16T04:48:12.070461+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:12.070461+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9158,"log_dups_size":0,"ondisk_log_size":9158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28194060,"num_objects":3442,"num_object_clones":0,"num_object_copies":13768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9156,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1229}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9228","reported_seq":13789,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:08.818479+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:08.818479+0000","last_peered":"2024-09-16T04:48:08.818479+0000","last_clean":"2024-09-16T04:48:08.818479+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:08.818479+0000","last_undegraded":"2024-09-16T04:48:08.818479+0000","last_fullsized":"2024-09-16T04:48:08.818479+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9228,"log_dups_size":0,"ondisk_log_size":9228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:12.036764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29136140,"num_objects":3557,"num_object_clones":0,"num_object_copies":14228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9337","reported_seq":14093,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:12.142322+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:12.142322+0000","last_peered":"2024-09-16T04:48:12.142322+0000","last_clean":"2024-09-16T04:48:12.142322+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:12.142322+0000","last_undegraded":"2024-09-16T04:48:12.142322+0000","last_fullsized":"2024-09-16T04:48:12.142322+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9337,"log_dups_size":0,"ondisk_log_size":9337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:15.832790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28002840,"num_objects":3419,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9334,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9186","reported_seq":9292,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:11.434260+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:11.434260+0000","last_peered":"2024-09-16T04:48:11.434260+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:11.434260+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9186,"log_dups_size":0,"ondisk_log_size":9186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29259020,"num_objects":3572,"num_object_clones":0,"num_object_copies":14288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9184,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1383}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9073","reported_seq":9204,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.299014+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:12.299014+0000","last_peered":"2024-09-16T04:48:12.299014+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:12.299014+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9073,"log_dups_size":0,"ondisk_log_size":9073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27996050,"num_objects":3418,"num_object_clones":0,"num_object_copies":13672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9070,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8887","reported_seq":8979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.125928+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:12.125928+0000","last_peered":"2024-09-16T04:48:12.125928+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:12.125928+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28211846,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8886,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1339}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9076","reported_seq":9164,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:11.410243+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:11.410243+0000","last_peered":"2024-09-16T04:48:11.410243+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:11.410243+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9076,"log_dups_size":0,"ondisk_log_size":9076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28318342,"num_objects":3457,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9075,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1236}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9217","reported_seq":9349,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.255020+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:12.255020+0000","last_peered":"2024-09-16T04:48:12.255020+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:12.255020+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9217,"log_dups_size":0,"ondisk_log_size":9217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29201676,"num_objects":3565,"num_object_clones":0,"num_object_copies":14260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9215,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1367}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8926","reported_seq":9029,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:11.347115+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:11.347115+0000","last_peered":"2024-09-16T04:48:11.347115+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:11.347115+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8926,"log_dups_size":0,"ondisk_log_size":8926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28857612,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8924,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1400}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8982","reported_seq":9076,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:11.867359+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:11.867359+0000","last_peered":"2024-09-16T04:48:11.867359+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:11.867359+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8982,"log_dups_size":0,"ondisk_log_size":8982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27235596,"num_objects":3325,"num_object_clones":0,"num_object_copies":13300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1097}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":455476928,"num_objects":55606,"num_object_clones":0,"num_object_copies":222424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55606,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145881,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5103,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":913719296,"data_stored":913719296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145908,"ondisk_log_size":145908,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723684,"kb_used_data":222696,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93648156,"statfs":{"total":96636764160,"available":95895711744,"internally_reserved":0,"allocated":228040704,"data_stored":227851502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512945871},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,1,1,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732364,"kb_used_data":223596,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93639476,"statfs":{"total":96636764160,"available":95886823424,"internally_reserved":0,"allocated":228962304,"data_stored":228773102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,4,0,4,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051665,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591436,"kb_used_data":226556,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93780404,"statfs":{"total":96636764160,"available":96031133696,"internally_reserved":0,"allocated":231993344,"data_stored":231804142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373616335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723868,"kb_used_data":222968,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93647972,"statfs":{"total":96636764160,"available":95895523328,"internally_reserved":0,"allocated":228319232,"data_stored":228130030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":512880334},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":227495936,"data_stored":227495936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":230805504,"data_stored":230805504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":228024320,"data_stored":228024320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":227393536,"data_stored":227393536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:16.970 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:18.088 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:18.090 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:18.110 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-09-16T04:48:16.701680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":453043904,"num_objects":55309,"num_object_clones":0,"num_object_copies":221236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146178,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5103,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146205,"ondisk_log_size":146205,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2790208,"kb_used_data":886912,"kb_used_omap":14,"kb_used_meta":1903153,"kb_avail":374697152,"statfs":{"total":386547056640,"available":383689883648,"internally_reserved":0,"allocated":908197888,"data_stored":907441080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1948829233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,11,12,4,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19668992,"num_objects":-2401,"num_object_clones":0,"num_object_copies":-9604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2401,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006659"},"pg_stats":[{"pgid":"1.f","version":"35'9143","reported_seq":9259,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.286068+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:12.286068+0000","last_peered":"2024-09-16T04:48:12.286068+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:12.286068+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9143,"log_dups_size":0,"ondisk_log_size":9143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28762112,"num_objects":3511,"num_object_clones":0,"num_object_copies":14044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9143,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9151","reported_seq":9239,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.150114+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:12.150114+0000","last_peered":"2024-09-16T04:48:12.150114+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:12.150114+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9151,"log_dups_size":0,"ondisk_log_size":9151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28857612,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9149,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9227","reported_seq":9325,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:16.498885+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:16.498885+0000","last_peered":"2024-09-16T04:48:16.498885+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:16.498885+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9227,"log_dups_size":0,"ondisk_log_size":9227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28203678,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9226,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1225}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9192","reported_seq":9741,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.134603+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:12.134603+0000","last_peered":"2024-09-16T04:48:12.134603+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:12.134603+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9192,"log_dups_size":0,"ondisk_log_size":9192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28488972,"num_objects":3478,"num_object_clones":0,"num_object_copies":13912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9190,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9236","reported_seq":9368,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.194063+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:12.194063+0000","last_peered":"2024-09-16T04:48:12.194063+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:12.194063+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9236,"log_dups_size":0,"ondisk_log_size":9236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28710156,"num_objects":3505,"num_object_clones":0,"num_object_copies":14020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8940","reported_seq":9047,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.324126+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:12.324126+0000","last_peered":"2024-09-16T04:48:12.324126+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:12.324126+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8940,"log_dups_size":0,"ondisk_log_size":8940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27623424,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8940,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9158","reported_seq":9254,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.070461+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:12.070461+0000","last_peered":"2024-09-16T04:48:12.070461+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:12.070461+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9158,"log_dups_size":0,"ondisk_log_size":9158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28194060,"num_objects":3442,"num_object_clones":0,"num_object_copies":13768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9156,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1229}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9278","reported_seq":13839,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:13.785783+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:13.785783+0000","last_peered":"2024-09-16T04:48:13.785783+0000","last_clean":"2024-09-16T04:48:13.785783+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:13.785783+0000","last_undegraded":"2024-09-16T04:48:13.785783+0000","last_fullsized":"2024-09-16T04:48:13.785783+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:17.067594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28726540,"num_objects":3507,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9337","reported_seq":14093,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:12.142322+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:12.142322+0000","last_peered":"2024-09-16T04:48:12.142322+0000","last_clean":"2024-09-16T04:48:12.142322+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:12.142322+0000","last_undegraded":"2024-09-16T04:48:12.142322+0000","last_fullsized":"2024-09-16T04:48:12.142322+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9337,"log_dups_size":0,"ondisk_log_size":9337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:15.832790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28002840,"num_objects":3419,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9334,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9253","reported_seq":9359,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:16.151185+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:16.151185+0000","last_peered":"2024-09-16T04:48:16.151185+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:16.151185+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9253,"log_dups_size":0,"ondisk_log_size":9253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28710156,"num_objects":3505,"num_object_clones":0,"num_object_copies":14020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9251,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1316}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9073","reported_seq":9204,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.299014+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:12.299014+0000","last_peered":"2024-09-16T04:48:12.299014+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:12.299014+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9073,"log_dups_size":0,"ondisk_log_size":9073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27996050,"num_objects":3418,"num_object_clones":0,"num_object_copies":13672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9070,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8887","reported_seq":8979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.125928+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:12.125928+0000","last_peered":"2024-09-16T04:48:12.125928+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:12.125928+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28211846,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8886,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1339}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9138","reported_seq":9224,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:16.078874+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:16.078874+0000","last_peered":"2024-09-16T04:48:16.078874+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:16.078874+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9138,"log_dups_size":0,"ondisk_log_size":9138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27810438,"num_objects":3395,"num_object_clones":0,"num_object_copies":13580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9137,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1174}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9217","reported_seq":9349,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:12.255020+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:12.255020+0000","last_peered":"2024-09-16T04:48:12.255020+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:12.255020+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9217,"log_dups_size":0,"ondisk_log_size":9217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29201676,"num_objects":3565,"num_object_clones":0,"num_object_copies":14260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9215,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1367}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8993","reported_seq":9095,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:16.544077+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:16.544077+0000","last_peered":"2024-09-16T04:48:16.544077+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:16.544077+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8993,"log_dups_size":0,"ondisk_log_size":8993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28308748,"num_objects":3456,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8991,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1333}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8982","reported_seq":9076,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:11.867359+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:11.867359+0000","last_peered":"2024-09-16T04:48:11.867359+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:11.867359+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8982,"log_dups_size":0,"ondisk_log_size":8982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27235596,"num_objects":3325,"num_object_clones":0,"num_object_copies":13300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1097}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":453043904,"num_objects":55309,"num_object_clones":0,"num_object_copies":221236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146178,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5103,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":905355264,"data_stored":905355264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146205,"ondisk_log_size":146205,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723684,"kb_used_data":222696,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93648156,"statfs":{"total":96636764160,"available":95895711744,"internally_reserved":0,"allocated":228040704,"data_stored":227851502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512945871},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,1,1,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732524,"kb_used_data":219612,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":93639316,"statfs":{"total":96636764160,"available":95886659584,"internally_reserved":0,"allocated":224882688,"data_stored":224693486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":525203909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051666,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610132,"kb_used_data":221636,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93761708,"statfs":{"total":96636764160,"available":96011988992,"internally_reserved":0,"allocated":226955264,"data_stored":226766062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397799119},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723868,"kb_used_data":222968,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93647972,"statfs":{"total":96636764160,"available":95895523328,"internally_reserved":0,"allocated":228319232,"data_stored":228130030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":512880334},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":227495936,"data_stored":227495936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":226574336,"data_stored":226574336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":223891456,"data_stored":223891456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":227393536,"data_stored":227393536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:18.111 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:18.708 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:18.708 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:18.732 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-09-16T04:48:18.702509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":447227584,"num_objects":54599,"num_object_clones":0,"num_object_copies":218396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146888,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5158,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146915,"ondisk_log_size":146915,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2790464,"kb_used_data":878896,"kb_used_omap":14,"kb_used_meta":1911473,"kb_avail":374696896,"statfs":{"total":386547056640,"available":383689621504,"internally_reserved":0,"allocated":899989504,"data_stored":899232696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1957348913},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,9,8,16,12,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23158784,"num_objects":-2827,"num_object_clones":0,"num_object_copies":-11308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2827,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007019"},"pg_stats":[{"pgid":"1.f","version":"35'9209","reported_seq":9325,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.291210+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:17.291210+0000","last_peered":"2024-09-16T04:48:17.291210+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:17.291210+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9209,"log_dups_size":0,"ondisk_log_size":9209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28221440,"num_objects":3445,"num_object_clones":0,"num_object_copies":13780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9209,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9214","reported_seq":9302,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:16.945045+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:16.945045+0000","last_peered":"2024-09-16T04:48:16.945045+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:16.945045+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9214,"log_dups_size":0,"ondisk_log_size":9214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28341516,"num_objects":3460,"num_object_clones":0,"num_object_copies":13840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9212,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9227","reported_seq":9325,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:16.498885+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:16.498885+0000","last_peered":"2024-09-16T04:48:16.498885+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:16.498885+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9227,"log_dups_size":0,"ondisk_log_size":9227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28203678,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9226,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1225}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9254","reported_seq":9912,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.175437+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:17.175437+0000","last_peered":"2024-09-16T04:48:17.175437+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:17.175437+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9254,"log_dups_size":0,"ondisk_log_size":9254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27981068,"num_objects":3416,"num_object_clones":0,"num_object_copies":13664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9252,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1478}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9301","reported_seq":9433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.230977+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:17.230977+0000","last_peered":"2024-09-16T04:48:17.230977+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:17.230977+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9301,"log_dups_size":0,"ondisk_log_size":9301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28177676,"num_objects":3440,"num_object_clones":0,"num_object_copies":13760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9299,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9008","reported_seq":9118,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.235854+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:17.235854+0000","last_peered":"2024-09-16T04:48:17.235854+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:17.235854+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9008,"log_dups_size":0,"ondisk_log_size":9008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27066368,"num_objects":3304,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9008,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1126}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9222","reported_seq":9319,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.046946+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:17.046946+0000","last_peered":"2024-09-16T04:48:17.046946+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:17.046946+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27669772,"num_objects":3378,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9220,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1165}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9278","reported_seq":13839,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:13.785783+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:13.785783+0000","last_peered":"2024-09-16T04:48:13.785783+0000","last_clean":"2024-09-16T04:48:13.785783+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:13.785783+0000","last_undegraded":"2024-09-16T04:48:13.785783+0000","last_fullsized":"2024-09-16T04:48:13.785783+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:17.067594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28726540,"num_objects":3507,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9385","reported_seq":14142,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:17.109349+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:17.109349+0000","last_peered":"2024-09-16T04:48:17.109349+0000","last_clean":"2024-09-16T04:48:17.109349+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:17.109349+0000","last_undegraded":"2024-09-16T04:48:17.109349+0000","last_fullsized":"2024-09-16T04:48:17.109349+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9385,"log_dups_size":0,"ondisk_log_size":9385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:20.862882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27609624,"num_objects":3371,"num_object_clones":0,"num_object_copies":13484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9382,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9253","reported_seq":9359,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:16.151185+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:16.151185+0000","last_peered":"2024-09-16T04:48:16.151185+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:16.151185+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9253,"log_dups_size":0,"ondisk_log_size":9253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28710156,"num_objects":3505,"num_object_clones":0,"num_object_copies":14020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9251,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1316}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9145","reported_seq":9276,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.286484+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:17.286484+0000","last_peered":"2024-09-16T04:48:17.286484+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:17.286484+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9145,"log_dups_size":0,"ondisk_log_size":9145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27406226,"num_objects":3346,"num_object_clones":0,"num_object_copies":13384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9142,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8959","reported_seq":9051,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.289708+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:17.289708+0000","last_peered":"2024-09-16T04:48:17.289708+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:17.289708+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8959,"log_dups_size":0,"ondisk_log_size":8959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27622022,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8958,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1267}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9138","reported_seq":9224,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:16.078874+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:16.078874+0000","last_peered":"2024-09-16T04:48:16.078874+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:16.078874+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9138,"log_dups_size":0,"ondisk_log_size":9138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27810438,"num_objects":3395,"num_object_clones":0,"num_object_copies":13580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9137,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1174}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9288","reported_seq":9420,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.278672+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:17.278672+0000","last_peered":"2024-09-16T04:48:17.278672+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:17.278672+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9288,"log_dups_size":0,"ondisk_log_size":9288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28620044,"num_objects":3494,"num_object_clones":0,"num_object_copies":13976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9286,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1296}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8993","reported_seq":9095,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:16.544077+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:16.544077+0000","last_peered":"2024-09-16T04:48:16.544077+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:16.544077+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8993,"log_dups_size":0,"ondisk_log_size":8993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28308748,"num_objects":3456,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8991,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1333}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9041","reported_seq":9133,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.145245+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:17.145245+0000","last_peered":"2024-09-16T04:48:17.145245+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:17.145245+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9041,"log_dups_size":0,"ondisk_log_size":9041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26752268,"num_objects":3266,"num_object_clones":0,"num_object_copies":13064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9039,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":447227584,"num_objects":54599,"num_object_clones":0,"num_object_copies":218396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146888,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5158,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":896978944,"data_stored":896978944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146915,"ondisk_log_size":146915,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723748,"kb_used_data":218644,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93648092,"statfs":{"total":96636764160,"available":95895646208,"internally_reserved":0,"allocated":223891456,"data_stored":223702254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517205711},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732524,"kb_used_data":219612,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":93639316,"statfs":{"total":96636764160,"available":95886659584,"internally_reserved":0,"allocated":224882688,"data_stored":224693486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":525203909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051666,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610132,"kb_used_data":221636,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93761708,"statfs":{"total":96636764160,"available":96011988992,"internally_reserved":0,"allocated":226955264,"data_stored":226766062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397799119},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724060,"kb_used_data":219004,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93647780,"statfs":{"total":96636764160,"available":95895326720,"internally_reserved":0,"allocated":224260096,"data_stored":224070894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,1,4,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":223330304,"data_stored":223330304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":226574336,"data_stored":226574336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":223891456,"data_stored":223891456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":223182848,"data_stored":223182848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:18.733 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:48:18.733 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9209", 'reported_seq': 9325, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:17.291210+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:48:17.291210+0000', 'last_peered': '2024-09-16T04:48:17.291210+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:48:17.291210+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9209, 'log_dups_size': 0, 'ondisk_log_size': 9209, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28221440, 'num_objects': 3445, 'num_object_clones': 0, 'num_object_copies': 13780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3445, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9209, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1259}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:18.734 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:48:18.734 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9214", 'reported_seq': 9302, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:16.945045+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:48:16.945045+0000', 'last_peered': '2024-09-16T04:48:16.945045+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:48:16.945045+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9214, 'log_dups_size': 0, 'ondisk_log_size': 9214, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28341516, 'num_objects': 3460, 'num_object_clones': 0, 'num_object_copies': 13840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9212, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1257}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:18.734 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:48:18.734 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9227", 'reported_seq': 9325, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:16.498885+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:48:16.498885+0000', 'last_peered': '2024-09-16T04:48:16.498885+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:48:16.498885+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9227, 'log_dups_size': 0, 'ondisk_log_size': 9227, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28203678, 'num_objects': 3444, 'num_object_clones': 0, 'num_object_copies': 13776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3444, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9226, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1225}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:18.734 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:48:18.734 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9254", 'reported_seq': 9912, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:17.175437+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:48:17.175437+0000', 'last_peered': '2024-09-16T04:48:17.175437+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:48:17.175437+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9254, 'log_dups_size': 0, 'ondisk_log_size': 9254, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27981068, 'num_objects': 3416, 'num_object_clones': 0, 'num_object_copies': 13664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9252, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 295, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1478}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:18.734 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:48:18.734 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9301", 'reported_seq': 9433, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:17.230977+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:48:17.230977+0000', 'last_peered': '2024-09-16T04:48:17.230977+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:48:17.230977+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9301, 'log_dups_size': 0, 'ondisk_log_size': 9301, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28177676, 'num_objects': 3440, 'num_object_clones': 0, 'num_object_copies': 13760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9299, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1192}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:18.734 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:48:18.735 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9008", 'reported_seq': 9118, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:17.235854+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:48:17.235854+0000', 'last_peered': '2024-09-16T04:48:17.235854+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:48:17.235854+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9008, 'log_dups_size': 0, 'ondisk_log_size': 9008, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27066368, 'num_objects': 3304, 'num_object_clones': 0, 'num_object_copies': 13216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3304, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9008, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1126}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:18.735 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:48:18.735 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9222", 'reported_seq': 9319, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:17.046946+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:48:17.046946+0000', 'last_peered': '2024-09-16T04:48:17.046946+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:48:17.046946+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9222, 'log_dups_size': 0, 'ondisk_log_size': 9222, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27669772, 'num_objects': 3378, 'num_object_clones': 0, 'num_object_copies': 13512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9220, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1165}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:18.735 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:48:18.735 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9253", 'reported_seq': 9359, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:16.151185+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:48:16.151185+0000', 'last_peered': '2024-09-16T04:48:16.151185+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:48:16.151185+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9253, 'log_dups_size': 0, 'ondisk_log_size': 9253, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28710156, 'num_objects': 3505, 'num_object_clones': 0, 'num_object_copies': 14020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3505, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9251, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1316}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:18.735 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:48:18.735 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9145", 'reported_seq': 9276, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:17.286484+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:48:17.286484+0000', 'last_peered': '2024-09-16T04:48:17.286484+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:48:17.286484+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9145, 'log_dups_size': 0, 'ondisk_log_size': 9145, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27406226, 'num_objects': 3346, 'num_object_clones': 0, 'num_object_copies': 13384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9142, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1191}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:18.735 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:48:18.735 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'8959", 'reported_seq': 9051, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:17.289708+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:48:17.289708+0000', 'last_peered': '2024-09-16T04:48:17.289708+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:48:17.289708+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8959, 'log_dups_size': 0, 'ondisk_log_size': 8959, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27622022, 'num_objects': 3372, 'num_object_clones': 0, 'num_object_copies': 13488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3372, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8958, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1267}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:18.735 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:48:18.735 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9138", 'reported_seq': 9224, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:16.078874+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:48:16.078874+0000', 'last_peered': '2024-09-16T04:48:16.078874+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:48:16.078874+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9138, 'log_dups_size': 0, 'ondisk_log_size': 9138, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27810438, 'num_objects': 3395, 'num_object_clones': 0, 'num_object_copies': 13580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9137, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1174}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:18.736 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:48:18.736 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9288", 'reported_seq': 9420, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:17.278672+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:48:17.278672+0000', 'last_peered': '2024-09-16T04:48:17.278672+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:48:17.278672+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9288, 'log_dups_size': 0, 'ondisk_log_size': 9288, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28620044, 'num_objects': 3494, 'num_object_clones': 0, 'num_object_copies': 13976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9286, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1296}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:18.736 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:48:18.736 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'8993", 'reported_seq': 9095, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:16.544077+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:48:16.544077+0000', 'last_peered': '2024-09-16T04:48:16.544077+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:48:16.544077+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 8993, 'log_dups_size': 0, 'ondisk_log_size': 8993, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28308748, 'num_objects': 3456, 'num_object_clones': 0, 'num_object_copies': 13824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3456, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8991, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1333}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:18.736 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:48:18.736 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9041", 'reported_seq': 9133, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:17.145245+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:48:17.145245+0000', 'last_peered': '2024-09-16T04:48:17.145245+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:48:17.145245+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9041, 'log_dups_size': 0, 'ondisk_log_size': 9041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26752268, 'num_objects': 3266, 'num_object_clones': 0, 'num_object_copies': 13064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3266, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9039, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1038}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:18.736 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:48:19.455 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:19.455 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30424/218396 objects degraded (13.931%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":511,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54599,"data_bytes":447227584,"bytes_used":2857435136,"bytes_avail":383689621504,"bytes_total":386547056640,"degraded_objects":30424,"degraded_total":218396,"degraded_ratio":0.13930658070660634,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":235},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:48:19.486 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:48:19.487 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:20.092 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:20.093 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:20.117 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-09-16T04:48:18.702509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":447227584,"num_objects":54599,"num_object_clones":0,"num_object_copies":218396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146888,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5158,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146915,"ondisk_log_size":146915,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2790464,"kb_used_data":878896,"kb_used_omap":14,"kb_used_meta":1911473,"kb_avail":374696896,"statfs":{"total":386547056640,"available":383689621504,"internally_reserved":0,"allocated":899989504,"data_stored":899232696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1957348913},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,9,8,16,12,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23158784,"num_objects":-2827,"num_object_clones":0,"num_object_copies":-11308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2827,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007019"},"pg_stats":[{"pgid":"1.f","version":"35'9209","reported_seq":9325,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.291210+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:17.291210+0000","last_peered":"2024-09-16T04:48:17.291210+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:17.291210+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9209,"log_dups_size":0,"ondisk_log_size":9209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28221440,"num_objects":3445,"num_object_clones":0,"num_object_copies":13780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9209,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9214","reported_seq":9302,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:16.945045+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:16.945045+0000","last_peered":"2024-09-16T04:48:16.945045+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:16.945045+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9214,"log_dups_size":0,"ondisk_log_size":9214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28341516,"num_objects":3460,"num_object_clones":0,"num_object_copies":13840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9212,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9227","reported_seq":9325,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:16.498885+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:16.498885+0000","last_peered":"2024-09-16T04:48:16.498885+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:16.498885+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9227,"log_dups_size":0,"ondisk_log_size":9227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28203678,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9226,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1225}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9254","reported_seq":9912,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.175437+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:17.175437+0000","last_peered":"2024-09-16T04:48:17.175437+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:17.175437+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9254,"log_dups_size":0,"ondisk_log_size":9254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27981068,"num_objects":3416,"num_object_clones":0,"num_object_copies":13664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9252,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1478}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9301","reported_seq":9433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.230977+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:17.230977+0000","last_peered":"2024-09-16T04:48:17.230977+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:17.230977+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9301,"log_dups_size":0,"ondisk_log_size":9301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28177676,"num_objects":3440,"num_object_clones":0,"num_object_copies":13760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9299,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9008","reported_seq":9118,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.235854+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:17.235854+0000","last_peered":"2024-09-16T04:48:17.235854+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:17.235854+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9008,"log_dups_size":0,"ondisk_log_size":9008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27066368,"num_objects":3304,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9008,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1126}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9222","reported_seq":9319,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.046946+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:17.046946+0000","last_peered":"2024-09-16T04:48:17.046946+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:17.046946+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27669772,"num_objects":3378,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9220,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1165}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9278","reported_seq":13839,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:13.785783+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:13.785783+0000","last_peered":"2024-09-16T04:48:13.785783+0000","last_clean":"2024-09-16T04:48:13.785783+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:13.785783+0000","last_undegraded":"2024-09-16T04:48:13.785783+0000","last_fullsized":"2024-09-16T04:48:13.785783+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:17.067594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28726540,"num_objects":3507,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9385","reported_seq":14142,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:17.109349+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:17.109349+0000","last_peered":"2024-09-16T04:48:17.109349+0000","last_clean":"2024-09-16T04:48:17.109349+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:17.109349+0000","last_undegraded":"2024-09-16T04:48:17.109349+0000","last_fullsized":"2024-09-16T04:48:17.109349+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9385,"log_dups_size":0,"ondisk_log_size":9385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:20.862882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27609624,"num_objects":3371,"num_object_clones":0,"num_object_copies":13484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9382,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9253","reported_seq":9359,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:16.151185+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:16.151185+0000","last_peered":"2024-09-16T04:48:16.151185+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:16.151185+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9253,"log_dups_size":0,"ondisk_log_size":9253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28710156,"num_objects":3505,"num_object_clones":0,"num_object_copies":14020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9251,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1316}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9145","reported_seq":9276,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.286484+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:17.286484+0000","last_peered":"2024-09-16T04:48:17.286484+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:17.286484+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9145,"log_dups_size":0,"ondisk_log_size":9145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27406226,"num_objects":3346,"num_object_clones":0,"num_object_copies":13384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9142,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8959","reported_seq":9051,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.289708+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:17.289708+0000","last_peered":"2024-09-16T04:48:17.289708+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:17.289708+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8959,"log_dups_size":0,"ondisk_log_size":8959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27622022,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8958,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1267}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9138","reported_seq":9224,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:16.078874+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:16.078874+0000","last_peered":"2024-09-16T04:48:16.078874+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:16.078874+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9138,"log_dups_size":0,"ondisk_log_size":9138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27810438,"num_objects":3395,"num_object_clones":0,"num_object_copies":13580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9137,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1174}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9288","reported_seq":9420,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.278672+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:17.278672+0000","last_peered":"2024-09-16T04:48:17.278672+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:17.278672+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9288,"log_dups_size":0,"ondisk_log_size":9288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28620044,"num_objects":3494,"num_object_clones":0,"num_object_copies":13976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9286,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1296}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8993","reported_seq":9095,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:16.544077+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:16.544077+0000","last_peered":"2024-09-16T04:48:16.544077+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:16.544077+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":8993,"log_dups_size":0,"ondisk_log_size":8993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28308748,"num_objects":3456,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8991,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1333}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9041","reported_seq":9133,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:17.145245+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:17.145245+0000","last_peered":"2024-09-16T04:48:17.145245+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:17.145245+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9041,"log_dups_size":0,"ondisk_log_size":9041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26752268,"num_objects":3266,"num_object_clones":0,"num_object_copies":13064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9039,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":447227584,"num_objects":54599,"num_object_clones":0,"num_object_copies":218396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146888,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5158,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":896978944,"data_stored":896978944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146915,"ondisk_log_size":146915,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723748,"kb_used_data":218644,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93648092,"statfs":{"total":96636764160,"available":95895646208,"internally_reserved":0,"allocated":223891456,"data_stored":223702254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517205711},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732524,"kb_used_data":219612,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":93639316,"statfs":{"total":96636764160,"available":95886659584,"internally_reserved":0,"allocated":224882688,"data_stored":224693486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":525203909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051666,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610132,"kb_used_data":221636,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93761708,"statfs":{"total":96636764160,"available":96011988992,"internally_reserved":0,"allocated":226955264,"data_stored":226766062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":397799119},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724060,"kb_used_data":219004,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93647780,"statfs":{"total":96636764160,"available":95895326720,"internally_reserved":0,"allocated":224260096,"data_stored":224070894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,1,4,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":223330304,"data_stored":223330304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":226574336,"data_stored":226574336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":223891456,"data_stored":223891456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":223182848,"data_stored":223182848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:23.119 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:23.741 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:23.742 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:23.765 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-09-16T04:48:22.703501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":439338688,"num_objects":53636,"num_object_clones":0,"num_object_copies":214544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53636,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147851,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5275,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147878,"ondisk_log_size":147878,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2787944,"kb_used_data":862504,"kb_used_omap":14,"kb_used_meta":1925361,"kb_avail":374699416,"statfs":{"total":386547056640,"available":383692201984,"internally_reserved":0,"allocated":883204096,"data_stored":882447288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1971570225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,22,16,15,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24535040,"num_objects":-2995,"num_object_clones":0,"num_object_copies":-11980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2995,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003295"},"pg_stats":[{"pgid":"1.f","version":"35'9276","reported_seq":9392,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.357727+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:22.357727+0000","last_peered":"2024-09-16T04:48:22.357727+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:22.357727+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9276,"log_dups_size":0,"ondisk_log_size":9276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27672576,"num_objects":3378,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9279","reported_seq":9367,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.102834+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:22.102834+0000","last_peered":"2024-09-16T04:48:22.102834+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:22.102834+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9279,"log_dups_size":0,"ondisk_log_size":9279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27809036,"num_objects":3395,"num_object_clones":0,"num_object_copies":13580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9277,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9304","reported_seq":9402,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:21.476049+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:21.476049+0000","last_peered":"2024-09-16T04:48:21.476049+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:21.476049+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9304,"log_dups_size":0,"ondisk_log_size":9304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27572894,"num_objects":3367,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9303,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1148}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9303","reported_seq":10196,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.187999+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:22.187999+0000","last_peered":"2024-09-16T04:48:22.187999+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:22.187999+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9303,"log_dups_size":0,"ondisk_log_size":9303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27579660,"num_objects":3367,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9301,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1546}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9369","reported_seq":9501,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.197075+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:22.197075+0000","last_peered":"2024-09-16T04:48:22.197075+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:22.197075+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9369,"log_dups_size":0,"ondisk_log_size":9369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27620620,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9367,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9062","reported_seq":9172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.087332+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:22.087332+0000","last_peered":"2024-09-16T04:48:22.087332+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:22.087332+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9062,"log_dups_size":0,"ondisk_log_size":9062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26624000,"num_objects":3250,"num_object_clones":0,"num_object_copies":13000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9062,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1072}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9278","reported_seq":9375,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:21.458675+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:21.458675+0000","last_peered":"2024-09-16T04:48:21.458675+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:21.458675+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27211020,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1109}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9336","reported_seq":13897,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:18.726760+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:18.726760+0000","last_peered":"2024-09-16T04:48:18.726760+0000","last_clean":"2024-09-16T04:48:18.726760+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:18.726760+0000","last_undegraded":"2024-09-16T04:48:18.726760+0000","last_fullsized":"2024-09-16T04:48:18.726760+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9336,"log_dups_size":0,"ondisk_log_size":9336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:22.115164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28251404,"num_objects":3449,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9334,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9434","reported_seq":14191,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:22.319817+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:22.319817+0000","last_peered":"2024-09-16T04:48:22.319817+0000","last_clean":"2024-09-16T04:48:22.319817+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:22.319817+0000","last_undegraded":"2024-09-16T04:48:22.319817+0000","last_fullsized":"2024-09-16T04:48:22.319817+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9434,"log_dups_size":0,"ondisk_log_size":9434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:25.893425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27208216,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9431,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9327","reported_seq":9433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:21.461171+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:21.461171+0000","last_peered":"2024-09-16T04:48:21.461171+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:21.461171+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9327,"log_dups_size":0,"ondisk_log_size":9327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28103948,"num_objects":3431,"num_object_clones":0,"num_object_copies":13724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9325,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1242}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9197","reported_seq":9329,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.288941+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:22.288941+0000","last_peered":"2024-09-16T04:48:22.288941+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:22.288941+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9197,"log_dups_size":0,"ondisk_log_size":9197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26980242,"num_objects":3294,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9194,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9018","reported_seq":9110,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.169867+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:22.169867+0000","last_peered":"2024-09-16T04:48:22.169867+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:22.169867+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9018,"log_dups_size":0,"ondisk_log_size":9018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27138694,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9017,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1208}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9203","reported_seq":9291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:21.484425+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:21.484425+0000","last_peered":"2024-09-16T04:48:21.484425+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:21.484425+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9203,"log_dups_size":0,"ondisk_log_size":9203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27277958,"num_objects":3330,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9202,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1109}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9341","reported_seq":9473,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.206973+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:22.206973+0000","last_peered":"2024-09-16T04:48:22.206973+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:22.206973+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9341,"log_dups_size":0,"ondisk_log_size":9341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28185868,"num_objects":3441,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9339,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1243}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9052","reported_seq":9155,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:21.430384+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:21.430384+0000","last_peered":"2024-09-16T04:48:21.430384+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:21.430384+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9052,"log_dups_size":0,"ondisk_log_size":9052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27825420,"num_objects":3397,"num_object_clones":0,"num_object_copies":13588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9050,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1274}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9099","reported_seq":9193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.157598+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:22.157598+0000","last_peered":"2024-09-16T04:48:22.157598+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:22.157598+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26277132,"num_objects":3208,"num_object_clones":0,"num_object_copies":12832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9097,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":439338688,"num_objects":53636,"num_object_clones":0,"num_object_copies":214544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53636,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147851,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5275,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":880713728,"data_stored":880713728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147878,"ondisk_log_size":147878,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719880,"kb_used_data":214776,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93651960,"statfs":{"total":96636764160,"available":95899607040,"internally_reserved":0,"allocated":219930624,"data_stored":219741422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517205711},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715960,"kb_used_data":215528,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93655880,"statfs":{"total":96636764160,"available":95903621120,"internally_reserved":0,"allocated":220700672,"data_stored":220511470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,6,4,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051667,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628128,"kb_used_data":217424,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93743712,"statfs":{"total":96636764160,"available":95993561088,"internally_reserved":0,"allocated":222642176,"data_stored":222452974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420540111},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723976,"kb_used_data":214776,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93647864,"statfs":{"total":96636764160,"available":95895412736,"internally_reserved":0,"allocated":219930624,"data_stored":219741422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":521400014},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,7,0,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":219439104,"data_stored":219439104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":222289920,"data_stored":222289920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219652096,"data_stored":219652096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":219332608,"data_stored":219332608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:23.766 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:24.333 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:24.333 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:24.361 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-09-16T04:48:22.703501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":439338688,"num_objects":53636,"num_object_clones":0,"num_object_copies":214544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53636,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147851,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5275,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147878,"ondisk_log_size":147878,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2787944,"kb_used_data":862504,"kb_used_omap":14,"kb_used_meta":1925361,"kb_avail":374699416,"statfs":{"total":386547056640,"available":383692201984,"internally_reserved":0,"allocated":883204096,"data_stored":882447288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1971570225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,22,16,15,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24535040,"num_objects":-2995,"num_object_clones":0,"num_object_copies":-11980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2995,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003295"},"pg_stats":[{"pgid":"1.f","version":"35'9276","reported_seq":9392,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.357727+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:22.357727+0000","last_peered":"2024-09-16T04:48:22.357727+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:22.357727+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9276,"log_dups_size":0,"ondisk_log_size":9276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27672576,"num_objects":3378,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9279","reported_seq":9367,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.102834+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:22.102834+0000","last_peered":"2024-09-16T04:48:22.102834+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:22.102834+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9279,"log_dups_size":0,"ondisk_log_size":9279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27809036,"num_objects":3395,"num_object_clones":0,"num_object_copies":13580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9277,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9304","reported_seq":9402,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:21.476049+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:21.476049+0000","last_peered":"2024-09-16T04:48:21.476049+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:21.476049+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9304,"log_dups_size":0,"ondisk_log_size":9304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27572894,"num_objects":3367,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9303,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1148}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9303","reported_seq":10196,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.187999+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:22.187999+0000","last_peered":"2024-09-16T04:48:22.187999+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:22.187999+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9303,"log_dups_size":0,"ondisk_log_size":9303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27579660,"num_objects":3367,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9301,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1546}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9369","reported_seq":9501,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.197075+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:22.197075+0000","last_peered":"2024-09-16T04:48:22.197075+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:22.197075+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9369,"log_dups_size":0,"ondisk_log_size":9369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27620620,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9367,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9062","reported_seq":9172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.087332+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:22.087332+0000","last_peered":"2024-09-16T04:48:22.087332+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:22.087332+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9062,"log_dups_size":0,"ondisk_log_size":9062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26624000,"num_objects":3250,"num_object_clones":0,"num_object_copies":13000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9062,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1072}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9278","reported_seq":9375,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:21.458675+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:21.458675+0000","last_peered":"2024-09-16T04:48:21.458675+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:21.458675+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27211020,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1109}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9336","reported_seq":13897,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:18.726760+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:18.726760+0000","last_peered":"2024-09-16T04:48:18.726760+0000","last_clean":"2024-09-16T04:48:18.726760+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:18.726760+0000","last_undegraded":"2024-09-16T04:48:18.726760+0000","last_fullsized":"2024-09-16T04:48:18.726760+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9336,"log_dups_size":0,"ondisk_log_size":9336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:22.115164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28251404,"num_objects":3449,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9334,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9434","reported_seq":14191,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:22.319817+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:22.319817+0000","last_peered":"2024-09-16T04:48:22.319817+0000","last_clean":"2024-09-16T04:48:22.319817+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:22.319817+0000","last_undegraded":"2024-09-16T04:48:22.319817+0000","last_fullsized":"2024-09-16T04:48:22.319817+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9434,"log_dups_size":0,"ondisk_log_size":9434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:25.893425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27208216,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9431,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9327","reported_seq":9433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:21.461171+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:21.461171+0000","last_peered":"2024-09-16T04:48:21.461171+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:21.461171+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9327,"log_dups_size":0,"ondisk_log_size":9327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28103948,"num_objects":3431,"num_object_clones":0,"num_object_copies":13724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9325,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1242}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9197","reported_seq":9329,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.288941+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:22.288941+0000","last_peered":"2024-09-16T04:48:22.288941+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:22.288941+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9197,"log_dups_size":0,"ondisk_log_size":9197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26980242,"num_objects":3294,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9194,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9018","reported_seq":9110,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.169867+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:22.169867+0000","last_peered":"2024-09-16T04:48:22.169867+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:22.169867+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9018,"log_dups_size":0,"ondisk_log_size":9018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27138694,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9017,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1208}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9203","reported_seq":9291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:21.484425+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:21.484425+0000","last_peered":"2024-09-16T04:48:21.484425+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:21.484425+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9203,"log_dups_size":0,"ondisk_log_size":9203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27277958,"num_objects":3330,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9202,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1109}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9341","reported_seq":9473,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.206973+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:22.206973+0000","last_peered":"2024-09-16T04:48:22.206973+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:22.206973+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9341,"log_dups_size":0,"ondisk_log_size":9341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28185868,"num_objects":3441,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9339,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1243}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9052","reported_seq":9155,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:21.430384+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:21.430384+0000","last_peered":"2024-09-16T04:48:21.430384+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:21.430384+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9052,"log_dups_size":0,"ondisk_log_size":9052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27825420,"num_objects":3397,"num_object_clones":0,"num_object_copies":13588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9050,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1274}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9099","reported_seq":9193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.157598+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:22.157598+0000","last_peered":"2024-09-16T04:48:22.157598+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:22.157598+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26277132,"num_objects":3208,"num_object_clones":0,"num_object_copies":12832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9097,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":439338688,"num_objects":53636,"num_object_clones":0,"num_object_copies":214544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53636,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147851,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5275,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":880713728,"data_stored":880713728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147878,"ondisk_log_size":147878,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719880,"kb_used_data":214776,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93651960,"statfs":{"total":96636764160,"available":95899607040,"internally_reserved":0,"allocated":219930624,"data_stored":219741422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517205711},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715960,"kb_used_data":215528,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93655880,"statfs":{"total":96636764160,"available":95903621120,"internally_reserved":0,"allocated":220700672,"data_stored":220511470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,6,4,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051667,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628128,"kb_used_data":217424,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93743712,"statfs":{"total":96636764160,"available":95993561088,"internally_reserved":0,"allocated":222642176,"data_stored":222452974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420540111},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723976,"kb_used_data":214776,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93647864,"statfs":{"total":96636764160,"available":95895412736,"internally_reserved":0,"allocated":219930624,"data_stored":219741422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":521400014},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,7,0,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":219439104,"data_stored":219439104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":222289920,"data_stored":222289920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219652096,"data_stored":219652096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":219332608,"data_stored":219332608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:24.362 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:48:24.362 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9276", 'reported_seq': 9392, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:22.357727+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:48:22.357727+0000', 'last_peered': '2024-09-16T04:48:22.357727+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:48:22.357727+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9276, 'log_dups_size': 0, 'ondisk_log_size': 9276, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27672576, 'num_objects': 3378, 'num_object_clones': 0, 'num_object_copies': 13512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9276, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1192}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:24.362 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:48:24.362 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9279", 'reported_seq': 9367, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:22.102834+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:48:22.102834+0000', 'last_peered': '2024-09-16T04:48:22.102834+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:48:22.102834+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9279, 'log_dups_size': 0, 'ondisk_log_size': 9279, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27809036, 'num_objects': 3395, 'num_object_clones': 0, 'num_object_copies': 13580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9277, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1192}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:24.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:48:24.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9304", 'reported_seq': 9402, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:21.476049+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:48:21.476049+0000', 'last_peered': '2024-09-16T04:48:21.476049+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:48:21.476049+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9304, 'log_dups_size': 0, 'ondisk_log_size': 9304, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27572894, 'num_objects': 3367, 'num_object_clones': 0, 'num_object_copies': 13468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3367, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9303, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1148}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:24.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:48:24.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9303", 'reported_seq': 10196, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:22.187999+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:48:22.187999+0000', 'last_peered': '2024-09-16T04:48:22.187999+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:48:22.187999+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9303, 'log_dups_size': 0, 'ondisk_log_size': 9303, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27579660, 'num_objects': 3367, 'num_object_clones': 0, 'num_object_copies': 13468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1821, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9301, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 412, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1546}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:24.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:48:24.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9369", 'reported_seq': 9501, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:22.197075+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:48:22.197075+0000', 'last_peered': '2024-09-16T04:48:22.197075+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:48:22.197075+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9369, 'log_dups_size': 0, 'ondisk_log_size': 9369, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27620620, 'num_objects': 3372, 'num_object_clones': 0, 'num_object_copies': 13488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3372, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9367, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1124}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:24.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:48:24.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9062", 'reported_seq': 9172, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:22.087332+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:48:22.087332+0000', 'last_peered': '2024-09-16T04:48:22.087332+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:48:22.087332+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9062, 'log_dups_size': 0, 'ondisk_log_size': 9062, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26624000, 'num_objects': 3250, 'num_object_clones': 0, 'num_object_copies': 13000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9062, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1072}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:24.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:48:24.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9278", 'reported_seq': 9375, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:21.458675+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:48:21.458675+0000', 'last_peered': '2024-09-16T04:48:21.458675+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:48:21.458675+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9278, 'log_dups_size': 0, 'ondisk_log_size': 9278, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27211020, 'num_objects': 3322, 'num_object_clones': 0, 'num_object_copies': 13288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3322, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9276, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1109}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:24.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:48:24.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9327", 'reported_seq': 9433, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:21.461171+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:48:21.461171+0000', 'last_peered': '2024-09-16T04:48:21.461171+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:48:21.461171+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9327, 'log_dups_size': 0, 'ondisk_log_size': 9327, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28103948, 'num_objects': 3431, 'num_object_clones': 0, 'num_object_copies': 13724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9325, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1242}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:24.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:48:24.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9197", 'reported_seq': 9329, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:22.288941+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:48:22.288941+0000', 'last_peered': '2024-09-16T04:48:22.288941+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:48:22.288941+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9197, 'log_dups_size': 0, 'ondisk_log_size': 9197, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26980242, 'num_objects': 3294, 'num_object_clones': 0, 'num_object_copies': 13176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9194, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1139}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:24.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:48:24.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9018", 'reported_seq': 9110, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:22.169867+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:48:22.169867+0000', 'last_peered': '2024-09-16T04:48:22.169867+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:48:22.169867+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9018, 'log_dups_size': 0, 'ondisk_log_size': 9018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27138694, 'num_objects': 3313, 'num_object_clones': 0, 'num_object_copies': 13252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3313, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9017, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1208}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:24.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:48:24.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9203", 'reported_seq': 9291, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:21.484425+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:48:21.484425+0000', 'last_peered': '2024-09-16T04:48:21.484425+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:48:21.484425+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9203, 'log_dups_size': 0, 'ondisk_log_size': 9203, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27277958, 'num_objects': 3330, 'num_object_clones': 0, 'num_object_copies': 13320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9202, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:24.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:48:24.365 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9341", 'reported_seq': 9473, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:22.206973+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:48:22.206973+0000', 'last_peered': '2024-09-16T04:48:22.206973+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:48:22.206973+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9341, 'log_dups_size': 0, 'ondisk_log_size': 9341, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28185868, 'num_objects': 3441, 'num_object_clones': 0, 'num_object_copies': 13764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9339, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1243}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:24.365 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:48:24.365 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9052", 'reported_seq': 9155, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:21.430384+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:48:21.430384+0000', 'last_peered': '2024-09-16T04:48:21.430384+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:48:21.430384+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9052, 'log_dups_size': 0, 'ondisk_log_size': 9052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27825420, 'num_objects': 3397, 'num_object_clones': 0, 'num_object_copies': 13588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9050, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1274}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:24.365 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:48:24.365 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9099", 'reported_seq': 9193, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:22.157598+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:48:22.157598+0000', 'last_peered': '2024-09-16T04:48:22.157598+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:48:22.157598+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9099, 'log_dups_size': 0, 'ondisk_log_size': 9099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26277132, 'num_objects': 3208, 'num_object_clones': 0, 'num_object_copies': 12832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3208, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9097, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 980}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:24.365 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:48:25.094 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:25.094 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30307/214276 objects degraded (14.144%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":517,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":53569,"data_bytes":438789824,"bytes_used":2855194624,"bytes_avail":383691862016,"bytes_total":386547056640,"degraded_objects":30307,"degraded_total":214276,"degraded_ratio":0.1414390785715619,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":169},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:48:25.116 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:48:25.116 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:25.755 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:25.755 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:25.775 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-16T04:48:24.704039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":438789824,"num_objects":53569,"num_object_clones":0,"num_object_copies":214276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147918,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5275,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147945,"ondisk_log_size":147945,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788276,"kb_used_data":858676,"kb_used_omap":14,"kb_used_meta":1929521,"kb_avail":374699084,"statfs":{"total":386547056640,"available":383691862016,"internally_reserved":0,"allocated":879284224,"data_stored":878527416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1975830065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,22,16,15,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16687104,"num_objects":-2037,"num_object_clones":0,"num_object_copies":-8148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2037,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003351"},"pg_stats":[{"pgid":"1.f","version":"35'9276","reported_seq":9392,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.357727+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:22.357727+0000","last_peered":"2024-09-16T04:48:22.357727+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:22.357727+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9276,"log_dups_size":0,"ondisk_log_size":9276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27672576,"num_objects":3378,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9279","reported_seq":9367,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.102834+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:22.102834+0000","last_peered":"2024-09-16T04:48:22.102834+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:22.102834+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9279,"log_dups_size":0,"ondisk_log_size":9279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27809036,"num_objects":3395,"num_object_clones":0,"num_object_copies":13580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9277,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9304","reported_seq":9402,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:21.476049+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:21.476049+0000","last_peered":"2024-09-16T04:48:21.476049+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:21.476049+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9304,"log_dups_size":0,"ondisk_log_size":9304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27572894,"num_objects":3367,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9303,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1148}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9303","reported_seq":10196,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.187999+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:22.187999+0000","last_peered":"2024-09-16T04:48:22.187999+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:22.187999+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9303,"log_dups_size":0,"ondisk_log_size":9303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27579660,"num_objects":3367,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9301,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1546}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9369","reported_seq":9501,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.197075+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:22.197075+0000","last_peered":"2024-09-16T04:48:22.197075+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:22.197075+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9369,"log_dups_size":0,"ondisk_log_size":9369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27620620,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9367,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9062","reported_seq":9172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.087332+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:22.087332+0000","last_peered":"2024-09-16T04:48:22.087332+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:22.087332+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9062,"log_dups_size":0,"ondisk_log_size":9062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26624000,"num_objects":3250,"num_object_clones":0,"num_object_copies":13000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9062,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1072}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9278","reported_seq":9375,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:21.458675+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:21.458675+0000","last_peered":"2024-09-16T04:48:21.458675+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:21.458675+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27211020,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1109}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9403","reported_seq":13962,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:24.350617+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:24.350617+0000","last_peered":"2024-09-16T04:48:24.350617+0000","last_clean":"2024-09-16T04:48:24.350617+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:24.350617+0000","last_undegraded":"2024-09-16T04:48:24.350617+0000","last_fullsized":"2024-09-16T04:48:24.350617+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9403,"log_dups_size":0,"ondisk_log_size":9403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:27.127804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27702540,"num_objects":3382,"num_object_clones":0,"num_object_copies":13528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9401,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9434","reported_seq":14191,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:22.319817+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:22.319817+0000","last_peered":"2024-09-16T04:48:22.319817+0000","last_clean":"2024-09-16T04:48:22.319817+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:22.319817+0000","last_undegraded":"2024-09-16T04:48:22.319817+0000","last_fullsized":"2024-09-16T04:48:22.319817+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9434,"log_dups_size":0,"ondisk_log_size":9434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:25.893425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27208216,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9431,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9327","reported_seq":9433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:21.461171+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:21.461171+0000","last_peered":"2024-09-16T04:48:21.461171+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:21.461171+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9327,"log_dups_size":0,"ondisk_log_size":9327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28103948,"num_objects":3431,"num_object_clones":0,"num_object_copies":13724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9325,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1242}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9197","reported_seq":9329,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.288941+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:22.288941+0000","last_peered":"2024-09-16T04:48:22.288941+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:22.288941+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9197,"log_dups_size":0,"ondisk_log_size":9197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26980242,"num_objects":3294,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9194,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9018","reported_seq":9110,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.169867+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:22.169867+0000","last_peered":"2024-09-16T04:48:22.169867+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:22.169867+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9018,"log_dups_size":0,"ondisk_log_size":9018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27138694,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9017,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1208}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9203","reported_seq":9291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:21.484425+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:21.484425+0000","last_peered":"2024-09-16T04:48:21.484425+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:21.484425+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9203,"log_dups_size":0,"ondisk_log_size":9203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27277958,"num_objects":3330,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9202,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1109}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9341","reported_seq":9473,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.206973+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:22.206973+0000","last_peered":"2024-09-16T04:48:22.206973+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:22.206973+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9341,"log_dups_size":0,"ondisk_log_size":9341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28185868,"num_objects":3441,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9339,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1243}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9052","reported_seq":9155,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:21.430384+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:21.430384+0000","last_peered":"2024-09-16T04:48:21.430384+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:21.430384+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9052,"log_dups_size":0,"ondisk_log_size":9052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27825420,"num_objects":3397,"num_object_clones":0,"num_object_copies":13588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9050,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1274}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9099","reported_seq":9193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:22.157598+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:22.157598+0000","last_peered":"2024-09-16T04:48:22.157598+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:22.157598+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9099,"log_dups_size":0,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26277132,"num_objects":3208,"num_object_clones":0,"num_object_copies":12832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9097,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":438789824,"num_objects":53569,"num_object_clones":0,"num_object_copies":214276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147918,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5275,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":876687360,"data_stored":876687360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147945,"ondisk_log_size":147945,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719880,"kb_used_data":214776,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93651960,"statfs":{"total":96636764160,"available":95899607040,"internally_reserved":0,"allocated":219930624,"data_stored":219741422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517205711},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715960,"kb_used_data":215528,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93655880,"statfs":{"total":96636764160,"available":95903621120,"internally_reserved":0,"allocated":220700672,"data_stored":220511470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,6,4,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051668,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628460,"kb_used_data":213596,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93743380,"statfs":{"total":96636764160,"available":95993221120,"internally_reserved":0,"allocated":218722304,"data_stored":218533102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424799951},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723976,"kb_used_data":214776,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93647864,"statfs":{"total":96636764160,"available":95895412736,"internally_reserved":0,"allocated":219930624,"data_stored":219741422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":521400014},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,7,0,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":219439104,"data_stored":219439104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":218263552,"data_stored":218263552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219652096,"data_stored":219652096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":219332608,"data_stored":219332608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:28.777 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:29.430 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:29.432 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:29.462 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-16T04:48:28.705041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":431023808,"num_objects":52621,"num_object_clones":0,"num_object_copies":210484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148866,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5320,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148893,"ondisk_log_size":148893,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772696,"kb_used_data":847228,"kb_used_omap":14,"kb_used_meta":1925361,"kb_avail":374714664,"statfs":{"total":386547056640,"available":383707815936,"internally_reserved":0,"allocated":867561472,"data_stored":866804664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1971570225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,0,2,9,11,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22020096,"num_objects":-2688,"num_object_clones":0,"num_object_copies":-10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2688,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003361"},"pg_stats":[{"pgid":"1.f","version":"35'9343","reported_seq":9458,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.410347+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:27.410347+0000","last_peered":"2024-09-16T04:48:27.410347+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:27.410347+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9343,"log_dups_size":0,"ondisk_log_size":9343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27123712,"num_objects":3311,"num_object_clones":0,"num_object_copies":13244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9343,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9336","reported_seq":9424,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.152548+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:27.152548+0000","last_peered":"2024-09-16T04:48:27.152548+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:27.152548+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9336,"log_dups_size":0,"ondisk_log_size":9336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27342092,"num_objects":3338,"num_object_clones":0,"num_object_copies":13352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9334,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1135}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9358","reported_seq":9456,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:26.461011+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:26.461011+0000","last_peered":"2024-09-16T04:48:26.461011+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:26.461011+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9358,"log_dups_size":0,"ondisk_log_size":9358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27130526,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9357,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1094}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9372","reported_seq":10355,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.222763+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:27.222763+0000","last_peered":"2024-09-16T04:48:27.222763+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:27.222763+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9372,"log_dups_size":0,"ondisk_log_size":9372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27014412,"num_objects":3298,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9370,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9439","reported_seq":9571,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.279642+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:27.279642+0000","last_peered":"2024-09-16T04:48:27.279642+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:27.279642+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9439,"log_dups_size":0,"ondisk_log_size":9439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27047180,"num_objects":3302,"num_object_clones":0,"num_object_copies":13208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9437,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1054}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9108","reported_seq":9218,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.191306+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:27.191306+0000","last_peered":"2024-09-16T04:48:27.191306+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:27.191306+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9108,"log_dups_size":0,"ondisk_log_size":9108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26247168,"num_objects":3204,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9108,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1026}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9338","reported_seq":9435,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:26.962974+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:26.962974+0000","last_peered":"2024-09-16T04:48:26.962974+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:26.962974+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9338,"log_dups_size":0,"ondisk_log_size":9338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26719500,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9336,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1049}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9403","reported_seq":13962,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:24.350617+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:24.350617+0000","last_peered":"2024-09-16T04:48:24.350617+0000","last_clean":"2024-09-16T04:48:24.350617+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:24.350617+0000","last_undegraded":"2024-09-16T04:48:24.350617+0000","last_fullsized":"2024-09-16T04:48:24.350617+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9403,"log_dups_size":0,"ondisk_log_size":9403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:27.127804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27702540,"num_objects":3382,"num_object_clones":0,"num_object_copies":13528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9401,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9502","reported_seq":14259,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:27.205057+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:27.205057+0000","last_peered":"2024-09-16T04:48:27.205057+0000","last_clean":"2024-09-16T04:48:27.205057+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:27.205057+0000","last_undegraded":"2024-09-16T04:48:27.205057+0000","last_fullsized":"2024-09-16T04:48:27.205057+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9502,"log_dups_size":0,"ondisk_log_size":9502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:31.868098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26651160,"num_objects":3254,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9499,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9386","reported_seq":9492,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:26.402351+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:26.402351+0000","last_peered":"2024-09-16T04:48:26.402351+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:26.402351+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9386,"log_dups_size":0,"ondisk_log_size":9386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27620620,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9384,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1183}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9272","reported_seq":9403,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.336988+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:27.336988+0000","last_peered":"2024-09-16T04:48:27.336988+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:27.336988+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9272,"log_dups_size":0,"ondisk_log_size":9272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26365842,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9269,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9079","reported_seq":9171,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.285954+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:27.285954+0000","last_peered":"2024-09-16T04:48:27.285954+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:27.285954+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9079,"log_dups_size":0,"ondisk_log_size":9079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26638982,"num_objects":3252,"num_object_clones":0,"num_object_copies":13008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9078,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1147}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9257","reported_seq":9345,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:26.298715+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:26.298715+0000","last_peered":"2024-09-16T04:48:26.298715+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:26.298715+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9257,"log_dups_size":0,"ondisk_log_size":9257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835590,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9256,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1055}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9419","reported_seq":9551,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.298606+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:27.298606+0000","last_peered":"2024-09-16T04:48:27.298606+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:27.298606+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9419,"log_dups_size":0,"ondisk_log_size":9419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27546892,"num_objects":3363,"num_object_clones":0,"num_object_copies":13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9417,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1165}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9112","reported_seq":9215,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:26.563844+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:26.563844+0000","last_peered":"2024-09-16T04:48:26.563844+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:26.563844+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9112,"log_dups_size":0,"ondisk_log_size":9112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27333900,"num_objects":3337,"num_object_clones":0,"num_object_copies":13348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9110,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1214}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9169","reported_seq":9263,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.259540+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:27.259540+0000","last_peered":"2024-09-16T04:48:27.259540+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:27.259540+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9169,"log_dups_size":0,"ondisk_log_size":9169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25703692,"num_objects":3138,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9167,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":910}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":431023808,"num_objects":52621,"num_object_clones":0,"num_object_copies":210484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148866,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5320,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":864329728,"data_stored":864329728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148893,"ondisk_log_size":148893,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720092,"kb_used_data":210828,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93651748,"statfs":{"total":96636764160,"available":95899389952,"internally_reserved":0,"allocated":215887872,"data_stored":215698670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521465551},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,0,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716392,"kb_used_data":211796,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93655448,"statfs":{"total":96636764160,"available":95903178752,"internally_reserved":0,"allocated":216879104,"data_stored":216689902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051668,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628460,"kb_used_data":213596,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93743380,"statfs":{"total":96636764160,"available":95993221120,"internally_reserved":0,"allocated":218722304,"data_stored":218533102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424799951},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707752,"kb_used_data":211008,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93664088,"statfs":{"total":96636764160,"available":95912026112,"internally_reserved":0,"allocated":216072192,"data_stored":215882990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":508620494},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,0,1,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":215195648,"data_stored":215195648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":218263552,"data_stored":218263552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":215801856,"data_stored":215801856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":215068672,"data_stored":215068672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:29.463 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:30.049 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:30.049 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:30.078 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-16T04:48:28.705041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":431023808,"num_objects":52621,"num_object_clones":0,"num_object_copies":210484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148866,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5320,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148893,"ondisk_log_size":148893,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772696,"kb_used_data":847228,"kb_used_omap":14,"kb_used_meta":1925361,"kb_avail":374714664,"statfs":{"total":386547056640,"available":383707815936,"internally_reserved":0,"allocated":867561472,"data_stored":866804664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1971570225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,0,2,9,11,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22020096,"num_objects":-2688,"num_object_clones":0,"num_object_copies":-10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2688,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003361"},"pg_stats":[{"pgid":"1.f","version":"35'9343","reported_seq":9458,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.410347+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:27.410347+0000","last_peered":"2024-09-16T04:48:27.410347+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:27.410347+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9343,"log_dups_size":0,"ondisk_log_size":9343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27123712,"num_objects":3311,"num_object_clones":0,"num_object_copies":13244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9343,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9336","reported_seq":9424,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.152548+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:27.152548+0000","last_peered":"2024-09-16T04:48:27.152548+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:27.152548+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9336,"log_dups_size":0,"ondisk_log_size":9336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27342092,"num_objects":3338,"num_object_clones":0,"num_object_copies":13352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9334,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1135}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9358","reported_seq":9456,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:26.461011+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:26.461011+0000","last_peered":"2024-09-16T04:48:26.461011+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:26.461011+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9358,"log_dups_size":0,"ondisk_log_size":9358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27130526,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9357,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1094}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9372","reported_seq":10355,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.222763+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:27.222763+0000","last_peered":"2024-09-16T04:48:27.222763+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:27.222763+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9372,"log_dups_size":0,"ondisk_log_size":9372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27014412,"num_objects":3298,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9370,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9439","reported_seq":9571,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.279642+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:27.279642+0000","last_peered":"2024-09-16T04:48:27.279642+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:27.279642+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9439,"log_dups_size":0,"ondisk_log_size":9439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27047180,"num_objects":3302,"num_object_clones":0,"num_object_copies":13208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9437,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1054}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9108","reported_seq":9218,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.191306+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:27.191306+0000","last_peered":"2024-09-16T04:48:27.191306+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:27.191306+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9108,"log_dups_size":0,"ondisk_log_size":9108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26247168,"num_objects":3204,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9108,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1026}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9338","reported_seq":9435,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:26.962974+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:26.962974+0000","last_peered":"2024-09-16T04:48:26.962974+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:26.962974+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9338,"log_dups_size":0,"ondisk_log_size":9338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26719500,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9336,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1049}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9403","reported_seq":13962,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:24.350617+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:24.350617+0000","last_peered":"2024-09-16T04:48:24.350617+0000","last_clean":"2024-09-16T04:48:24.350617+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:24.350617+0000","last_undegraded":"2024-09-16T04:48:24.350617+0000","last_fullsized":"2024-09-16T04:48:24.350617+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9403,"log_dups_size":0,"ondisk_log_size":9403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:27.127804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27702540,"num_objects":3382,"num_object_clones":0,"num_object_copies":13528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9401,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9502","reported_seq":14259,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:27.205057+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:27.205057+0000","last_peered":"2024-09-16T04:48:27.205057+0000","last_clean":"2024-09-16T04:48:27.205057+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:27.205057+0000","last_undegraded":"2024-09-16T04:48:27.205057+0000","last_fullsized":"2024-09-16T04:48:27.205057+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9502,"log_dups_size":0,"ondisk_log_size":9502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:31.868098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26651160,"num_objects":3254,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9499,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9386","reported_seq":9492,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:26.402351+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:26.402351+0000","last_peered":"2024-09-16T04:48:26.402351+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:26.402351+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9386,"log_dups_size":0,"ondisk_log_size":9386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27620620,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9384,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1183}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9272","reported_seq":9403,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.336988+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:27.336988+0000","last_peered":"2024-09-16T04:48:27.336988+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:27.336988+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9272,"log_dups_size":0,"ondisk_log_size":9272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26365842,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9269,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9079","reported_seq":9171,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.285954+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:27.285954+0000","last_peered":"2024-09-16T04:48:27.285954+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:27.285954+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9079,"log_dups_size":0,"ondisk_log_size":9079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26638982,"num_objects":3252,"num_object_clones":0,"num_object_copies":13008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9078,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1147}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9257","reported_seq":9345,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:26.298715+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:26.298715+0000","last_peered":"2024-09-16T04:48:26.298715+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:26.298715+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9257,"log_dups_size":0,"ondisk_log_size":9257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835590,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9256,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1055}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9419","reported_seq":9551,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.298606+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:27.298606+0000","last_peered":"2024-09-16T04:48:27.298606+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:27.298606+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9419,"log_dups_size":0,"ondisk_log_size":9419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27546892,"num_objects":3363,"num_object_clones":0,"num_object_copies":13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9417,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1165}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9112","reported_seq":9215,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:26.563844+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:26.563844+0000","last_peered":"2024-09-16T04:48:26.563844+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:26.563844+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9112,"log_dups_size":0,"ondisk_log_size":9112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27333900,"num_objects":3337,"num_object_clones":0,"num_object_copies":13348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9110,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1214}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9169","reported_seq":9263,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.259540+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:27.259540+0000","last_peered":"2024-09-16T04:48:27.259540+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:27.259540+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9169,"log_dups_size":0,"ondisk_log_size":9169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25703692,"num_objects":3138,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9167,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":910}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":431023808,"num_objects":52621,"num_object_clones":0,"num_object_copies":210484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148866,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5320,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":864329728,"data_stored":864329728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148893,"ondisk_log_size":148893,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720092,"kb_used_data":210828,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93651748,"statfs":{"total":96636764160,"available":95899389952,"internally_reserved":0,"allocated":215887872,"data_stored":215698670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521465551},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,0,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716392,"kb_used_data":211796,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93655448,"statfs":{"total":96636764160,"available":95903178752,"internally_reserved":0,"allocated":216879104,"data_stored":216689902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051668,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628460,"kb_used_data":213596,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93743380,"statfs":{"total":96636764160,"available":95993221120,"internally_reserved":0,"allocated":218722304,"data_stored":218533102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424799951},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707752,"kb_used_data":211008,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93664088,"statfs":{"total":96636764160,"available":95912026112,"internally_reserved":0,"allocated":216072192,"data_stored":215882990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":508620494},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,0,1,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":215195648,"data_stored":215195648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":218263552,"data_stored":218263552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":215801856,"data_stored":215801856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":215068672,"data_stored":215068672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:30.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:48:30.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9343", 'reported_seq': 9458, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:27.410347+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:48:27.410347+0000', 'last_peered': '2024-09-16T04:48:27.410347+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:48:27.410347+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9343, 'log_dups_size': 0, 'ondisk_log_size': 9343, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27123712, 'num_objects': 3311, 'num_object_clones': 0, 'num_object_copies': 13244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3311, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9343, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1125}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:30.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:48:30.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9336", 'reported_seq': 9424, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:27.152548+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:48:27.152548+0000', 'last_peered': '2024-09-16T04:48:27.152548+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:48:27.152548+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9336, 'log_dups_size': 0, 'ondisk_log_size': 9336, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27342092, 'num_objects': 3338, 'num_object_clones': 0, 'num_object_copies': 13352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3338, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9334, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1135}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:30.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:48:30.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9358", 'reported_seq': 9456, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:26.461011+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:48:26.461011+0000', 'last_peered': '2024-09-16T04:48:26.461011+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:48:26.461011+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9358, 'log_dups_size': 0, 'ondisk_log_size': 9358, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27130526, 'num_objects': 3313, 'num_object_clones': 0, 'num_object_copies': 13252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3313, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9357, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1094}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:30.080 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:48:30.080 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9372", 'reported_seq': 10355, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:27.222763+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:48:27.222763+0000', 'last_peered': '2024-09-16T04:48:27.222763+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:48:27.222763+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9372, 'log_dups_size': 0, 'ondisk_log_size': 9372, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27014412, 'num_objects': 3298, 'num_object_clones': 0, 'num_object_copies': 13192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1776, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3298, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9370, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 457, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:30.080 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:48:30.080 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9439", 'reported_seq': 9571, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:27.279642+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:48:27.279642+0000', 'last_peered': '2024-09-16T04:48:27.279642+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:48:27.279642+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9439, 'log_dups_size': 0, 'ondisk_log_size': 9439, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27047180, 'num_objects': 3302, 'num_object_clones': 0, 'num_object_copies': 13208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3302, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9437, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1054}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:30.080 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:48:30.080 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9108", 'reported_seq': 9218, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:27.191306+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:48:27.191306+0000', 'last_peered': '2024-09-16T04:48:27.191306+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:48:27.191306+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9108, 'log_dups_size': 0, 'ondisk_log_size': 9108, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26247168, 'num_objects': 3204, 'num_object_clones': 0, 'num_object_copies': 12816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9108, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1026}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:30.080 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:48:30.080 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9338", 'reported_seq': 9435, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:26.962974+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:48:26.962974+0000', 'last_peered': '2024-09-16T04:48:26.962974+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:48:26.962974+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9338, 'log_dups_size': 0, 'ondisk_log_size': 9338, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26719500, 'num_objects': 3262, 'num_object_clones': 0, 'num_object_copies': 13048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9336, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1049}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:30.080 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:48:30.080 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9386", 'reported_seq': 9492, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:26.402351+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:48:26.402351+0000', 'last_peered': '2024-09-16T04:48:26.402351+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:48:26.402351+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9386, 'log_dups_size': 0, 'ondisk_log_size': 9386, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27620620, 'num_objects': 3372, 'num_object_clones': 0, 'num_object_copies': 13488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3372, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9384, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1183}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:30.081 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:48:30.081 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9272", 'reported_seq': 9403, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:27.336988+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:48:27.336988+0000', 'last_peered': '2024-09-16T04:48:27.336988+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:48:27.336988+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9272, 'log_dups_size': 0, 'ondisk_log_size': 9272, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26365842, 'num_objects': 3219, 'num_object_clones': 0, 'num_object_copies': 12876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9269, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1064}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:30.081 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:48:30.081 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9079", 'reported_seq': 9171, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:27.285954+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:48:27.285954+0000', 'last_peered': '2024-09-16T04:48:27.285954+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:48:27.285954+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9079, 'log_dups_size': 0, 'ondisk_log_size': 9079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26638982, 'num_objects': 3252, 'num_object_clones': 0, 'num_object_copies': 13008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9078, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1147}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:30.081 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:48:30.081 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9257", 'reported_seq': 9345, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:26.298715+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:48:26.298715+0000', 'last_peered': '2024-09-16T04:48:26.298715+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:48:26.298715+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9257, 'log_dups_size': 0, 'ondisk_log_size': 9257, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26835590, 'num_objects': 3276, 'num_object_clones': 0, 'num_object_copies': 13104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9256, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1055}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:30.081 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:48:30.081 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9419", 'reported_seq': 9551, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:27.298606+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:48:27.298606+0000', 'last_peered': '2024-09-16T04:48:27.298606+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:48:27.298606+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9419, 'log_dups_size': 0, 'ondisk_log_size': 9419, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27546892, 'num_objects': 3363, 'num_object_clones': 0, 'num_object_copies': 13452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9417, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1165}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:30.081 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:48:30.081 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9112", 'reported_seq': 9215, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:26.563844+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:48:26.563844+0000', 'last_peered': '2024-09-16T04:48:26.563844+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:48:26.563844+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9112, 'log_dups_size': 0, 'ondisk_log_size': 9112, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27333900, 'num_objects': 3337, 'num_object_clones': 0, 'num_object_copies': 13348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9110, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1214}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:30.082 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:48:30.082 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9169", 'reported_seq': 9263, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:27.259540+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:48:27.259540+0000', 'last_peered': '2024-09-16T04:48:27.259540+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:48:27.259540+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9169, 'log_dups_size': 0, 'ondisk_log_size': 9169, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25703692, 'num_objects': 3138, 'num_object_clones': 0, 'num_object_copies': 12552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3138, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9167, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:30.082 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:48:30.819 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:30.819 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30262/210484 objects degraded (14.377%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":523,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":52621,"data_bytes":431023808,"bytes_used":2839240704,"bytes_avail":383707815936,"bytes_total":386547056640,"degraded_objects":30262,"degraded_total":210484,"degraded_ratio":0.14377339845308906,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":217,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":223},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:48:30.861 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:48:30.861 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:31.503 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:31.504 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:31.535 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-09-16T04:48:30.705560+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":430425792,"num_objects":52548,"num_object_clones":0,"num_object_copies":210192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52548,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148939,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5320,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148966,"ondisk_log_size":148966,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772572,"kb_used_data":842944,"kb_used_omap":14,"kb_used_meta":1929521,"kb_avail":374714788,"statfs":{"total":386547056640,"available":383707942912,"internally_reserved":0,"allocated":863174656,"data_stored":862417848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1975830065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,1,3,5,6,12,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16801792,"num_objects":-2051,"num_object_clones":0,"num_object_copies":-8204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2051,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003050"},"pg_stats":[{"pgid":"1.f","version":"35'9343","reported_seq":9458,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.410347+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:27.410347+0000","last_peered":"2024-09-16T04:48:27.410347+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:27.410347+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9343,"log_dups_size":0,"ondisk_log_size":9343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27123712,"num_objects":3311,"num_object_clones":0,"num_object_copies":13244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9343,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9336","reported_seq":9424,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.152548+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:27.152548+0000","last_peered":"2024-09-16T04:48:27.152548+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:27.152548+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9336,"log_dups_size":0,"ondisk_log_size":9336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27342092,"num_objects":3338,"num_object_clones":0,"num_object_copies":13352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9334,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1135}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9358","reported_seq":9456,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:26.461011+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:26.461011+0000","last_peered":"2024-09-16T04:48:26.461011+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:26.461011+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9358,"log_dups_size":0,"ondisk_log_size":9358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27130526,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9357,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1094}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9372","reported_seq":10355,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.222763+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:27.222763+0000","last_peered":"2024-09-16T04:48:27.222763+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:27.222763+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9372,"log_dups_size":0,"ondisk_log_size":9372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27014412,"num_objects":3298,"num_object_clones":0,"num_object_copies":13192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9370,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9439","reported_seq":9571,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.279642+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:27.279642+0000","last_peered":"2024-09-16T04:48:27.279642+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:27.279642+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9439,"log_dups_size":0,"ondisk_log_size":9439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27047180,"num_objects":3302,"num_object_clones":0,"num_object_copies":13208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9437,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1054}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9108","reported_seq":9218,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.191306+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:27.191306+0000","last_peered":"2024-09-16T04:48:27.191306+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:27.191306+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9108,"log_dups_size":0,"ondisk_log_size":9108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26247168,"num_objects":3204,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9108,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1026}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9338","reported_seq":9435,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:26.962974+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:26.962974+0000","last_peered":"2024-09-16T04:48:26.962974+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:26.962974+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9338,"log_dups_size":0,"ondisk_log_size":9338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26719500,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9336,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1049}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9476","reported_seq":14037,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:29.360842+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:29.360842+0000","last_peered":"2024-09-16T04:48:29.360842+0000","last_clean":"2024-09-16T04:48:29.360842+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:29.360842+0000","last_undegraded":"2024-09-16T04:48:29.360842+0000","last_fullsized":"2024-09-16T04:48:29.360842+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9476,"log_dups_size":0,"ondisk_log_size":9476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:32.278106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27104524,"num_objects":3309,"num_object_clones":0,"num_object_copies":13236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9474,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9502","reported_seq":14259,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:27.205057+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:27.205057+0000","last_peered":"2024-09-16T04:48:27.205057+0000","last_clean":"2024-09-16T04:48:27.205057+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:27.205057+0000","last_undegraded":"2024-09-16T04:48:27.205057+0000","last_fullsized":"2024-09-16T04:48:27.205057+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9502,"log_dups_size":0,"ondisk_log_size":9502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:31.868098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26651160,"num_objects":3254,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9499,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9386","reported_seq":9492,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:26.402351+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:26.402351+0000","last_peered":"2024-09-16T04:48:26.402351+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:26.402351+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9386,"log_dups_size":0,"ondisk_log_size":9386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27620620,"num_objects":3372,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9384,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1183}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9272","reported_seq":9403,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.336988+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:27.336988+0000","last_peered":"2024-09-16T04:48:27.336988+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:27.336988+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9272,"log_dups_size":0,"ondisk_log_size":9272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26365842,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9269,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9079","reported_seq":9171,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.285954+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:27.285954+0000","last_peered":"2024-09-16T04:48:27.285954+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:27.285954+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9079,"log_dups_size":0,"ondisk_log_size":9079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26638982,"num_objects":3252,"num_object_clones":0,"num_object_copies":13008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9078,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1147}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9257","reported_seq":9345,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:26.298715+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:26.298715+0000","last_peered":"2024-09-16T04:48:26.298715+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:26.298715+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9257,"log_dups_size":0,"ondisk_log_size":9257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835590,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9256,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1055}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9419","reported_seq":9551,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.298606+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:27.298606+0000","last_peered":"2024-09-16T04:48:27.298606+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:27.298606+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9419,"log_dups_size":0,"ondisk_log_size":9419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27546892,"num_objects":3363,"num_object_clones":0,"num_object_copies":13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9417,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1165}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9112","reported_seq":9215,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:26.563844+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:26.563844+0000","last_peered":"2024-09-16T04:48:26.563844+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:26.563844+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9112,"log_dups_size":0,"ondisk_log_size":9112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27333900,"num_objects":3337,"num_object_clones":0,"num_object_copies":13348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9110,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1214}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9169","reported_seq":9263,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:27.259540+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:27.259540+0000","last_peered":"2024-09-16T04:48:27.259540+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:27.259540+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9169,"log_dups_size":0,"ondisk_log_size":9169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25703692,"num_objects":3138,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9167,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":910}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":430425792,"num_objects":52548,"num_object_clones":0,"num_object_copies":210192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52548,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148939,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5320,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":859959296,"data_stored":859959296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148966,"ondisk_log_size":148966,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720092,"kb_used_data":210828,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93651748,"statfs":{"total":96636764160,"available":95899389952,"internally_reserved":0,"allocated":215887872,"data_stored":215698670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521465551},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,0,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716392,"kb_used_data":211796,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93655448,"statfs":{"total":96636764160,"available":95903178752,"internally_reserved":0,"allocated":216879104,"data_stored":216689902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628336,"kb_used_data":209312,"kb_used_omap":4,"kb_used_meta":419003,"kb_avail":93743504,"statfs":{"total":96636764160,"available":95993348096,"internally_reserved":0,"allocated":214335488,"data_stored":214146286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":429059791},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,2,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707752,"kb_used_data":211008,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93664088,"statfs":{"total":96636764160,"available":95912026112,"internally_reserved":0,"allocated":216072192,"data_stored":215882990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":508620494},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,0,1,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":215195648,"data_stored":215195648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213893120,"data_stored":213893120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":215801856,"data_stored":215801856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":215068672,"data_stored":215068672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:34.538 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:35.141 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:35.142 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:35.165 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-09-16T04:48:34.708780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":422241984,"num_objects":51549,"num_object_clones":0,"num_object_copies":206196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51549,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149938,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5375,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149965,"ondisk_log_size":149965,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2740092,"kb_used_data":827108,"kb_used_omap":14,"kb_used_meta":1912881,"kb_avail":374747268,"statfs":{"total":386547056640,"available":383741202432,"internally_reserved":0,"allocated":846958592,"data_stored":846201784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1958790705},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,8,6,21,23,15,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17096704,"num_objects":-2087,"num_object_clones":0,"num_object_copies":-8348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2087,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005278"},"pg_stats":[{"pgid":"1.f","version":"35'9403","reported_seq":9519,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.576203+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:32.576203+0000","last_peered":"2024-09-16T04:48:32.576203+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:32.576203+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9403,"log_dups_size":0,"ondisk_log_size":9403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26632192,"num_objects":3251,"num_object_clones":0,"num_object_copies":13004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9403,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1065}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9393","reported_seq":9481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.327916+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:32.327916+0000","last_peered":"2024-09-16T04:48:32.327916+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:32.327916+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9393,"log_dups_size":0,"ondisk_log_size":9393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26875148,"num_objects":3281,"num_object_clones":0,"num_object_copies":13124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9391,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1078}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9424","reported_seq":9522,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:31.482739+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:31.482739+0000","last_peered":"2024-09-16T04:48:31.482739+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:31.482739+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9424,"log_dups_size":0,"ondisk_log_size":9424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26589854,"num_objects":3247,"num_object_clones":0,"num_object_copies":12988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3247,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9423,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1028}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9431","reported_seq":10524,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.329834+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:32.329834+0000","last_peered":"2024-09-16T04:48:32.329834+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:32.329834+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26531084,"num_objects":3239,"num_object_clones":0,"num_object_copies":12956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9429,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1518}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9493","reported_seq":9625,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.442165+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:32.442165+0000","last_peered":"2024-09-16T04:48:32.442165+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:32.442165+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9493,"log_dups_size":0,"ondisk_log_size":9493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26604812,"num_objects":3248,"num_object_clones":0,"num_object_copies":12992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9491,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9172","reported_seq":9281,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.510495+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:32.510495+0000","last_peered":"2024-09-16T04:48:32.510495+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:32.510495+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9172,"log_dups_size":0,"ondisk_log_size":9172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25722880,"num_objects":3140,"num_object_clones":0,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9172,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":962}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9406","reported_seq":9503,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.267402+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:32.267402+0000","last_peered":"2024-09-16T04:48:32.267402+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:32.267402+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9406,"log_dups_size":0,"ondisk_log_size":9406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26162444,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9404,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":981}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9541","reported_seq":14103,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:34.480214+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:34.480214+0000","last_peered":"2024-09-16T04:48:34.480214+0000","last_clean":"2024-09-16T04:48:34.480214+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:34.480214+0000","last_undegraded":"2024-09-16T04:48:34.480214+0000","last_fullsized":"2024-09-16T04:48:34.480214+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9541,"log_dups_size":0,"ondisk_log_size":9541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:37.320756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26572044,"num_objects":3244,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9539,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9562","reported_seq":14318,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:32.555569+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:32.555569+0000","last_peered":"2024-09-16T04:48:32.555569+0000","last_clean":"2024-09-16T04:48:32.555569+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:32.555569+0000","last_undegraded":"2024-09-16T04:48:32.555569+0000","last_fullsized":"2024-09-16T04:48:32.555569+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9562,"log_dups_size":0,"ondisk_log_size":9562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26159640,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9559,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9453","reported_seq":9557,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:31.642373+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:31.642373+0000","last_peered":"2024-09-16T04:48:31.642373+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:31.642373+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9453,"log_dups_size":0,"ondisk_log_size":9453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27071756,"num_objects":3305,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9451,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9328","reported_seq":9460,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.525089+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:32.525089+0000","last_peered":"2024-09-16T04:48:32.525089+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:32.525089+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9328,"log_dups_size":0,"ondisk_log_size":9328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25907090,"num_objects":3163,"num_object_clones":0,"num_object_copies":12652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9325,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9157","reported_seq":9249,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.523537+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:32.523537+0000","last_peered":"2024-09-16T04:48:32.523537+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:32.523537+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9157,"log_dups_size":0,"ondisk_log_size":9157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26000006,"num_objects":3174,"num_object_clones":0,"num_object_copies":12696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9156,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1069}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9310","reported_seq":9397,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:31.638158+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:31.638158+0000","last_peered":"2024-09-16T04:48:31.638158+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:31.638158+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9310,"log_dups_size":0,"ondisk_log_size":9310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26401414,"num_objects":3223,"num_object_clones":0,"num_object_copies":12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9309,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1002}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9479","reported_seq":9611,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.510624+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:32.510624+0000","last_peered":"2024-09-16T04:48:32.510624+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:32.510624+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9479,"log_dups_size":0,"ondisk_log_size":9479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27055372,"num_objects":3303,"num_object_clones":0,"num_object_copies":13212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9477,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9186","reported_seq":9288,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:31.613820+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:31.613820+0000","last_peered":"2024-09-16T04:48:31.613820+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:31.613820+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9186,"log_dups_size":0,"ondisk_log_size":9186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26727692,"num_objects":3263,"num_object_clones":0,"num_object_copies":13052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9184,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1140}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9227","reported_seq":9320,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.210163+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:32.210163+0000","last_peered":"2024-09-16T04:48:32.210163+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:32.210163+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9227,"log_dups_size":0,"ondisk_log_size":9227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228556,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9225,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":852}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":422241984,"num_objects":51549,"num_object_clones":0,"num_object_copies":206196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51549,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149938,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5375,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843743232,"data_stored":843743232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149965,"ondisk_log_size":149965,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703744,"kb_used_data":206940,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93668096,"statfs":{"total":96636764160,"available":95916130304,"internally_reserved":0,"allocated":211906560,"data_stored":211717358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508686031},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,13,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716392,"kb_used_data":207640,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93655448,"statfs":{"total":96636764160,"available":95903178752,"internally_reserved":0,"allocated":212623360,"data_stored":212434158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,11,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611968,"kb_used_data":205420,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93759872,"statfs":{"total":96636764160,"available":96010108928,"internally_reserved":0,"allocated":210350080,"data_stored":210160878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":416280271},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707988,"kb_used_data":207108,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93663852,"statfs":{"total":96636764160,"available":95911784448,"internally_reserved":0,"allocated":212078592,"data_stored":211889390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":512880334},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,5,5,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211197952,"data_stored":211197952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":209846272,"data_stored":209846272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":211742720,"data_stored":211742720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":210956288,"data_stored":210956288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:35.166 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:35.796 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:35.797 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:35.821 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-09-16T04:48:34.708780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":422241984,"num_objects":51549,"num_object_clones":0,"num_object_copies":206196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51549,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149938,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5375,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149965,"ondisk_log_size":149965,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2740092,"kb_used_data":827108,"kb_used_omap":14,"kb_used_meta":1912881,"kb_avail":374747268,"statfs":{"total":386547056640,"available":383741202432,"internally_reserved":0,"allocated":846958592,"data_stored":846201784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1958790705},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,8,6,21,23,15,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17096704,"num_objects":-2087,"num_object_clones":0,"num_object_copies":-8348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2087,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005278"},"pg_stats":[{"pgid":"1.f","version":"35'9403","reported_seq":9519,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.576203+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:32.576203+0000","last_peered":"2024-09-16T04:48:32.576203+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:32.576203+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9403,"log_dups_size":0,"ondisk_log_size":9403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26632192,"num_objects":3251,"num_object_clones":0,"num_object_copies":13004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9403,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1065}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9393","reported_seq":9481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.327916+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:32.327916+0000","last_peered":"2024-09-16T04:48:32.327916+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:32.327916+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9393,"log_dups_size":0,"ondisk_log_size":9393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26875148,"num_objects":3281,"num_object_clones":0,"num_object_copies":13124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9391,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1078}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9424","reported_seq":9522,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:31.482739+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:31.482739+0000","last_peered":"2024-09-16T04:48:31.482739+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:31.482739+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9424,"log_dups_size":0,"ondisk_log_size":9424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26589854,"num_objects":3247,"num_object_clones":0,"num_object_copies":12988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3247,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9423,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1028}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9431","reported_seq":10524,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.329834+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:32.329834+0000","last_peered":"2024-09-16T04:48:32.329834+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:32.329834+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26531084,"num_objects":3239,"num_object_clones":0,"num_object_copies":12956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9429,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1518}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9493","reported_seq":9625,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.442165+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:32.442165+0000","last_peered":"2024-09-16T04:48:32.442165+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:32.442165+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9493,"log_dups_size":0,"ondisk_log_size":9493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26604812,"num_objects":3248,"num_object_clones":0,"num_object_copies":12992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9491,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9172","reported_seq":9281,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.510495+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:32.510495+0000","last_peered":"2024-09-16T04:48:32.510495+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:32.510495+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9172,"log_dups_size":0,"ondisk_log_size":9172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25722880,"num_objects":3140,"num_object_clones":0,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9172,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":962}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9406","reported_seq":9503,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.267402+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:32.267402+0000","last_peered":"2024-09-16T04:48:32.267402+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:32.267402+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9406,"log_dups_size":0,"ondisk_log_size":9406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26162444,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9404,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":981}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9541","reported_seq":14103,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:34.480214+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:34.480214+0000","last_peered":"2024-09-16T04:48:34.480214+0000","last_clean":"2024-09-16T04:48:34.480214+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:34.480214+0000","last_undegraded":"2024-09-16T04:48:34.480214+0000","last_fullsized":"2024-09-16T04:48:34.480214+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9541,"log_dups_size":0,"ondisk_log_size":9541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:37.320756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26572044,"num_objects":3244,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9539,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9562","reported_seq":14318,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:32.555569+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:32.555569+0000","last_peered":"2024-09-16T04:48:32.555569+0000","last_clean":"2024-09-16T04:48:32.555569+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:32.555569+0000","last_undegraded":"2024-09-16T04:48:32.555569+0000","last_fullsized":"2024-09-16T04:48:32.555569+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9562,"log_dups_size":0,"ondisk_log_size":9562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26159640,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9559,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9453","reported_seq":9557,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:31.642373+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:31.642373+0000","last_peered":"2024-09-16T04:48:31.642373+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:31.642373+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9453,"log_dups_size":0,"ondisk_log_size":9453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27071756,"num_objects":3305,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9451,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9328","reported_seq":9460,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.525089+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:32.525089+0000","last_peered":"2024-09-16T04:48:32.525089+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:32.525089+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9328,"log_dups_size":0,"ondisk_log_size":9328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25907090,"num_objects":3163,"num_object_clones":0,"num_object_copies":12652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9325,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9157","reported_seq":9249,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.523537+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:32.523537+0000","last_peered":"2024-09-16T04:48:32.523537+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:32.523537+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9157,"log_dups_size":0,"ondisk_log_size":9157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26000006,"num_objects":3174,"num_object_clones":0,"num_object_copies":12696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9156,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1069}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9310","reported_seq":9397,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:31.638158+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:31.638158+0000","last_peered":"2024-09-16T04:48:31.638158+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:31.638158+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9310,"log_dups_size":0,"ondisk_log_size":9310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26401414,"num_objects":3223,"num_object_clones":0,"num_object_copies":12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9309,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1002}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9479","reported_seq":9611,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.510624+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:32.510624+0000","last_peered":"2024-09-16T04:48:32.510624+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:32.510624+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9479,"log_dups_size":0,"ondisk_log_size":9479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27055372,"num_objects":3303,"num_object_clones":0,"num_object_copies":13212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9477,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9186","reported_seq":9288,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:31.613820+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:31.613820+0000","last_peered":"2024-09-16T04:48:31.613820+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:31.613820+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9186,"log_dups_size":0,"ondisk_log_size":9186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26727692,"num_objects":3263,"num_object_clones":0,"num_object_copies":13052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9184,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1140}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9227","reported_seq":9320,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.210163+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:32.210163+0000","last_peered":"2024-09-16T04:48:32.210163+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:32.210163+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9227,"log_dups_size":0,"ondisk_log_size":9227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228556,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9225,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":852}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":422241984,"num_objects":51549,"num_object_clones":0,"num_object_copies":206196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51549,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149938,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5375,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843743232,"data_stored":843743232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149965,"ondisk_log_size":149965,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703744,"kb_used_data":206940,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93668096,"statfs":{"total":96636764160,"available":95916130304,"internally_reserved":0,"allocated":211906560,"data_stored":211717358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508686031},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,13,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716392,"kb_used_data":207640,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93655448,"statfs":{"total":96636764160,"available":95903178752,"internally_reserved":0,"allocated":212623360,"data_stored":212434158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,11,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611968,"kb_used_data":205420,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93759872,"statfs":{"total":96636764160,"available":96010108928,"internally_reserved":0,"allocated":210350080,"data_stored":210160878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":416280271},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707988,"kb_used_data":207108,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93663852,"statfs":{"total":96636764160,"available":95911784448,"internally_reserved":0,"allocated":212078592,"data_stored":211889390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":512880334},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,5,5,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211197952,"data_stored":211197952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":209846272,"data_stored":209846272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":211742720,"data_stored":211742720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":210956288,"data_stored":210956288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:35.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:48:35.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9403", 'reported_seq': 9519, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:32.576203+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:48:32.576203+0000', 'last_peered': '2024-09-16T04:48:32.576203+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:48:32.576203+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9403, 'log_dups_size': 0, 'ondisk_log_size': 9403, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26632192, 'num_objects': 3251, 'num_object_clones': 0, 'num_object_copies': 13004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9403, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1065}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:35.823 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:48:35.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9393", 'reported_seq': 9481, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:32.327916+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:48:32.327916+0000', 'last_peered': '2024-09-16T04:48:32.327916+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:48:32.327916+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9393, 'log_dups_size': 0, 'ondisk_log_size': 9393, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26875148, 'num_objects': 3281, 'num_object_clones': 0, 'num_object_copies': 13124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9391, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1078}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:35.823 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:48:35.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9424", 'reported_seq': 9522, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:31.482739+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:48:31.482739+0000', 'last_peered': '2024-09-16T04:48:31.482739+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:48:31.482739+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9424, 'log_dups_size': 0, 'ondisk_log_size': 9424, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26589854, 'num_objects': 3247, 'num_object_clones': 0, 'num_object_copies': 12988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3247, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9423, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1028}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:35.823 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:48:35.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9431", 'reported_seq': 10524, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:32.329834+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:48:32.329834+0000', 'last_peered': '2024-09-16T04:48:32.329834+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:48:32.329834+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9431, 'log_dups_size': 0, 'ondisk_log_size': 9431, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26531084, 'num_objects': 3239, 'num_object_clones': 0, 'num_object_copies': 12956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1721, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3239, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9429, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 512, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1518}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:35.823 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:48:35.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9493", 'reported_seq': 9625, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:32.442165+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:48:32.442165+0000', 'last_peered': '2024-09-16T04:48:32.442165+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:48:32.442165+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9493, 'log_dups_size': 0, 'ondisk_log_size': 9493, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26604812, 'num_objects': 3248, 'num_object_clones': 0, 'num_object_copies': 12992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3248, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9491, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1000}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:35.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:48:35.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9172", 'reported_seq': 9281, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:32.510495+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:48:32.510495+0000', 'last_peered': '2024-09-16T04:48:32.510495+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:48:32.510495+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9172, 'log_dups_size': 0, 'ondisk_log_size': 9172, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25722880, 'num_objects': 3140, 'num_object_clones': 0, 'num_object_copies': 12560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3140, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9172, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 962}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:35.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:48:35.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9406", 'reported_seq': 9503, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:32.267402+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:48:32.267402+0000', 'last_peered': '2024-09-16T04:48:32.267402+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:48:32.267402+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9406, 'log_dups_size': 0, 'ondisk_log_size': 9406, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26162444, 'num_objects': 3194, 'num_object_clones': 0, 'num_object_copies': 12776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9404, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 981}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:35.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:48:35.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9453", 'reported_seq': 9557, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:31.642373+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:48:31.642373+0000', 'last_peered': '2024-09-16T04:48:31.642373+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:48:31.642373+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9453, 'log_dups_size': 0, 'ondisk_log_size': 9453, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27071756, 'num_objects': 3305, 'num_object_clones': 0, 'num_object_copies': 13220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9451, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1116}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:35.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:48:35.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9328", 'reported_seq': 9460, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:32.525089+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:48:32.525089+0000', 'last_peered': '2024-09-16T04:48:32.525089+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:48:32.525089+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9328, 'log_dups_size': 0, 'ondisk_log_size': 9328, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25907090, 'num_objects': 3163, 'num_object_clones': 0, 'num_object_copies': 12652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9325, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1008}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:35.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:48:35.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9157", 'reported_seq': 9249, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:32.523537+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:48:32.523537+0000', 'last_peered': '2024-09-16T04:48:32.523537+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:48:32.523537+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9157, 'log_dups_size': 0, 'ondisk_log_size': 9157, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26000006, 'num_objects': 3174, 'num_object_clones': 0, 'num_object_copies': 12696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9156, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1069}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:35.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:48:35.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9310", 'reported_seq': 9397, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:31.638158+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:48:31.638158+0000', 'last_peered': '2024-09-16T04:48:31.638158+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:48:31.638158+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9310, 'log_dups_size': 0, 'ondisk_log_size': 9310, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26401414, 'num_objects': 3223, 'num_object_clones': 0, 'num_object_copies': 12892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3223, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9309, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1002}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:35.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:48:35.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9479", 'reported_seq': 9611, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:32.510624+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:48:32.510624+0000', 'last_peered': '2024-09-16T04:48:32.510624+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:48:32.510624+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9479, 'log_dups_size': 0, 'ondisk_log_size': 9479, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27055372, 'num_objects': 3303, 'num_object_clones': 0, 'num_object_copies': 13212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3303, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9477, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1105}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:35.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:48:35.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9186", 'reported_seq': 9288, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:31.613820+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:48:31.613820+0000', 'last_peered': '2024-09-16T04:48:31.613820+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:48:31.613820+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9186, 'log_dups_size': 0, 'ondisk_log_size': 9186, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26727692, 'num_objects': 3263, 'num_object_clones': 0, 'num_object_copies': 13052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9184, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1140}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:35.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:48:35.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9227", 'reported_seq': 9320, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:32.210163+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:48:32.210163+0000', 'last_peered': '2024-09-16T04:48:32.210163+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:48:32.210163+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9227, 'log_dups_size': 0, 'ondisk_log_size': 9227, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25228556, 'num_objects': 3080, 'num_object_clones': 0, 'num_object_copies': 12320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9225, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 852}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:35.825 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:48:36.522 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:36.522 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30207/206196 objects degraded (14.650%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":528,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":51549,"data_bytes":422241984,"bytes_used":2805854208,"bytes_avail":383741202432,"bytes_total":386547056640,"degraded_objects":30207,"degraded_total":206196,"degraded_ratio":0.14649653727521386,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":173},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:48:36.560 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:48:36.560 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:37.193 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:37.194 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:37.227 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-16T04:48:36.709257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":422241984,"num_objects":51549,"num_object_clones":0,"num_object_copies":206196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51549,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149938,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5375,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149965,"ondisk_log_size":149965,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2740092,"kb_used_data":827108,"kb_used_omap":14,"kb_used_meta":1912881,"kb_avail":374747268,"statfs":{"total":386547056640,"available":383741202432,"internally_reserved":0,"allocated":846958592,"data_stored":846201784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1958790705},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,8,6,21,23,15,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16547840,"num_objects":-2020,"num_object_clones":0,"num_object_copies":-8080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2020,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005218"},"pg_stats":[{"pgid":"1.f","version":"35'9403","reported_seq":9519,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.576203+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:32.576203+0000","last_peered":"2024-09-16T04:48:32.576203+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:32.576203+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9403,"log_dups_size":0,"ondisk_log_size":9403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26632192,"num_objects":3251,"num_object_clones":0,"num_object_copies":13004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9403,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1065}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9393","reported_seq":9481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.327916+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:32.327916+0000","last_peered":"2024-09-16T04:48:32.327916+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:32.327916+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9393,"log_dups_size":0,"ondisk_log_size":9393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26875148,"num_objects":3281,"num_object_clones":0,"num_object_copies":13124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9391,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1078}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9424","reported_seq":9522,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:31.482739+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:31.482739+0000","last_peered":"2024-09-16T04:48:31.482739+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:31.482739+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9424,"log_dups_size":0,"ondisk_log_size":9424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26589854,"num_objects":3247,"num_object_clones":0,"num_object_copies":12988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3247,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9423,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1028}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9431","reported_seq":10524,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.329834+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:32.329834+0000","last_peered":"2024-09-16T04:48:32.329834+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:32.329834+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26531084,"num_objects":3239,"num_object_clones":0,"num_object_copies":12956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9429,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1518}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9493","reported_seq":9625,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.442165+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:32.442165+0000","last_peered":"2024-09-16T04:48:32.442165+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:32.442165+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9493,"log_dups_size":0,"ondisk_log_size":9493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26604812,"num_objects":3248,"num_object_clones":0,"num_object_copies":12992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9491,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9172","reported_seq":9281,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.510495+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:32.510495+0000","last_peered":"2024-09-16T04:48:32.510495+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:32.510495+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9172,"log_dups_size":0,"ondisk_log_size":9172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25722880,"num_objects":3140,"num_object_clones":0,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9172,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":962}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9406","reported_seq":9503,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.267402+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:32.267402+0000","last_peered":"2024-09-16T04:48:32.267402+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:32.267402+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9406,"log_dups_size":0,"ondisk_log_size":9406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26162444,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9404,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":981}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9541","reported_seq":14103,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:34.480214+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:34.480214+0000","last_peered":"2024-09-16T04:48:34.480214+0000","last_clean":"2024-09-16T04:48:34.480214+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:34.480214+0000","last_undegraded":"2024-09-16T04:48:34.480214+0000","last_fullsized":"2024-09-16T04:48:34.480214+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9541,"log_dups_size":0,"ondisk_log_size":9541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:37.320756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26572044,"num_objects":3244,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9539,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9562","reported_seq":14318,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:32.555569+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:32.555569+0000","last_peered":"2024-09-16T04:48:32.555569+0000","last_clean":"2024-09-16T04:48:32.555569+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:32.555569+0000","last_undegraded":"2024-09-16T04:48:32.555569+0000","last_fullsized":"2024-09-16T04:48:32.555569+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9562,"log_dups_size":0,"ondisk_log_size":9562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26159640,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9559,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9453","reported_seq":9557,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:31.642373+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:31.642373+0000","last_peered":"2024-09-16T04:48:31.642373+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:31.642373+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9453,"log_dups_size":0,"ondisk_log_size":9453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27071756,"num_objects":3305,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9451,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1116}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9328","reported_seq":9460,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.525089+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:32.525089+0000","last_peered":"2024-09-16T04:48:32.525089+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:32.525089+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9328,"log_dups_size":0,"ondisk_log_size":9328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25907090,"num_objects":3163,"num_object_clones":0,"num_object_copies":12652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9325,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9157","reported_seq":9249,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.523537+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:32.523537+0000","last_peered":"2024-09-16T04:48:32.523537+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:32.523537+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9157,"log_dups_size":0,"ondisk_log_size":9157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26000006,"num_objects":3174,"num_object_clones":0,"num_object_copies":12696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9156,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1069}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9310","reported_seq":9397,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:31.638158+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:31.638158+0000","last_peered":"2024-09-16T04:48:31.638158+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:31.638158+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9310,"log_dups_size":0,"ondisk_log_size":9310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26401414,"num_objects":3223,"num_object_clones":0,"num_object_copies":12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9309,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1002}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9479","reported_seq":9611,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.510624+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:32.510624+0000","last_peered":"2024-09-16T04:48:32.510624+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:32.510624+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9479,"log_dups_size":0,"ondisk_log_size":9479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27055372,"num_objects":3303,"num_object_clones":0,"num_object_copies":13212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9477,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9186","reported_seq":9288,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:31.613820+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:31.613820+0000","last_peered":"2024-09-16T04:48:31.613820+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:31.613820+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9186,"log_dups_size":0,"ondisk_log_size":9186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26727692,"num_objects":3263,"num_object_clones":0,"num_object_copies":13052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9184,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1140}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9227","reported_seq":9320,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:32.210163+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:32.210163+0000","last_peered":"2024-09-16T04:48:32.210163+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:32.210163+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9227,"log_dups_size":0,"ondisk_log_size":9227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228556,"num_objects":3080,"num_object_clones":0,"num_object_copies":12320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9225,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":852}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":422241984,"num_objects":51549,"num_object_clones":0,"num_object_copies":206196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51549,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149938,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5375,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843743232,"data_stored":843743232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149965,"ondisk_log_size":149965,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703744,"kb_used_data":206940,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93668096,"statfs":{"total":96636764160,"available":95916130304,"internally_reserved":0,"allocated":211906560,"data_stored":211717358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508686031},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,13,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716392,"kb_used_data":207640,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93655448,"statfs":{"total":96636764160,"available":95903178752,"internally_reserved":0,"allocated":212623360,"data_stored":212434158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,11,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611968,"kb_used_data":205420,"kb_used_omap":4,"kb_used_meta":406523,"kb_avail":93759872,"statfs":{"total":96636764160,"available":96010108928,"internally_reserved":0,"allocated":210350080,"data_stored":210160878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":416280271},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707988,"kb_used_data":207108,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93663852,"statfs":{"total":96636764160,"available":95911784448,"internally_reserved":0,"allocated":212078592,"data_stored":211889390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":512880334},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,5,5,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211197952,"data_stored":211197952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":209846272,"data_stored":209846272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":211742720,"data_stored":211742720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":210956288,"data_stored":210956288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:40.229 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:40.796 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:40.796 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:40.821 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-09-16T04:48:40.710256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":413755072,"num_objects":50513,"num_object_clones":0,"num_object_copies":202052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150974,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5426,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151001,"ondisk_log_size":151001,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2736408,"kb_used_data":810956,"kb_used_omap":14,"kb_used_meta":1925361,"kb_avail":374750952,"statfs":{"total":386547056640,"available":383744974848,"internally_reserved":0,"allocated":830418944,"data_stored":829662136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1971570225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,10,0,5,9,16,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17268736,"num_objects":-2108,"num_object_clones":0,"num_object_copies":-8432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2108,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005215"},"pg_stats":[{"pgid":"1.f","version":"35'9471","reported_seq":9587,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.599812+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:37.599812+0000","last_peered":"2024-09-16T04:48:37.599812+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:37.599812+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9471,"log_dups_size":0,"ondisk_log_size":9471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26075136,"num_objects":3183,"num_object_clones":0,"num_object_copies":12732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9471,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9468","reported_seq":9555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.431429+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:37.431429+0000","last_peered":"2024-09-16T04:48:37.431429+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:37.431429+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9468,"log_dups_size":0,"ondisk_log_size":9468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26260748,"num_objects":3206,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9466,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1003}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9482","reported_seq":9580,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:36.519906+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:36.519906+0000","last_peered":"2024-09-16T04:48:36.519906+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:36.519906+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9482,"log_dups_size":0,"ondisk_log_size":9482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26114718,"num_objects":3189,"num_object_clones":0,"num_object_copies":12756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3189,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9481,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":970}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9506","reported_seq":10701,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.415401+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:37.415401+0000","last_peered":"2024-09-16T04:48:37.415401+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:37.415401+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9506,"log_dups_size":0,"ondisk_log_size":9506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25916684,"num_objects":3164,"num_object_clones":0,"num_object_copies":12656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9504,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1494}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9556","reported_seq":9688,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.603145+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:37.603145+0000","last_peered":"2024-09-16T04:48:37.603145+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:37.603145+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9556,"log_dups_size":0,"ondisk_log_size":9556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26088716,"num_objects":3185,"num_object_clones":0,"num_object_copies":12740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9554,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":937}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9226","reported_seq":9336,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.583799+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:37.583799+0000","last_peered":"2024-09-16T04:48:37.583799+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:37.583799+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9226,"log_dups_size":0,"ondisk_log_size":9226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25280512,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":908}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9472","reported_seq":9569,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.334787+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:37.334787+0000","last_peered":"2024-09-16T04:48:37.334787+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:37.334787+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9472,"log_dups_size":0,"ondisk_log_size":9472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25621772,"num_objects":3128,"num_object_clones":0,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9470,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":915}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9602","reported_seq":14164,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:39.543182+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:39.543182+0000","last_peered":"2024-09-16T04:48:39.543182+0000","last_clean":"2024-09-16T04:48:39.543182+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:39.543182+0000","last_undegraded":"2024-09-16T04:48:39.543182+0000","last_fullsized":"2024-09-16T04:48:39.543182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9602,"log_dups_size":0,"ondisk_log_size":9602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:42.433065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26072332,"num_objects":3183,"num_object_clones":0,"num_object_copies":12732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9600,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9640","reported_seq":14397,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:37.449994+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:37.449994+0000","last_peered":"2024-09-16T04:48:37.449994+0000","last_clean":"2024-09-16T04:48:37.449994+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:37.449994+0000","last_undegraded":"2024-09-16T04:48:37.449994+0000","last_fullsized":"2024-09-16T04:48:37.449994+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9640,"log_dups_size":0,"ondisk_log_size":9640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:37.841614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25520664,"num_objects":3116,"num_object_clones":0,"num_object_copies":12464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9637,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9514","reported_seq":9621,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:36.638387+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:36.638387+0000","last_peered":"2024-09-16T04:48:36.638387+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:36.638387+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9514,"log_dups_size":0,"ondisk_log_size":9514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26572044,"num_objects":3244,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9512,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1055}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9405","reported_seq":9535,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.658468+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:37.658468+0000","last_peered":"2024-09-16T04:48:37.658468+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:37.658468+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9405,"log_dups_size":0,"ondisk_log_size":9405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25276306,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9402,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9218","reported_seq":9310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.455699+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:37.455699+0000","last_peered":"2024-09-16T04:48:37.455699+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:37.455699+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9218,"log_dups_size":0,"ondisk_log_size":9218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25500294,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9217,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1008}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9374","reported_seq":9462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:36.612404+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:36.612404+0000","last_peered":"2024-09-16T04:48:36.612404+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:36.612404+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9374,"log_dups_size":0,"ondisk_log_size":9374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25877126,"num_objects":3159,"num_object_clones":0,"num_object_copies":12636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9373,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":938}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9546","reported_seq":9678,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.470584+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:37.470584+0000","last_peered":"2024-09-16T04:48:37.470584+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:37.470584+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9546,"log_dups_size":0,"ondisk_log_size":9546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26506508,"num_objects":3236,"num_object_clones":0,"num_object_copies":12944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9544,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9243","reported_seq":9345,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:36.505296+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:36.505296+0000","last_peered":"2024-09-16T04:48:36.505296+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:36.505296+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9243,"log_dups_size":0,"ondisk_log_size":9243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26260748,"num_objects":3206,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9241,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1083}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9278","reported_seq":9369,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.313999+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:37.313999+0000","last_peered":"2024-09-16T04:48:37.313999+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:37.313999+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24810764,"num_objects":3029,"num_object_clones":0,"num_object_copies":12116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":801}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":413755072,"num_objects":50513,"num_object_clones":0,"num_object_copies":202052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150974,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5426,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":826941440,"data_stored":826941440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151001,"ondisk_log_size":151001,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703948,"kb_used_data":203004,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93667892,"statfs":{"total":96636764160,"available":95915921408,"internally_reserved":0,"allocated":207876096,"data_stored":207686894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512945871},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,0,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712368,"kb_used_data":203604,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93659472,"statfs":{"total":96636764160,"available":95907299328,"internally_reserved":0,"allocated":208490496,"data_stored":208301294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,0,3,0,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611964,"kb_used_data":201260,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93759876,"statfs":{"total":96636764160,"available":96010113024,"internally_reserved":0,"allocated":206090240,"data_stored":205901038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420540111},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708128,"kb_used_data":203088,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93663712,"statfs":{"total":96636764160,"available":95911641088,"internally_reserved":0,"allocated":207962112,"data_stored":207772910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":206974976,"data_stored":206974976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":205627392,"data_stored":205627392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207585280,"data_stored":207585280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":206753792,"data_stored":206753792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:40.822 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:41.451 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:41.451 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:41.475 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-09-16T04:48:40.710256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":413755072,"num_objects":50513,"num_object_clones":0,"num_object_copies":202052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150974,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5426,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151001,"ondisk_log_size":151001,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2736408,"kb_used_data":810956,"kb_used_omap":14,"kb_used_meta":1925361,"kb_avail":374750952,"statfs":{"total":386547056640,"available":383744974848,"internally_reserved":0,"allocated":830418944,"data_stored":829662136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":1971570225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,10,0,5,9,16,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17268736,"num_objects":-2108,"num_object_clones":0,"num_object_copies":-8432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2108,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005215"},"pg_stats":[{"pgid":"1.f","version":"35'9471","reported_seq":9587,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.599812+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:37.599812+0000","last_peered":"2024-09-16T04:48:37.599812+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:37.599812+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9471,"log_dups_size":0,"ondisk_log_size":9471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26075136,"num_objects":3183,"num_object_clones":0,"num_object_copies":12732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9471,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9468","reported_seq":9555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.431429+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:37.431429+0000","last_peered":"2024-09-16T04:48:37.431429+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:37.431429+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9468,"log_dups_size":0,"ondisk_log_size":9468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26260748,"num_objects":3206,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9466,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1003}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9482","reported_seq":9580,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:36.519906+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:36.519906+0000","last_peered":"2024-09-16T04:48:36.519906+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:36.519906+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9482,"log_dups_size":0,"ondisk_log_size":9482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26114718,"num_objects":3189,"num_object_clones":0,"num_object_copies":12756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3189,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9481,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":970}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9506","reported_seq":10701,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.415401+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:37.415401+0000","last_peered":"2024-09-16T04:48:37.415401+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:37.415401+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9506,"log_dups_size":0,"ondisk_log_size":9506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25916684,"num_objects":3164,"num_object_clones":0,"num_object_copies":12656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9504,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1494}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9556","reported_seq":9688,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.603145+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:37.603145+0000","last_peered":"2024-09-16T04:48:37.603145+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:37.603145+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9556,"log_dups_size":0,"ondisk_log_size":9556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26088716,"num_objects":3185,"num_object_clones":0,"num_object_copies":12740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9554,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":937}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9226","reported_seq":9336,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.583799+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:37.583799+0000","last_peered":"2024-09-16T04:48:37.583799+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:37.583799+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9226,"log_dups_size":0,"ondisk_log_size":9226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25280512,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":908}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9472","reported_seq":9569,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.334787+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:37.334787+0000","last_peered":"2024-09-16T04:48:37.334787+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:37.334787+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9472,"log_dups_size":0,"ondisk_log_size":9472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25621772,"num_objects":3128,"num_object_clones":0,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9470,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":915}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9602","reported_seq":14164,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:39.543182+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:39.543182+0000","last_peered":"2024-09-16T04:48:39.543182+0000","last_clean":"2024-09-16T04:48:39.543182+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:39.543182+0000","last_undegraded":"2024-09-16T04:48:39.543182+0000","last_fullsized":"2024-09-16T04:48:39.543182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9602,"log_dups_size":0,"ondisk_log_size":9602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:42.433065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26072332,"num_objects":3183,"num_object_clones":0,"num_object_copies":12732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9600,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9640","reported_seq":14397,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:37.449994+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:37.449994+0000","last_peered":"2024-09-16T04:48:37.449994+0000","last_clean":"2024-09-16T04:48:37.449994+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:37.449994+0000","last_undegraded":"2024-09-16T04:48:37.449994+0000","last_fullsized":"2024-09-16T04:48:37.449994+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9640,"log_dups_size":0,"ondisk_log_size":9640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:37.841614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25520664,"num_objects":3116,"num_object_clones":0,"num_object_copies":12464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9637,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9514","reported_seq":9621,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:36.638387+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:36.638387+0000","last_peered":"2024-09-16T04:48:36.638387+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:36.638387+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9514,"log_dups_size":0,"ondisk_log_size":9514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26572044,"num_objects":3244,"num_object_clones":0,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9512,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1055}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9405","reported_seq":9535,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.658468+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:37.658468+0000","last_peered":"2024-09-16T04:48:37.658468+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:37.658468+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9405,"log_dups_size":0,"ondisk_log_size":9405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25276306,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9402,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9218","reported_seq":9310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.455699+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:37.455699+0000","last_peered":"2024-09-16T04:48:37.455699+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:37.455699+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9218,"log_dups_size":0,"ondisk_log_size":9218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25500294,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9217,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1008}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9374","reported_seq":9462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:36.612404+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:36.612404+0000","last_peered":"2024-09-16T04:48:36.612404+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:36.612404+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9374,"log_dups_size":0,"ondisk_log_size":9374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25877126,"num_objects":3159,"num_object_clones":0,"num_object_copies":12636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9373,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":938}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9546","reported_seq":9678,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.470584+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:37.470584+0000","last_peered":"2024-09-16T04:48:37.470584+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:37.470584+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9546,"log_dups_size":0,"ondisk_log_size":9546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26506508,"num_objects":3236,"num_object_clones":0,"num_object_copies":12944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9544,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9243","reported_seq":9345,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:36.505296+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:36.505296+0000","last_peered":"2024-09-16T04:48:36.505296+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:36.505296+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9243,"log_dups_size":0,"ondisk_log_size":9243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26260748,"num_objects":3206,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9241,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1083}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9278","reported_seq":9369,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.313999+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:37.313999+0000","last_peered":"2024-09-16T04:48:37.313999+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:37.313999+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24810764,"num_objects":3029,"num_object_clones":0,"num_object_copies":12116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":801}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":413755072,"num_objects":50513,"num_object_clones":0,"num_object_copies":202052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150974,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5426,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":826941440,"data_stored":826941440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151001,"ondisk_log_size":151001,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703948,"kb_used_data":203004,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93667892,"statfs":{"total":96636764160,"available":95915921408,"internally_reserved":0,"allocated":207876096,"data_stored":207686894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512945871},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,0,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712368,"kb_used_data":203604,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93659472,"statfs":{"total":96636764160,"available":95907299328,"internally_reserved":0,"allocated":208490496,"data_stored":208301294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,0,3,0,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611964,"kb_used_data":201260,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93759876,"statfs":{"total":96636764160,"available":96010113024,"internally_reserved":0,"allocated":206090240,"data_stored":205901038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420540111},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708128,"kb_used_data":203088,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93663712,"statfs":{"total":96636764160,"available":95911641088,"internally_reserved":0,"allocated":207962112,"data_stored":207772910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":206974976,"data_stored":206974976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":205627392,"data_stored":205627392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207585280,"data_stored":207585280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":206753792,"data_stored":206753792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:41.476 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:48:41.476 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9471", 'reported_seq': 9587, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:37.599812+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:48:37.599812+0000', 'last_peered': '2024-09-16T04:48:37.599812+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:48:37.599812+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9471, 'log_dups_size': 0, 'ondisk_log_size': 9471, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26075136, 'num_objects': 3183, 'num_object_clones': 0, 'num_object_copies': 12732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9471, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 997}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:41.476 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:48:41.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9468", 'reported_seq': 9555, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:37.431429+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:48:37.431429+0000', 'last_peered': '2024-09-16T04:48:37.431429+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:48:37.431429+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9468, 'log_dups_size': 0, 'ondisk_log_size': 9468, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26260748, 'num_objects': 3206, 'num_object_clones': 0, 'num_object_copies': 12824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9466, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1003}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:41.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:48:41.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9482", 'reported_seq': 9580, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:36.519906+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:48:36.519906+0000', 'last_peered': '2024-09-16T04:48:36.519906+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:48:36.519906+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9482, 'log_dups_size': 0, 'ondisk_log_size': 9482, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26114718, 'num_objects': 3189, 'num_object_clones': 0, 'num_object_copies': 12756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3189, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9481, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 970}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:41.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:48:41.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9506", 'reported_seq': 10701, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:37.415401+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:48:37.415401+0000', 'last_peered': '2024-09-16T04:48:37.415401+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:48:37.415401+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9506, 'log_dups_size': 0, 'ondisk_log_size': 9506, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25916684, 'num_objects': 3164, 'num_object_clones': 0, 'num_object_copies': 12656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1670, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3164, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9504, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 563, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1494}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:41.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:48:41.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9556", 'reported_seq': 9688, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:37.603145+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:48:37.603145+0000', 'last_peered': '2024-09-16T04:48:37.603145+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:48:37.603145+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9556, 'log_dups_size': 0, 'ondisk_log_size': 9556, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26088716, 'num_objects': 3185, 'num_object_clones': 0, 'num_object_copies': 12740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9554, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 937}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:41.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:48:41.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9226", 'reported_seq': 9336, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:37.583799+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:48:37.583799+0000', 'last_peered': '2024-09-16T04:48:37.583799+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:48:37.583799+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9226, 'log_dups_size': 0, 'ondisk_log_size': 9226, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25280512, 'num_objects': 3086, 'num_object_clones': 0, 'num_object_copies': 12344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9226, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 908}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:41.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:48:41.478 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9472", 'reported_seq': 9569, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:37.334787+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:48:37.334787+0000', 'last_peered': '2024-09-16T04:48:37.334787+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:48:37.334787+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9472, 'log_dups_size': 0, 'ondisk_log_size': 9472, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25621772, 'num_objects': 3128, 'num_object_clones': 0, 'num_object_copies': 12512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9470, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 915}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:41.478 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:48:41.478 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9514", 'reported_seq': 9621, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:36.638387+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:48:36.638387+0000', 'last_peered': '2024-09-16T04:48:36.638387+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:48:36.638387+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9514, 'log_dups_size': 0, 'ondisk_log_size': 9514, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26572044, 'num_objects': 3244, 'num_object_clones': 0, 'num_object_copies': 12976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3244, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9512, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1055}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:41.478 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:48:41.478 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9405", 'reported_seq': 9535, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:37.658468+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:48:37.658468+0000', 'last_peered': '2024-09-16T04:48:37.658468+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:48:37.658468+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9405, 'log_dups_size': 0, 'ondisk_log_size': 9405, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25276306, 'num_objects': 3086, 'num_object_clones': 0, 'num_object_copies': 12344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9402, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 931}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:41.478 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:48:41.478 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9218", 'reported_seq': 9310, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:37.455699+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:48:37.455699+0000', 'last_peered': '2024-09-16T04:48:37.455699+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:48:37.455699+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9218, 'log_dups_size': 0, 'ondisk_log_size': 9218, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25500294, 'num_objects': 3113, 'num_object_clones': 0, 'num_object_copies': 12452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9217, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1008}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:41.478 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:48:41.478 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9374", 'reported_seq': 9462, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:36.612404+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:48:36.612404+0000', 'last_peered': '2024-09-16T04:48:36.612404+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:48:36.612404+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9374, 'log_dups_size': 0, 'ondisk_log_size': 9374, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25877126, 'num_objects': 3159, 'num_object_clones': 0, 'num_object_copies': 12636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9373, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 938}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:41.478 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:48:41.478 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9546", 'reported_seq': 9678, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:37.470584+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:48:37.470584+0000', 'last_peered': '2024-09-16T04:48:37.470584+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:48:37.470584+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9546, 'log_dups_size': 0, 'ondisk_log_size': 9546, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26506508, 'num_objects': 3236, 'num_object_clones': 0, 'num_object_copies': 12944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9544, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1038}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:41.479 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:48:41.479 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9243", 'reported_seq': 9345, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:36.505296+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:48:36.505296+0000', 'last_peered': '2024-09-16T04:48:36.505296+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:48:36.505296+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9243, 'log_dups_size': 0, 'ondisk_log_size': 9243, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26260748, 'num_objects': 3206, 'num_object_clones': 0, 'num_object_copies': 12824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9241, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:41.479 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:48:41.479 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9278", 'reported_seq': 9369, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:37.313999+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:48:37.313999+0000', 'last_peered': '2024-09-16T04:48:37.313999+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:48:37.313999+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9278, 'log_dups_size': 0, 'ondisk_log_size': 9278, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24810764, 'num_objects': 3029, 'num_object_clones': 0, 'num_object_copies': 12116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3029, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9276, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 801}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:41.479 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:48:42.214 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:42.214 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30156/202052 objects degraded (14.925%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":534,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":50513,"data_bytes":413755072,"bytes_used":2802081792,"bytes_avail":383744974848,"bytes_total":386547056640,"degraded_objects":30156,"degraded_total":202052,"degraded_ratio":0.14924870825332093,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":175},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:48:42.245 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:48:42.245 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:42.860 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:42.860 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:42.887 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-16T04:48:42.710715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":409781952,"num_objects":50028,"num_object_clones":0,"num_object_copies":200112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50028,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5483,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151486,"ondisk_log_size":151486,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2755856,"kb_used_data":802632,"kb_used_omap":14,"kb_used_meta":1953137,"kb_avail":374731504,"statfs":{"total":386547056640,"available":383725060096,"internally_reserved":0,"allocated":821895168,"data_stored":821138360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2000012849},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,3,6,14,18,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20643840,"num_objects":-2520,"num_object_clones":0,"num_object_copies":-10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2520,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005155"},"pg_stats":[{"pgid":"1.f","version":"35'9471","reported_seq":9587,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.599812+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:37.599812+0000","last_peered":"2024-09-16T04:48:37.599812+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:37.599812+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9471,"log_dups_size":0,"ondisk_log_size":9471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26075136,"num_objects":3183,"num_object_clones":0,"num_object_copies":12732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9471,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9533","reported_seq":9621,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.233124+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:42.233124+0000","last_peered":"2024-09-16T04:48:42.233124+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:42.233124+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9533,"log_dups_size":0,"ondisk_log_size":9533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25728268,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9531,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":938}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9538","reported_seq":9636,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.730322+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:41.730322+0000","last_peered":"2024-09-16T04:48:41.730322+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:41.730322+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9538,"log_dups_size":0,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25655966,"num_objects":3133,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9537,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":914}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9567","reported_seq":10875,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.424899+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:42.424899+0000","last_peered":"2024-09-16T04:48:42.424899+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:42.424899+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9567,"log_dups_size":0,"ondisk_log_size":9567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25416972,"num_objects":3103,"num_object_clones":0,"num_object_copies":12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9565,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":620,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9556","reported_seq":9688,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.603145+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:37.603145+0000","last_peered":"2024-09-16T04:48:37.603145+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:37.603145+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9556,"log_dups_size":0,"ondisk_log_size":9556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26088716,"num_objects":3185,"num_object_clones":0,"num_object_copies":12740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9554,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":937}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9226","reported_seq":9336,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.583799+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:37.583799+0000","last_peered":"2024-09-16T04:48:37.583799+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:37.583799+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9226,"log_dups_size":0,"ondisk_log_size":9226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25280512,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":908}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9545","reported_seq":9642,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.342693+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:42.342693+0000","last_peered":"2024-09-16T04:48:42.342693+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:42.342693+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9545,"log_dups_size":0,"ondisk_log_size":9545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25023756,"num_objects":3055,"num_object_clones":0,"num_object_copies":12220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9543,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":842}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9602","reported_seq":14164,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:39.543182+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:39.543182+0000","last_peered":"2024-09-16T04:48:39.543182+0000","last_clean":"2024-09-16T04:48:39.543182+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:39.543182+0000","last_undegraded":"2024-09-16T04:48:39.543182+0000","last_fullsized":"2024-09-16T04:48:39.543182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9602,"log_dups_size":0,"ondisk_log_size":9602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:42.433065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26072332,"num_objects":3183,"num_object_clones":0,"num_object_copies":12732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9600,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9640","reported_seq":14397,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:37.449994+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:37.449994+0000","last_peered":"2024-09-16T04:48:37.449994+0000","last_clean":"2024-09-16T04:48:37.449994+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:37.449994+0000","last_undegraded":"2024-09-16T04:48:37.449994+0000","last_fullsized":"2024-09-16T04:48:37.449994+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9640,"log_dups_size":0,"ondisk_log_size":9640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:37.841614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25520664,"num_objects":3116,"num_object_clones":0,"num_object_copies":12464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9637,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9570","reported_seq":9677,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.439717+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:41.439717+0000","last_peered":"2024-09-16T04:48:41.439717+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:41.439717+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9570,"log_dups_size":0,"ondisk_log_size":9570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26113292,"num_objects":3188,"num_object_clones":0,"num_object_copies":12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9568,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":999}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9405","reported_seq":9535,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.658468+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:37.658468+0000","last_peered":"2024-09-16T04:48:37.658468+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:37.658468+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9405,"log_dups_size":0,"ondisk_log_size":9405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25276306,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9402,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9218","reported_seq":9310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.455699+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:37.455699+0000","last_peered":"2024-09-16T04:48:37.455699+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:37.455699+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9218,"log_dups_size":0,"ondisk_log_size":9218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25500294,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9217,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1008}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9434","reported_seq":9521,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.278655+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:41.278655+0000","last_peered":"2024-09-16T04:48:41.278655+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:41.278655+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9434,"log_dups_size":0,"ondisk_log_size":9434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25385606,"num_objects":3099,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9433,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":878}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9546","reported_seq":9678,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:37.470584+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:37.470584+0000","last_peered":"2024-09-16T04:48:37.470584+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:37.470584+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9546,"log_dups_size":0,"ondisk_log_size":9546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26506508,"num_objects":3236,"num_object_clones":0,"num_object_copies":12944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9544,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9306","reported_seq":9409,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.605004+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:41.605004+0000","last_peered":"2024-09-16T04:48:41.605004+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:41.605004+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9306,"log_dups_size":0,"ondisk_log_size":9306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25744652,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9304,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1020}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9329","reported_seq":9423,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.371802+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:42.371802+0000","last_peered":"2024-09-16T04:48:42.371802+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:42.371802+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24392972,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9327,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":750}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":409781952,"num_objects":50028,"num_object_clones":0,"num_object_copies":200112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50028,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5483,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":818417664,"data_stored":818417664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151486,"ondisk_log_size":151486,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703948,"kb_used_data":203004,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93667892,"statfs":{"total":96636764160,"available":95915921408,"internally_reserved":0,"allocated":207876096,"data_stored":207686894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512945871},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,0,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713752,"kb_used_data":199432,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93658088,"statfs":{"total":96636764160,"available":95905882112,"internally_reserved":0,"allocated":204218368,"data_stored":204029166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611964,"kb_used_data":201260,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93759876,"statfs":{"total":96636764160,"available":96010113024,"internally_reserved":0,"allocated":206090240,"data_stored":205901038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420540111},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247910,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726192,"kb_used_data":198936,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93645648,"statfs":{"total":96636764160,"available":95893143552,"internally_reserved":0,"allocated":203710464,"data_stored":203521262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":539881166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,3,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202760192,"data_stored":202760192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":205627392,"data_stored":205627392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203276288,"data_stored":203276288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":206753792,"data_stored":206753792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:45.889 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:46.490 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:46.490 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:46.518 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-09-16T04:48:44.711218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":405923520,"num_objects":49557,"num_object_clones":0,"num_object_copies":198228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49557,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151930,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5483,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151957,"ondisk_log_size":151957,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773848,"kb_used_data":798412,"kb_used_omap":14,"kb_used_meta":1975345,"kb_avail":374713512,"statfs":{"total":386547056640,"available":383706636288,"internally_reserved":0,"allocated":817573888,"data_stored":816817080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2022753841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,3,6,14,18,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16850944,"num_objects":-2057,"num_object_clones":0,"num_object_copies":-8228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2057,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005187"},"pg_stats":[{"pgid":"1.f","version":"35'9534","reported_seq":9650,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.471928+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:42.471928+0000","last_peered":"2024-09-16T04:48:42.471928+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:42.471928+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9534,"log_dups_size":0,"ondisk_log_size":9534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25559040,"num_objects":3120,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9534,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":934}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9533","reported_seq":9621,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.233124+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:42.233124+0000","last_peered":"2024-09-16T04:48:42.233124+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:42.233124+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9533,"log_dups_size":0,"ondisk_log_size":9533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25728268,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9531,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":938}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9538","reported_seq":9636,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.730322+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:41.730322+0000","last_peered":"2024-09-16T04:48:41.730322+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:41.730322+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9538,"log_dups_size":0,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25655966,"num_objects":3133,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9537,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":914}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9567","reported_seq":10875,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.424899+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:42.424899+0000","last_peered":"2024-09-16T04:48:42.424899+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:42.424899+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9567,"log_dups_size":0,"ondisk_log_size":9567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25416972,"num_objects":3103,"num_object_clones":0,"num_object_copies":12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9565,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":620,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9615","reported_seq":9747,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.637913+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:42.637913+0000","last_peered":"2024-09-16T04:48:42.637913+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:42.637913+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9615,"log_dups_size":0,"ondisk_log_size":9615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25605388,"num_objects":3126,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":878}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9292","reported_seq":9402,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.645889+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:42.645889+0000","last_peered":"2024-09-16T04:48:42.645889+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:42.645889+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9292,"log_dups_size":0,"ondisk_log_size":9292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24739840,"num_objects":3020,"num_object_clones":0,"num_object_copies":12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9545","reported_seq":9642,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.342693+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:42.342693+0000","last_peered":"2024-09-16T04:48:42.342693+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:42.342693+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9545,"log_dups_size":0,"ondisk_log_size":9545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25023756,"num_objects":3055,"num_object_clones":0,"num_object_copies":12220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9543,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":842}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9602","reported_seq":14164,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:39.543182+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:39.543182+0000","last_peered":"2024-09-16T04:48:39.543182+0000","last_clean":"2024-09-16T04:48:39.543182+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:39.543182+0000","last_undegraded":"2024-09-16T04:48:39.543182+0000","last_fullsized":"2024-09-16T04:48:39.543182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9602,"log_dups_size":0,"ondisk_log_size":9602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:42.433065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26072332,"num_objects":3183,"num_object_clones":0,"num_object_copies":12732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9600,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9707","reported_seq":14464,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:42.605903+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:42.605903+0000","last_peered":"2024-09-16T04:48:42.605903+0000","last_clean":"2024-09-16T04:48:42.605903+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:42.605903+0000","last_undegraded":"2024-09-16T04:48:42.605903+0000","last_fullsized":"2024-09-16T04:48:42.605903+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:43.814884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24971800,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9704,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9570","reported_seq":9677,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.439717+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:41.439717+0000","last_peered":"2024-09-16T04:48:41.439717+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:41.439717+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9570,"log_dups_size":0,"ondisk_log_size":9570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26113292,"num_objects":3188,"num_object_clones":0,"num_object_copies":12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9568,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":999}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9486","reported_seq":9616,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.736200+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:42.736200+0000","last_peered":"2024-09-16T04:48:42.736200+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:42.736200+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9486,"log_dups_size":0,"ondisk_log_size":9486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24612754,"num_objects":3005,"num_object_clones":0,"num_object_copies":12020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9483,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":850}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9282","reported_seq":9374,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.546736+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:42.546736+0000","last_peered":"2024-09-16T04:48:42.546736+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:42.546736+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24976006,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":944}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9434","reported_seq":9521,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.278655+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:41.278655+0000","last_peered":"2024-09-16T04:48:41.278655+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:41.278655+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9434,"log_dups_size":0,"ondisk_log_size":9434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25385606,"num_objects":3099,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9433,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":878}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9617","reported_seq":9749,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.265732+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:42.265732+0000","last_peered":"2024-09-16T04:48:42.265732+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:42.265732+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9617,"log_dups_size":0,"ondisk_log_size":9617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25924876,"num_objects":3165,"num_object_clones":0,"num_object_copies":12660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9615,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":967}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9306","reported_seq":9409,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.605004+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:41.605004+0000","last_peered":"2024-09-16T04:48:41.605004+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:41.605004+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9306,"log_dups_size":0,"ondisk_log_size":9306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25744652,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9304,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1020}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9329","reported_seq":9423,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.371802+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:42.371802+0000","last_peered":"2024-09-16T04:48:42.371802+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:42.371802+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24392972,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9327,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":750}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":405923520,"num_objects":49557,"num_object_clones":0,"num_object_copies":198228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49557,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151930,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5483,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":814202880,"data_stored":814202880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151957,"ondisk_log_size":151957,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721940,"kb_used_data":198784,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93649900,"statfs":{"total":96636764160,"available":95897497600,"internally_reserved":0,"allocated":203554816,"data_stored":203365614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":535686863},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,0,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713752,"kb_used_data":199432,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93658088,"statfs":{"total":96636764160,"available":95905882112,"internally_reserved":0,"allocated":204218368,"data_stored":204029166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611964,"kb_used_data":201260,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93759876,"statfs":{"total":96636764160,"available":96010113024,"internally_reserved":0,"allocated":206090240,"data_stored":205901038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420540111},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247910,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726192,"kb_used_data":198936,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93645648,"statfs":{"total":96636764160,"available":95893143552,"internally_reserved":0,"allocated":203710464,"data_stored":203521262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":539881166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,3,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202760192,"data_stored":202760192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":205627392,"data_stored":205627392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203276288,"data_stored":203276288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":202539008,"data_stored":202539008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:46.519 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:47.125 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:47.125 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:47.154 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-16T04:48:46.711729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":405513920,"num_objects":49507,"num_object_clones":0,"num_object_copies":198028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49507,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151980,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5483,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152007,"ondisk_log_size":152007,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773864,"kb_used_data":794268,"kb_used_omap":14,"kb_used_meta":1979505,"kb_avail":374713496,"statfs":{"total":386547056640,"available":383706619904,"internally_reserved":0,"allocated":813330432,"data_stored":812573624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2027013681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,4,6,11,15,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16728064,"num_objects":-2042,"num_object_clones":0,"num_object_copies":-8168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2042,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002949"},"pg_stats":[{"pgid":"1.f","version":"35'9534","reported_seq":9650,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.471928+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:42.471928+0000","last_peered":"2024-09-16T04:48:42.471928+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:42.471928+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9534,"log_dups_size":0,"ondisk_log_size":9534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25559040,"num_objects":3120,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9534,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":934}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9533","reported_seq":9621,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.233124+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:42.233124+0000","last_peered":"2024-09-16T04:48:42.233124+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:42.233124+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9533,"log_dups_size":0,"ondisk_log_size":9533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25728268,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9531,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":938}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9538","reported_seq":9636,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.730322+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:41.730322+0000","last_peered":"2024-09-16T04:48:41.730322+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:41.730322+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9538,"log_dups_size":0,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25655966,"num_objects":3133,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9537,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":914}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9567","reported_seq":10875,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.424899+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:42.424899+0000","last_peered":"2024-09-16T04:48:42.424899+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:42.424899+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9567,"log_dups_size":0,"ondisk_log_size":9567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25416972,"num_objects":3103,"num_object_clones":0,"num_object_copies":12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9565,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":620,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9615","reported_seq":9747,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.637913+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:42.637913+0000","last_peered":"2024-09-16T04:48:42.637913+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:42.637913+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9615,"log_dups_size":0,"ondisk_log_size":9615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25605388,"num_objects":3126,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":878}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9292","reported_seq":9402,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.645889+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:42.645889+0000","last_peered":"2024-09-16T04:48:42.645889+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:42.645889+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9292,"log_dups_size":0,"ondisk_log_size":9292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24739840,"num_objects":3020,"num_object_clones":0,"num_object_copies":12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9545","reported_seq":9642,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.342693+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:42.342693+0000","last_peered":"2024-09-16T04:48:42.342693+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:42.342693+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9545,"log_dups_size":0,"ondisk_log_size":9545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25023756,"num_objects":3055,"num_object_clones":0,"num_object_copies":12220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9543,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":842}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9652","reported_seq":14214,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:44.685947+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:44.685947+0000","last_peered":"2024-09-16T04:48:44.685947+0000","last_clean":"2024-09-16T04:48:44.685947+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:44.685947+0000","last_undegraded":"2024-09-16T04:48:44.685947+0000","last_fullsized":"2024-09-16T04:48:44.685947+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:47.528598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25662732,"num_objects":3133,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9707","reported_seq":14464,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:42.605903+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:42.605903+0000","last_peered":"2024-09-16T04:48:42.605903+0000","last_clean":"2024-09-16T04:48:42.605903+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:42.605903+0000","last_undegraded":"2024-09-16T04:48:42.605903+0000","last_fullsized":"2024-09-16T04:48:42.605903+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:43.814884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24971800,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9704,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9570","reported_seq":9677,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.439717+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:41.439717+0000","last_peered":"2024-09-16T04:48:41.439717+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:41.439717+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9570,"log_dups_size":0,"ondisk_log_size":9570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26113292,"num_objects":3188,"num_object_clones":0,"num_object_copies":12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9568,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":999}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9486","reported_seq":9616,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.736200+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:42.736200+0000","last_peered":"2024-09-16T04:48:42.736200+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:42.736200+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9486,"log_dups_size":0,"ondisk_log_size":9486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24612754,"num_objects":3005,"num_object_clones":0,"num_object_copies":12020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9483,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":850}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9282","reported_seq":9374,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.546736+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:42.546736+0000","last_peered":"2024-09-16T04:48:42.546736+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:42.546736+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24976006,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":944}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9434","reported_seq":9521,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.278655+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:41.278655+0000","last_peered":"2024-09-16T04:48:41.278655+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:41.278655+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9434,"log_dups_size":0,"ondisk_log_size":9434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25385606,"num_objects":3099,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9433,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":878}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9617","reported_seq":9749,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.265732+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:42.265732+0000","last_peered":"2024-09-16T04:48:42.265732+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:42.265732+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9617,"log_dups_size":0,"ondisk_log_size":9617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25924876,"num_objects":3165,"num_object_clones":0,"num_object_copies":12660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9615,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":967}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9306","reported_seq":9409,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.605004+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:41.605004+0000","last_peered":"2024-09-16T04:48:41.605004+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:41.605004+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9306,"log_dups_size":0,"ondisk_log_size":9306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25744652,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9304,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1020}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9329","reported_seq":9423,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.371802+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:42.371802+0000","last_peered":"2024-09-16T04:48:42.371802+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:42.371802+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24392972,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9327,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":750}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":405513920,"num_objects":49507,"num_object_clones":0,"num_object_copies":198028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49507,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151980,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5483,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":809996288,"data_stored":809996288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152007,"ondisk_log_size":152007,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721940,"kb_used_data":198784,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93649900,"statfs":{"total":96636764160,"available":95897497600,"internally_reserved":0,"allocated":203554816,"data_stored":203365614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":535686863},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,0,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713752,"kb_used_data":199432,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93658088,"statfs":{"total":96636764160,"available":95905882112,"internally_reserved":0,"allocated":204218368,"data_stored":204029166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611980,"kb_used_data":197116,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93759860,"statfs":{"total":96636764160,"available":96010096640,"internally_reserved":0,"allocated":201846784,"data_stored":201657582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424799951},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247910,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726192,"kb_used_data":198936,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93645648,"statfs":{"total":96636764160,"available":95893143552,"internally_reserved":0,"allocated":203710464,"data_stored":203521262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":539881166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,3,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202760192,"data_stored":202760192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":201420800,"data_stored":201420800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203276288,"data_stored":203276288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":202539008,"data_stored":202539008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:47.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:48:47.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9534", 'reported_seq': 9650, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:42.471928+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:48:42.471928+0000', 'last_peered': '2024-09-16T04:48:42.471928+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:48:42.471928+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9534, 'log_dups_size': 0, 'ondisk_log_size': 9534, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25559040, 'num_objects': 3120, 'num_object_clones': 0, 'num_object_copies': 12480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9534, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 934}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:47.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:48:47.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9533", 'reported_seq': 9621, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:42.233124+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:48:42.233124+0000', 'last_peered': '2024-09-16T04:48:42.233124+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:48:42.233124+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9533, 'log_dups_size': 0, 'ondisk_log_size': 9533, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25728268, 'num_objects': 3141, 'num_object_clones': 0, 'num_object_copies': 12564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9531, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 938}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:47.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:48:47.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9538", 'reported_seq': 9636, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:41.730322+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:48:41.730322+0000', 'last_peered': '2024-09-16T04:48:41.730322+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:48:41.730322+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9538, 'log_dups_size': 0, 'ondisk_log_size': 9538, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25655966, 'num_objects': 3133, 'num_object_clones': 0, 'num_object_copies': 12532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3133, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9537, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 914}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:47.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:48:47.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9567", 'reported_seq': 10875, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:42.424899+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:48:42.424899+0000', 'last_peered': '2024-09-16T04:48:42.424899+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:48:42.424899+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9567, 'log_dups_size': 0, 'ondisk_log_size': 9567, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25416972, 'num_objects': 3103, 'num_object_clones': 0, 'num_object_copies': 12412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1613, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3103, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9565, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 620, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1490}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:47.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:48:47.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9615", 'reported_seq': 9747, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:42.637913+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:48:42.637913+0000', 'last_peered': '2024-09-16T04:48:42.637913+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:48:42.637913+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9615, 'log_dups_size': 0, 'ondisk_log_size': 9615, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25605388, 'num_objects': 3126, 'num_object_clones': 0, 'num_object_copies': 12504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9613, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 878}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:47.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:48:47.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9292", 'reported_seq': 9402, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:42.645889+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:48:42.645889+0000', 'last_peered': '2024-09-16T04:48:42.645889+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:48:42.645889+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9292, 'log_dups_size': 0, 'ondisk_log_size': 9292, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24739840, 'num_objects': 3020, 'num_object_clones': 0, 'num_object_copies': 12080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9292, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 842}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:47.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:48:47.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9545", 'reported_seq': 9642, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:42.342693+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:48:42.342693+0000', 'last_peered': '2024-09-16T04:48:42.342693+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:48:42.342693+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9545, 'log_dups_size': 0, 'ondisk_log_size': 9545, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25023756, 'num_objects': 3055, 'num_object_clones': 0, 'num_object_copies': 12220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3055, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9543, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 842}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:47.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:48:47.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9570", 'reported_seq': 9677, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:41.439717+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:48:41.439717+0000', 'last_peered': '2024-09-16T04:48:41.439717+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:48:41.439717+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9570, 'log_dups_size': 0, 'ondisk_log_size': 9570, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26113292, 'num_objects': 3188, 'num_object_clones': 0, 'num_object_copies': 12752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9568, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 999}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:47.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:48:47.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9486", 'reported_seq': 9616, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:42.736200+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:48:42.736200+0000', 'last_peered': '2024-09-16T04:48:42.736200+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:48:42.736200+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9486, 'log_dups_size': 0, 'ondisk_log_size': 9486, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24612754, 'num_objects': 3005, 'num_object_clones': 0, 'num_object_copies': 12020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3005, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9483, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 850}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:47.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:48:47.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9282", 'reported_seq': 9374, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:42.546736+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:48:42.546736+0000', 'last_peered': '2024-09-16T04:48:42.546736+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:48:42.546736+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9282, 'log_dups_size': 0, 'ondisk_log_size': 9282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24976006, 'num_objects': 3049, 'num_object_clones': 0, 'num_object_copies': 12196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9281, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 944}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:47.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:48:47.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9434", 'reported_seq': 9521, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:41.278655+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:48:41.278655+0000', 'last_peered': '2024-09-16T04:48:41.278655+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:48:41.278655+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9434, 'log_dups_size': 0, 'ondisk_log_size': 9434, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25385606, 'num_objects': 3099, 'num_object_clones': 0, 'num_object_copies': 12396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3099, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9433, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 878}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:47.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:48:47.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9617", 'reported_seq': 9749, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:42.265732+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:48:42.265732+0000', 'last_peered': '2024-09-16T04:48:42.265732+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:48:42.265732+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9617, 'log_dups_size': 0, 'ondisk_log_size': 9617, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25924876, 'num_objects': 3165, 'num_object_clones': 0, 'num_object_copies': 12660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3165, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9615, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 967}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:47.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:48:47.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9306", 'reported_seq': 9409, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:41.605004+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:48:41.605004+0000', 'last_peered': '2024-09-16T04:48:41.605004+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:48:41.605004+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9306, 'log_dups_size': 0, 'ondisk_log_size': 9306, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25744652, 'num_objects': 3143, 'num_object_clones': 0, 'num_object_copies': 12572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3143, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9304, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1020}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:47.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:48:47.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9329", 'reported_seq': 9423, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:42.371802+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:48:42.371802+0000', 'last_peered': '2024-09-16T04:48:42.371802+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:48:42.371802+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9329, 'log_dups_size': 0, 'ondisk_log_size': 9329, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24392972, 'num_objects': 2978, 'num_object_clones': 0, 'num_object_copies': 11912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9327, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 750}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:47.158 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:48:47.924 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:47.924 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30099/198028 objects degraded (15.199%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":540,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":49507,"data_bytes":405513920,"bytes_used":2840436736,"bytes_avail":383706619904,"bytes_total":386547056640,"degraded_objects":30099,"degraded_total":198028,"degraded_ratio":0.15199365746258106,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":170},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:48:47.952 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:48:47.952 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:48.605 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:48.605 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:48.648 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-16T04:48:46.711729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":405513920,"num_objects":49507,"num_object_clones":0,"num_object_copies":198028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49507,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151980,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5483,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152007,"ondisk_log_size":152007,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773864,"kb_used_data":794268,"kb_used_omap":14,"kb_used_meta":1979505,"kb_avail":374713496,"statfs":{"total":386547056640,"available":383706619904,"internally_reserved":0,"allocated":813330432,"data_stored":812573624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2027013681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,4,6,11,15,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16728064,"num_objects":-2042,"num_object_clones":0,"num_object_copies":-8168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2042,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002949"},"pg_stats":[{"pgid":"1.f","version":"35'9534","reported_seq":9650,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.471928+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:42.471928+0000","last_peered":"2024-09-16T04:48:42.471928+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:42.471928+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9534,"log_dups_size":0,"ondisk_log_size":9534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25559040,"num_objects":3120,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9534,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":934}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9533","reported_seq":9621,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.233124+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:42.233124+0000","last_peered":"2024-09-16T04:48:42.233124+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:42.233124+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9533,"log_dups_size":0,"ondisk_log_size":9533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25728268,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9531,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":938}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9538","reported_seq":9636,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.730322+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:41.730322+0000","last_peered":"2024-09-16T04:48:41.730322+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:41.730322+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9538,"log_dups_size":0,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25655966,"num_objects":3133,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9537,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":914}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9567","reported_seq":10875,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.424899+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:42.424899+0000","last_peered":"2024-09-16T04:48:42.424899+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:42.424899+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9567,"log_dups_size":0,"ondisk_log_size":9567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25416972,"num_objects":3103,"num_object_clones":0,"num_object_copies":12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9565,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":620,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9615","reported_seq":9747,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.637913+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:42.637913+0000","last_peered":"2024-09-16T04:48:42.637913+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:42.637913+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9615,"log_dups_size":0,"ondisk_log_size":9615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25605388,"num_objects":3126,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":878}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9292","reported_seq":9402,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.645889+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:42.645889+0000","last_peered":"2024-09-16T04:48:42.645889+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:42.645889+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9292,"log_dups_size":0,"ondisk_log_size":9292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24739840,"num_objects":3020,"num_object_clones":0,"num_object_copies":12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9545","reported_seq":9642,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.342693+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:42.342693+0000","last_peered":"2024-09-16T04:48:42.342693+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:42.342693+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9545,"log_dups_size":0,"ondisk_log_size":9545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25023756,"num_objects":3055,"num_object_clones":0,"num_object_copies":12220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9543,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":842}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9652","reported_seq":14214,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:44.685947+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:44.685947+0000","last_peered":"2024-09-16T04:48:44.685947+0000","last_clean":"2024-09-16T04:48:44.685947+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:44.685947+0000","last_undegraded":"2024-09-16T04:48:44.685947+0000","last_fullsized":"2024-09-16T04:48:44.685947+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:47.528598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25662732,"num_objects":3133,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9707","reported_seq":14464,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:42.605903+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:42.605903+0000","last_peered":"2024-09-16T04:48:42.605903+0000","last_clean":"2024-09-16T04:48:42.605903+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:42.605903+0000","last_undegraded":"2024-09-16T04:48:42.605903+0000","last_fullsized":"2024-09-16T04:48:42.605903+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:43.814884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24971800,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9704,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9570","reported_seq":9677,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.439717+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:41.439717+0000","last_peered":"2024-09-16T04:48:41.439717+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:41.439717+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9570,"log_dups_size":0,"ondisk_log_size":9570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26113292,"num_objects":3188,"num_object_clones":0,"num_object_copies":12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9568,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":999}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9486","reported_seq":9616,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.736200+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:42.736200+0000","last_peered":"2024-09-16T04:48:42.736200+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:42.736200+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9486,"log_dups_size":0,"ondisk_log_size":9486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24612754,"num_objects":3005,"num_object_clones":0,"num_object_copies":12020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9483,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":850}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9282","reported_seq":9374,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.546736+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:42.546736+0000","last_peered":"2024-09-16T04:48:42.546736+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:42.546736+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24976006,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":944}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9434","reported_seq":9521,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.278655+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:41.278655+0000","last_peered":"2024-09-16T04:48:41.278655+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:41.278655+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9434,"log_dups_size":0,"ondisk_log_size":9434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25385606,"num_objects":3099,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9433,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":878}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9617","reported_seq":9749,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.265732+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:42.265732+0000","last_peered":"2024-09-16T04:48:42.265732+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:42.265732+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9617,"log_dups_size":0,"ondisk_log_size":9617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25924876,"num_objects":3165,"num_object_clones":0,"num_object_copies":12660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9615,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":967}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9306","reported_seq":9409,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:41.605004+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:41.605004+0000","last_peered":"2024-09-16T04:48:41.605004+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:41.605004+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9306,"log_dups_size":0,"ondisk_log_size":9306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25744652,"num_objects":3143,"num_object_clones":0,"num_object_copies":12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9304,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1020}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9329","reported_seq":9423,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:42.371802+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:42.371802+0000","last_peered":"2024-09-16T04:48:42.371802+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:42.371802+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24392972,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9327,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":750}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":405513920,"num_objects":49507,"num_object_clones":0,"num_object_copies":198028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49507,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151980,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5483,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":809996288,"data_stored":809996288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152007,"ondisk_log_size":152007,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721940,"kb_used_data":198784,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93649900,"statfs":{"total":96636764160,"available":95897497600,"internally_reserved":0,"allocated":203554816,"data_stored":203365614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":535686863},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,0,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713752,"kb_used_data":199432,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93658088,"statfs":{"total":96636764160,"available":95905882112,"internally_reserved":0,"allocated":204218368,"data_stored":204029166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611980,"kb_used_data":197116,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93759860,"statfs":{"total":96636764160,"available":96010096640,"internally_reserved":0,"allocated":201846784,"data_stored":201657582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424799951},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247910,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726192,"kb_used_data":198936,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93645648,"statfs":{"total":96636764160,"available":95893143552,"internally_reserved":0,"allocated":203710464,"data_stored":203521262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":539881166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,3,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202760192,"data_stored":202760192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":201420800,"data_stored":201420800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203276288,"data_stored":203276288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":202539008,"data_stored":202539008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:51.650 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:52.379 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:52.379 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:52.415 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2024-09-16T04:48:50.712719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":397076160,"num_objects":48477,"num_object_clones":0,"num_object_copies":193908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153010,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5525,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153037,"ondisk_log_size":153037,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2828528,"kb_used_data":776728,"kb_used_omap":14,"kb_used_meta":2051697,"kb_avail":374658832,"statfs":{"total":386547056640,"available":383650643968,"internally_reserved":0,"allocated":795369472,"data_stored":794612664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2100938289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,6,3,7,9,17,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17178624,"num_objects":-2097,"num_object_clones":0,"num_object_copies":-8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2097,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002976"},"pg_stats":[{"pgid":"1.f","version":"35'9585","reported_seq":9701,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.799850+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:47.799850+0000","last_peered":"2024-09-16T04:48:47.799850+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:47.799850+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9585,"log_dups_size":0,"ondisk_log_size":9585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25141248,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9585,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9589","reported_seq":9677,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.398332+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:47.398332+0000","last_peered":"2024-09-16T04:48:47.398332+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:47.398332+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9589,"log_dups_size":0,"ondisk_log_size":9589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25269516,"num_objects":3085,"num_object_clones":0,"num_object_copies":12340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9587,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":882}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9594","reported_seq":9692,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:46.561541+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:46.561541+0000","last_peered":"2024-09-16T04:48:46.561541+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:46.561541+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9594,"log_dups_size":0,"ondisk_log_size":9594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25197214,"num_objects":3077,"num_object_clones":0,"num_object_copies":12308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3077,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9593,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":858}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9637","reported_seq":11030,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.452105+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:47.452105+0000","last_peered":"2024-09-16T04:48:47.452105+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:47.452105+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9637,"log_dups_size":0,"ondisk_log_size":9637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24843532,"num_objects":3033,"num_object_clones":0,"num_object_copies":12132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9635,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":662,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9692","reported_seq":9822,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.659069+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:47.659069+0000","last_peered":"2024-09-16T04:48:47.659069+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:47.659069+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9692,"log_dups_size":0,"ondisk_log_size":9692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24974604,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":801}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9360","reported_seq":9470,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.793018+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:47.793018+0000","last_peered":"2024-09-16T04:48:47.793018+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:47.793018+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9360,"log_dups_size":0,"ondisk_log_size":9360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182784,"num_objects":2952,"num_object_clones":0,"num_object_copies":11808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9360,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9613","reported_seq":9710,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.237848+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:47.237848+0000","last_peered":"2024-09-16T04:48:47.237848+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:47.237848+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24466700,"num_objects":2987,"num_object_clones":0,"num_object_copies":11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9611,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":774}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9716","reported_seq":14278,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:49.739017+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:49.739017+0000","last_peered":"2024-09-16T04:48:49.739017+0000","last_clean":"2024-09-16T04:48:49.739017+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:49.739017+0000","last_undegraded":"2024-09-16T04:48:49.739017+0000","last_fullsized":"2024-09-16T04:48:49.739017+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9716,"log_dups_size":0,"ondisk_log_size":9716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:52.579933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25138444,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9714,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9773","reported_seq":14530,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:47.509186+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:47.509186+0000","last_peered":"2024-09-16T04:48:47.509186+0000","last_clean":"2024-09-16T04:48:47.509186+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:47.509186+0000","last_undegraded":"2024-09-16T04:48:47.509186+0000","last_fullsized":"2024-09-16T04:48:47.509186+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9773,"log_dups_size":0,"ondisk_log_size":9773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:48.834152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24431128,"num_objects":2983,"num_object_clones":0,"num_object_copies":11932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9770,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9635","reported_seq":9742,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:46.430076+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:46.430076+0000","last_peered":"2024-09-16T04:48:46.430076+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:46.430076+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9635,"log_dups_size":0,"ondisk_log_size":9635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25580812,"num_objects":3123,"num_object_clones":0,"num_object_copies":12492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9633,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":934}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9555","reported_seq":9687,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.632102+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:47.632102+0000","last_peered":"2024-09-16T04:48:47.632102+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:47.632102+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9555,"log_dups_size":0,"ondisk_log_size":9555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24047506,"num_objects":2936,"num_object_clones":0,"num_object_copies":11744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9552,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":781}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9329","reported_seq":9421,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.451552+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:47.451552+0000","last_peered":"2024-09-16T04:48:47.451552+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:47.451552+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24590982,"num_objects":3002,"num_object_clones":0,"num_object_copies":12008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9328,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9501","reported_seq":9588,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:46.663417+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:46.663417+0000","last_peered":"2024-09-16T04:48:46.663417+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:46.663417+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9501,"log_dups_size":0,"ondisk_log_size":9501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24836742,"num_objects":3032,"num_object_clones":0,"num_object_copies":12128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9500,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":811}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9684","reported_seq":9816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.727853+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:47.727853+0000","last_peered":"2024-09-16T04:48:47.727853+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:47.727853+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9684,"log_dups_size":0,"ondisk_log_size":9684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25376012,"num_objects":3098,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9682,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":900}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9376","reported_seq":9479,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:46.766531+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:46.766531+0000","last_peered":"2024-09-16T04:48:46.766531+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:46.766531+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9376,"log_dups_size":0,"ondisk_log_size":9376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25171212,"num_objects":3073,"num_object_clones":0,"num_object_copies":12292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9374,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":950}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9398","reported_seq":9492,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.365238+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:47.365238+0000","last_peered":"2024-09-16T04:48:47.365238+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:47.365238+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9398,"log_dups_size":0,"ondisk_log_size":9398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23827724,"num_objects":2909,"num_object_clones":0,"num_object_copies":11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9396,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":681}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":397076160,"num_objects":48477,"num_object_clones":0,"num_object_copies":193908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153010,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5525,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":793141248,"data_stored":793141248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153037,"ondisk_log_size":153037,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721348,"kb_used_data":194032,"kb_used_omap":4,"kb_used_meta":527291,"kb_avail":93650492,"statfs":{"total":96636764160,"available":95898103808,"internally_reserved":0,"allocated":198688768,"data_stored":198499566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":539946703},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,0,1,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":767916,"kb_used_data":195272,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93603924,"statfs":{"total":96636764160,"available":95850418176,"internally_reserved":0,"allocated":199958528,"data_stored":199769326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630260,"kb_used_data":193188,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93741580,"statfs":{"total":96636764160,"available":95991377920,"internally_reserved":0,"allocated":197824512,"data_stored":197635310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447540943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,5,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709004,"kb_used_data":194236,"kb_used_omap":4,"kb_used_meta":514747,"kb_avail":93662836,"statfs":{"total":96636764160,"available":95910744064,"internally_reserved":0,"allocated":198897664,"data_stored":198708462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":527101646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":198557696,"data_stored":198557696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":197251072,"data_stored":197251072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":199049216,"data_stored":199049216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":198283264,"data_stored":198283264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:52.416 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:52.971 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:52.972 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:52.998 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-09-16T04:48:52.713191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":392840896,"num_objects":47960,"num_object_clones":0,"num_object_copies":191840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153527,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5588,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153554,"ondisk_log_size":153554,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2828756,"kb_used_data":768640,"kb_used_omap":14,"kb_used_meta":2060017,"kb_avail":374658604,"statfs":{"total":386547056640,"available":383650410496,"internally_reserved":0,"allocated":787087360,"data_stored":786330552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2109457969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,5,8,5,13,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20914176,"num_objects":-2553,"num_object_clones":0,"num_object_copies":-10212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2553,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002934"},"pg_stats":[{"pgid":"1.f","version":"35'9585","reported_seq":9701,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.799850+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:47.799850+0000","last_peered":"2024-09-16T04:48:47.799850+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:47.799850+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9585,"log_dups_size":0,"ondisk_log_size":9585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25141248,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9585,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9649","reported_seq":9737,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.509086+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:52.509086+0000","last_peered":"2024-09-16T04:48:52.509086+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:52.509086+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9649,"log_dups_size":0,"ondisk_log_size":9649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24777996,"num_objects":3025,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9647,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9655","reported_seq":9753,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:51.842047+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:51.842047+0000","last_peered":"2024-09-16T04:48:51.842047+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:51.842047+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9655,"log_dups_size":0,"ondisk_log_size":9655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24697502,"num_objects":3016,"num_object_clones":0,"num_object_copies":12064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9654,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":797}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9705","reported_seq":11225,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.494905+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:52.494905+0000","last_peered":"2024-09-16T04:48:52.494905+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:52.494905+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9705,"log_dups_size":0,"ondisk_log_size":9705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24286476,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1508,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9703,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":725,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1457}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9692","reported_seq":9822,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.659069+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:47.659069+0000","last_peered":"2024-09-16T04:48:47.659069+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:47.659069+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9692,"log_dups_size":0,"ondisk_log_size":9692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24974604,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":801}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9360","reported_seq":9470,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.793018+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:47.793018+0000","last_peered":"2024-09-16T04:48:47.793018+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:47.793018+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9360,"log_dups_size":0,"ondisk_log_size":9360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182784,"num_objects":2952,"num_object_clones":0,"num_object_copies":11808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9360,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9678","reported_seq":9775,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.491746+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:52.491746+0000","last_peered":"2024-09-16T04:48:52.491746+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:52.491746+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9678,"log_dups_size":0,"ondisk_log_size":9678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23934220,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9676,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":709}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9716","reported_seq":14278,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:49.739017+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:49.739017+0000","last_peered":"2024-09-16T04:48:49.739017+0000","last_clean":"2024-09-16T04:48:49.739017+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:49.739017+0000","last_undegraded":"2024-09-16T04:48:49.739017+0000","last_fullsized":"2024-09-16T04:48:49.739017+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9716,"log_dups_size":0,"ondisk_log_size":9716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:52.579933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25138444,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9714,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9773","reported_seq":14530,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:47.509186+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:47.509186+0000","last_peered":"2024-09-16T04:48:47.509186+0000","last_clean":"2024-09-16T04:48:47.509186+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:47.509186+0000","last_undegraded":"2024-09-16T04:48:47.509186+0000","last_fullsized":"2024-09-16T04:48:47.509186+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9773,"log_dups_size":0,"ondisk_log_size":9773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:48.834152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24431128,"num_objects":2983,"num_object_clones":0,"num_object_copies":11932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9770,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9689","reported_seq":9796,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:51.856164+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:51.856164+0000","last_peered":"2024-09-16T04:48:51.856164+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:51.856164+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9689,"log_dups_size":0,"ondisk_log_size":9689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25138444,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9687,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9555","reported_seq":9687,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.632102+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:47.632102+0000","last_peered":"2024-09-16T04:48:47.632102+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:47.632102+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9555,"log_dups_size":0,"ondisk_log_size":9555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24047506,"num_objects":2936,"num_object_clones":0,"num_object_copies":11744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9552,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":781}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9329","reported_seq":9421,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.451552+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:47.451552+0000","last_peered":"2024-09-16T04:48:47.451552+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:47.451552+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24590982,"num_objects":3002,"num_object_clones":0,"num_object_copies":12008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9328,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9572","reported_seq":9660,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:51.678955+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:51.678955+0000","last_peered":"2024-09-16T04:48:51.678955+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:51.678955+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9572,"log_dups_size":0,"ondisk_log_size":9572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24255110,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9571,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":740}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9684","reported_seq":9816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.727853+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:47.727853+0000","last_peered":"2024-09-16T04:48:47.727853+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:47.727853+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9684,"log_dups_size":0,"ondisk_log_size":9684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25376012,"num_objects":3098,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9682,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":900}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9436","reported_seq":9537,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:51.652240+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:51.652240+0000","last_peered":"2024-09-16T04:48:51.652240+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:51.652240+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9436,"log_dups_size":0,"ondisk_log_size":9436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24679692,"num_objects":3013,"num_object_clones":0,"num_object_copies":12052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9434,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":890}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9476","reported_seq":9570,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.317124+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:52.317124+0000","last_peered":"2024-09-16T04:48:52.317124+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:52.317124+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9476,"log_dups_size":0,"ondisk_log_size":9476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23188748,"num_objects":2831,"num_object_clones":0,"num_object_copies":11324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9474,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":603}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":392840896,"num_objects":47960,"num_object_clones":0,"num_object_copies":191840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153527,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5588,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":784834560,"data_stored":784834560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153554,"ondisk_log_size":153554,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721348,"kb_used_data":194032,"kb_used_omap":4,"kb_used_meta":527291,"kb_avail":93650492,"statfs":{"total":96636764160,"available":95898103808,"internally_reserved":0,"allocated":198688768,"data_stored":198499566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":539946703},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,0,1,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768032,"kb_used_data":191248,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93603808,"statfs":{"total":96636764160,"available":95850299392,"internally_reserved":0,"allocated":195837952,"data_stored":195648750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,0,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630260,"kb_used_data":193188,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93741580,"statfs":{"total":96636764160,"available":95991377920,"internally_reserved":0,"allocated":197824512,"data_stored":197635310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447540943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,5,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247912,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709116,"kb_used_data":190172,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93662724,"statfs":{"total":96636764160,"available":95910629376,"internally_reserved":0,"allocated":194736128,"data_stored":194546926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":531361486},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":194400256,"data_stored":194400256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":197251072,"data_stored":197251072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194899968,"data_stored":194899968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":198283264,"data_stored":198283264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:53.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:48:53.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9585", 'reported_seq': 9701, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:47.799850+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:48:47.799850+0000', 'last_peered': '2024-09-16T04:48:47.799850+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:48:47.799850+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9585, 'log_dups_size': 0, 'ondisk_log_size': 9585, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25141248, 'num_objects': 3069, 'num_object_clones': 0, 'num_object_copies': 12276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3069, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9585, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 883}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:53.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:48:53.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9649", 'reported_seq': 9737, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:52.509086+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:48:52.509086+0000', 'last_peered': '2024-09-16T04:48:52.509086+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:48:52.509086+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9649, 'log_dups_size': 0, 'ondisk_log_size': 9649, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24777996, 'num_objects': 3025, 'num_object_clones': 0, 'num_object_copies': 12100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9647, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 822}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:53.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:48:53.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9655", 'reported_seq': 9753, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:51.842047+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:48:51.842047+0000', 'last_peered': '2024-09-16T04:48:51.842047+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:48:51.842047+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9655, 'log_dups_size': 0, 'ondisk_log_size': 9655, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24697502, 'num_objects': 3016, 'num_object_clones': 0, 'num_object_copies': 12064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3016, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9654, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 797}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:53.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:48:53.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9705", 'reported_seq': 11225, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:52.494905+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:48:52.494905+0000', 'last_peered': '2024-09-16T04:48:52.494905+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:48:52.494905+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9705, 'log_dups_size': 0, 'ondisk_log_size': 9705, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24286476, 'num_objects': 2965, 'num_object_clones': 0, 'num_object_copies': 11860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1508, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2965, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9703, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 725, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1457}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:53.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:48:53.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9692", 'reported_seq': 9822, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:47.659069+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:48:47.659069+0000', 'last_peered': '2024-09-16T04:48:47.659069+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:48:47.659069+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9692, 'log_dups_size': 0, 'ondisk_log_size': 9692, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24974604, 'num_objects': 3049, 'num_object_clones': 0, 'num_object_copies': 12196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9690, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 801}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:53.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:48:53.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9360", 'reported_seq': 9470, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:47.793018+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:48:47.793018+0000', 'last_peered': '2024-09-16T04:48:47.793018+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:48:47.793018+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9360, 'log_dups_size': 0, 'ondisk_log_size': 9360, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24182784, 'num_objects': 2952, 'num_object_clones': 0, 'num_object_copies': 11808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2952, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9360, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 774}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:53.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:48:53.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9678", 'reported_seq': 9775, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:52.491746+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:48:52.491746+0000', 'last_peered': '2024-09-16T04:48:52.491746+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:48:52.491746+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9678, 'log_dups_size': 0, 'ondisk_log_size': 9678, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23934220, 'num_objects': 2922, 'num_object_clones': 0, 'num_object_copies': 11688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9676, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 709}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:53.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:48:53.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9689", 'reported_seq': 9796, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:51.856164+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:48:51.856164+0000', 'last_peered': '2024-09-16T04:48:51.856164+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:48:51.856164+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9689, 'log_dups_size': 0, 'ondisk_log_size': 9689, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25138444, 'num_objects': 3069, 'num_object_clones': 0, 'num_object_copies': 12276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3069, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9687, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 880}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:53.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:48:53.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9555", 'reported_seq': 9687, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:47.632102+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:48:47.632102+0000', 'last_peered': '2024-09-16T04:48:47.632102+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:48:47.632102+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9555, 'log_dups_size': 0, 'ondisk_log_size': 9555, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24047506, 'num_objects': 2936, 'num_object_clones': 0, 'num_object_copies': 11744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9552, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 781}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:53.002 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:48:53.002 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9329", 'reported_seq': 9421, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:47.451552+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:48:47.451552+0000', 'last_peered': '2024-09-16T04:48:47.451552+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:48:47.451552+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9329, 'log_dups_size': 0, 'ondisk_log_size': 9329, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24590982, 'num_objects': 3002, 'num_object_clones': 0, 'num_object_copies': 12008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9328, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:53.002 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:48:53.002 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9572", 'reported_seq': 9660, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:51.678955+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:48:51.678955+0000', 'last_peered': '2024-09-16T04:48:51.678955+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:48:51.678955+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9572, 'log_dups_size': 0, 'ondisk_log_size': 9572, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24255110, 'num_objects': 2961, 'num_object_clones': 0, 'num_object_copies': 11844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2961, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9571, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 740}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:53.002 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:48:53.002 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9684", 'reported_seq': 9816, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:47.727853+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:48:47.727853+0000', 'last_peered': '2024-09-16T04:48:47.727853+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:48:47.727853+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9684, 'log_dups_size': 0, 'ondisk_log_size': 9684, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25376012, 'num_objects': 3098, 'num_object_clones': 0, 'num_object_copies': 12392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3098, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9682, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 900}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:53.002 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:48:53.002 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9436", 'reported_seq': 9537, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:51.652240+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:48:51.652240+0000', 'last_peered': '2024-09-16T04:48:51.652240+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:48:51.652240+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9436, 'log_dups_size': 0, 'ondisk_log_size': 9436, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24679692, 'num_objects': 3013, 'num_object_clones': 0, 'num_object_copies': 12052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9434, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 890}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:53.002 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:48:53.003 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9476", 'reported_seq': 9570, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:52.317124+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:48:52.317124+0000', 'last_peered': '2024-09-16T04:48:52.317124+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:48:52.317124+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9476, 'log_dups_size': 0, 'ondisk_log_size': 9476, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23188748, 'num_objects': 2831, 'num_object_clones': 0, 'num_object_copies': 11324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9474, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 603}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:53.003 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:48:53.779 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:53.779 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29994/191840 objects degraded (15.635%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":546,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47960,"data_bytes":392840896,"bytes_used":2896646144,"bytes_avail":383650410496,"bytes_total":386547056640,"degraded_objects":29994,"degraded_total":191840,"degraded_ratio":0.15634904086738949,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":212},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:48:53.825 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:48:53.825 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:54.456 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:54.457 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:54.474 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-09-16T04:48:52.713191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":392840896,"num_objects":47960,"num_object_clones":0,"num_object_copies":191840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153527,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5588,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153554,"ondisk_log_size":153554,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2828756,"kb_used_data":768640,"kb_used_omap":14,"kb_used_meta":2060017,"kb_avail":374658604,"statfs":{"total":386547056640,"available":383650410496,"internally_reserved":0,"allocated":787087360,"data_stored":786330552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2109457969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,5,8,5,13,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20914176,"num_objects":-2553,"num_object_clones":0,"num_object_copies":-10212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2553,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002934"},"pg_stats":[{"pgid":"1.f","version":"35'9585","reported_seq":9701,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.799850+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:47.799850+0000","last_peered":"2024-09-16T04:48:47.799850+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:47.799850+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9585,"log_dups_size":0,"ondisk_log_size":9585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25141248,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9585,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9649","reported_seq":9737,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.509086+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:52.509086+0000","last_peered":"2024-09-16T04:48:52.509086+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:52.509086+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9649,"log_dups_size":0,"ondisk_log_size":9649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24777996,"num_objects":3025,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9647,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9655","reported_seq":9753,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:51.842047+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:51.842047+0000","last_peered":"2024-09-16T04:48:51.842047+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:51.842047+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9655,"log_dups_size":0,"ondisk_log_size":9655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24697502,"num_objects":3016,"num_object_clones":0,"num_object_copies":12064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9654,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":797}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9705","reported_seq":11225,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.494905+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:52.494905+0000","last_peered":"2024-09-16T04:48:52.494905+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:52.494905+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9705,"log_dups_size":0,"ondisk_log_size":9705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24286476,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1508,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9703,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":725,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1457}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9692","reported_seq":9822,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.659069+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:47.659069+0000","last_peered":"2024-09-16T04:48:47.659069+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:47.659069+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9692,"log_dups_size":0,"ondisk_log_size":9692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24974604,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":801}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9360","reported_seq":9470,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.793018+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:47.793018+0000","last_peered":"2024-09-16T04:48:47.793018+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:47.793018+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9360,"log_dups_size":0,"ondisk_log_size":9360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182784,"num_objects":2952,"num_object_clones":0,"num_object_copies":11808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9360,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9678","reported_seq":9775,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.491746+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:52.491746+0000","last_peered":"2024-09-16T04:48:52.491746+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:52.491746+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9678,"log_dups_size":0,"ondisk_log_size":9678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23934220,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9676,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":709}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9716","reported_seq":14278,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:49.739017+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:49.739017+0000","last_peered":"2024-09-16T04:48:49.739017+0000","last_clean":"2024-09-16T04:48:49.739017+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:49.739017+0000","last_undegraded":"2024-09-16T04:48:49.739017+0000","last_fullsized":"2024-09-16T04:48:49.739017+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9716,"log_dups_size":0,"ondisk_log_size":9716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:52.579933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25138444,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9714,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9773","reported_seq":14530,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:47.509186+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:47.509186+0000","last_peered":"2024-09-16T04:48:47.509186+0000","last_clean":"2024-09-16T04:48:47.509186+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:47.509186+0000","last_undegraded":"2024-09-16T04:48:47.509186+0000","last_fullsized":"2024-09-16T04:48:47.509186+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9773,"log_dups_size":0,"ondisk_log_size":9773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:48.834152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24431128,"num_objects":2983,"num_object_clones":0,"num_object_copies":11932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9770,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9689","reported_seq":9796,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:51.856164+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:51.856164+0000","last_peered":"2024-09-16T04:48:51.856164+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:51.856164+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9689,"log_dups_size":0,"ondisk_log_size":9689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25138444,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9687,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9555","reported_seq":9687,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.632102+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:47.632102+0000","last_peered":"2024-09-16T04:48:47.632102+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:47.632102+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9555,"log_dups_size":0,"ondisk_log_size":9555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24047506,"num_objects":2936,"num_object_clones":0,"num_object_copies":11744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9552,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":781}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9329","reported_seq":9421,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.451552+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:47.451552+0000","last_peered":"2024-09-16T04:48:47.451552+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:47.451552+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24590982,"num_objects":3002,"num_object_clones":0,"num_object_copies":12008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9328,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9572","reported_seq":9660,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:51.678955+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:51.678955+0000","last_peered":"2024-09-16T04:48:51.678955+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:51.678955+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9572,"log_dups_size":0,"ondisk_log_size":9572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24255110,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9571,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":740}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9684","reported_seq":9816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:47.727853+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:47.727853+0000","last_peered":"2024-09-16T04:48:47.727853+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:47.727853+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9684,"log_dups_size":0,"ondisk_log_size":9684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25376012,"num_objects":3098,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9682,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":900}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9436","reported_seq":9537,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:51.652240+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:51.652240+0000","last_peered":"2024-09-16T04:48:51.652240+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:51.652240+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9436,"log_dups_size":0,"ondisk_log_size":9436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24679692,"num_objects":3013,"num_object_clones":0,"num_object_copies":12052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9434,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":890}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9476","reported_seq":9570,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.317124+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:52.317124+0000","last_peered":"2024-09-16T04:48:52.317124+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:52.317124+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9476,"log_dups_size":0,"ondisk_log_size":9476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23188748,"num_objects":2831,"num_object_clones":0,"num_object_copies":11324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9474,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":603}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":392840896,"num_objects":47960,"num_object_clones":0,"num_object_copies":191840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153527,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5588,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":784834560,"data_stored":784834560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153554,"ondisk_log_size":153554,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721348,"kb_used_data":194032,"kb_used_omap":4,"kb_used_meta":527291,"kb_avail":93650492,"statfs":{"total":96636764160,"available":95898103808,"internally_reserved":0,"allocated":198688768,"data_stored":198499566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":539946703},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,0,1,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768032,"kb_used_data":191248,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93603808,"statfs":{"total":96636764160,"available":95850299392,"internally_reserved":0,"allocated":195837952,"data_stored":195648750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,0,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630260,"kb_used_data":193188,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93741580,"statfs":{"total":96636764160,"available":95991377920,"internally_reserved":0,"allocated":197824512,"data_stored":197635310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447540943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,5,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247912,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709116,"kb_used_data":190172,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93662724,"statfs":{"total":96636764160,"available":95910629376,"internally_reserved":0,"allocated":194736128,"data_stored":194546926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":531361486},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":194400256,"data_stored":194400256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":197251072,"data_stored":197251072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194899968,"data_stored":194899968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":198283264,"data_stored":198283264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:57.476 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:58.138 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:58.138 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:58.159 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-09-16T04:48:56.714173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":388523712,"num_objects":47433,"num_object_clones":0,"num_object_copies":189732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47433,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154054,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5588,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154081,"ondisk_log_size":154081,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2795692,"kb_used_data":760532,"kb_used_omap":14,"kb_used_meta":2035057,"kb_avail":374691668,"statfs":{"total":386547056640,"available":383684268032,"internally_reserved":0,"allocated":778784768,"data_stored":778027960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2083898929},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,10,11,18,11,17,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17399808,"num_objects":-2124,"num_object_clones":0,"num_object_copies":-8496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2124,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002955"},"pg_stats":[{"pgid":"1.f","version":"35'9648","reported_seq":9764,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.891418+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:52.891418+0000","last_peered":"2024-09-16T04:48:52.891418+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:52.891418+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9648,"log_dups_size":0,"ondisk_log_size":9648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24625152,"num_objects":3006,"num_object_clones":0,"num_object_copies":12024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9648,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":820}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9649","reported_seq":9737,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.509086+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:52.509086+0000","last_peered":"2024-09-16T04:48:52.509086+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:52.509086+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9649,"log_dups_size":0,"ondisk_log_size":9649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24777996,"num_objects":3025,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9647,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9655","reported_seq":9753,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:51.842047+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:51.842047+0000","last_peered":"2024-09-16T04:48:51.842047+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:51.842047+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9655,"log_dups_size":0,"ondisk_log_size":9655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24697502,"num_objects":3016,"num_object_clones":0,"num_object_copies":12064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9654,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":797}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9705","reported_seq":11225,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.494905+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:52.494905+0000","last_peered":"2024-09-16T04:48:52.494905+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:52.494905+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9705,"log_dups_size":0,"ondisk_log_size":9705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24286476,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1508,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9703,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":725,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1457}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9761","reported_seq":9894,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.977377+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:52.977377+0000","last_peered":"2024-09-16T04:48:52.977377+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:52.977377+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9761,"log_dups_size":0,"ondisk_log_size":9761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24409356,"num_objects":2980,"num_object_clones":0,"num_object_copies":11920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9759,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":732}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9438","reported_seq":9547,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.827831+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:52.827831+0000","last_peered":"2024-09-16T04:48:52.827831+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:52.827831+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9438,"log_dups_size":0,"ondisk_log_size":9438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23543808,"num_objects":2874,"num_object_clones":0,"num_object_copies":11496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9438,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":696}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9678","reported_seq":9775,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.491746+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:52.491746+0000","last_peered":"2024-09-16T04:48:52.491746+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:52.491746+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9678,"log_dups_size":0,"ondisk_log_size":9678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23934220,"num_objects":2922,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9676,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":709}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9792","reported_seq":14354,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:54.879450+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:54.879450+0000","last_peered":"2024-09-16T04:48:54.879450+0000","last_clean":"2024-09-16T04:48:54.879450+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:54.879450+0000","last_undegraded":"2024-09-16T04:48:54.879450+0000","last_fullsized":"2024-09-16T04:48:54.879450+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9792,"log_dups_size":0,"ondisk_log_size":9792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:57.597397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24515852,"num_objects":2993,"num_object_clones":0,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9790,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9827","reported_seq":14584,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:52.862913+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:52.862913+0000","last_peered":"2024-09-16T04:48:52.862913+0000","last_clean":"2024-09-16T04:48:52.862913+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:52.862913+0000","last_undegraded":"2024-09-16T04:48:52.862913+0000","last_fullsized":"2024-09-16T04:48:52.862913+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9827,"log_dups_size":0,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:54.766811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23988760,"num_objects":2929,"num_object_clones":0,"num_object_copies":11716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9824,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9689","reported_seq":9796,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:51.856164+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:51.856164+0000","last_peered":"2024-09-16T04:48:51.856164+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:51.856164+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9689,"log_dups_size":0,"ondisk_log_size":9689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25138444,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9687,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9618","reported_seq":9750,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.913582+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:52.913582+0000","last_peered":"2024-09-16T04:48:52.913582+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:52.913582+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9618,"log_dups_size":0,"ondisk_log_size":9618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23531410,"num_objects":2873,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9615,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9390","reported_seq":9482,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.870082+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:52.870082+0000","last_peered":"2024-09-16T04:48:52.870082+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:52.870082+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9390,"log_dups_size":0,"ondisk_log_size":9390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24091270,"num_objects":2941,"num_object_clones":0,"num_object_copies":11764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9389,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":836}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9572","reported_seq":9660,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:51.678955+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:51.678955+0000","last_peered":"2024-09-16T04:48:51.678955+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:51.678955+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9572,"log_dups_size":0,"ondisk_log_size":9572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24255110,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9571,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":740}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9747","reported_seq":9879,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.967482+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:52.967482+0000","last_peered":"2024-09-16T04:48:52.967482+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:52.967482+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9747,"log_dups_size":0,"ondisk_log_size":9747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24859916,"num_objects":3035,"num_object_clones":0,"num_object_copies":12140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":837}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9436","reported_seq":9537,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:51.652240+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:51.652240+0000","last_peered":"2024-09-16T04:48:51.652240+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:51.652240+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9436,"log_dups_size":0,"ondisk_log_size":9436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24679692,"num_objects":3013,"num_object_clones":0,"num_object_copies":12052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9434,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":890}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9476","reported_seq":9570,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:52.317124+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:52.317124+0000","last_peered":"2024-09-16T04:48:52.317124+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:52.317124+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9476,"log_dups_size":0,"ondisk_log_size":9476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23188748,"num_objects":2831,"num_object_clones":0,"num_object_copies":11324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9474,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":603}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":388523712,"num_objects":47433,"num_object_clones":0,"num_object_copies":189732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47433,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154054,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5588,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":776404992,"data_stored":776404992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154081,"ondisk_log_size":154081,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704812,"kb_used_data":189972,"kb_used_omap":4,"kb_used_meta":514811,"kb_avail":93667028,"statfs":{"total":96636764160,"available":95915036672,"internally_reserved":0,"allocated":194531328,"data_stored":194342126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":527167183},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,8,6,9,0,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768032,"kb_used_data":191248,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93603808,"statfs":{"total":96636764160,"available":95850299392,"internally_reserved":0,"allocated":195837952,"data_stored":195648750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,0,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613732,"kb_used_data":189140,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93758108,"statfs":{"total":96636764160,"available":96008302592,"internally_reserved":0,"allocated":193679360,"data_stored":193490158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434761423},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247912,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709116,"kb_used_data":190172,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93662724,"statfs":{"total":96636764160,"available":95910629376,"internally_reserved":0,"allocated":194736128,"data_stored":194546926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":531361486},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":194400256,"data_stored":194400256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":193114112,"data_stored":193114112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194899968,"data_stored":194899968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":193990656,"data_stored":193990656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:58.160 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:58.746 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:58.746 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:48:58.772 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-16T04:48:58.714648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":380438208,"num_objects":46446,"num_object_clones":0,"num_object_copies":185784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46446,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155041,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5638,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155068,"ondisk_log_size":155068,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2850184,"kb_used_data":748424,"kb_used_omap":14,"kb_used_meta":2101681,"kb_avail":374637176,"statfs":{"total":386547056640,"available":383628468224,"internally_reserved":0,"allocated":766386176,"data_stored":765629368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2152121905},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,3,11,15,26,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25075712,"num_objects":-3061,"num_object_clones":0,"num_object_copies":-12244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3061,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002919"},"pg_stats":[{"pgid":"1.f","version":"35'9717","reported_seq":9833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.086362+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:58.086362+0000","last_peered":"2024-09-16T04:48:58.086362+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:58.086362+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9717,"log_dups_size":0,"ondisk_log_size":9717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24059904,"num_objects":2937,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9717,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9707","reported_seq":9795,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:57.487337+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:57.487337+0000","last_peered":"2024-09-16T04:48:57.487337+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:57.487337+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24302860,"num_objects":2967,"num_object_clones":0,"num_object_copies":11868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9705,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9705","reported_seq":9803,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:56.765586+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:56.765586+0000","last_peered":"2024-09-16T04:48:56.765586+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:56.765586+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9705,"log_dups_size":0,"ondisk_log_size":9705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24287902,"num_objects":2966,"num_object_clones":0,"num_object_copies":11864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9704,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":747}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9777","reported_seq":11396,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:57.590398+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:57.590398+0000","last_peered":"2024-09-16T04:48:57.590398+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:57.590398+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9777,"log_dups_size":0,"ondisk_log_size":9777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23696652,"num_objects":2893,"num_object_clones":0,"num_object_copies":11572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9775,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":775,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1435}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9832","reported_seq":9965,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.090887+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:58.090887+0000","last_peered":"2024-09-16T04:48:58.090887+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:58.090887+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9832,"log_dups_size":0,"ondisk_log_size":9832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23827724,"num_objects":2909,"num_object_clones":0,"num_object_copies":11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":661}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9494","reported_seq":9604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.047401+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:58.047401+0000","last_peered":"2024-09-16T04:48:58.047401+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:58.047401+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9494,"log_dups_size":0,"ondisk_log_size":9494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23085056,"num_objects":2818,"num_object_clones":0,"num_object_copies":11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":640}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9749","reported_seq":9845,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:57.458088+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:57.458088+0000","last_peered":"2024-09-16T04:48:57.458088+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:57.458088+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9749,"log_dups_size":0,"ondisk_log_size":9749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23352588,"num_objects":2851,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9747,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":638}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9792","reported_seq":14354,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:54.879450+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:54.879450+0000","last_peered":"2024-09-16T04:48:54.879450+0000","last_clean":"2024-09-16T04:48:54.879450+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:54.879450+0000","last_undegraded":"2024-09-16T04:48:54.879450+0000","last_fullsized":"2024-09-16T04:48:54.879450+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9792,"log_dups_size":0,"ondisk_log_size":9792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:57.597397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24515852,"num_objects":2993,"num_object_clones":0,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9790,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9891","reported_seq":14648,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:58.110544+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:58.110544+0000","last_peered":"2024-09-16T04:48:58.110544+0000","last_clean":"2024-09-16T04:48:58.110544+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:58.110544+0000","last_undegraded":"2024-09-16T04:48:58.110544+0000","last_fullsized":"2024-09-16T04:48:58.110544+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:59.843797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23464472,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9888,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9765","reported_seq":9872,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:56.923132+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:56.923132+0000","last_peered":"2024-09-16T04:48:56.923132+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:56.923132+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9765,"log_dups_size":0,"ondisk_log_size":9765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24515852,"num_objects":2993,"num_object_clones":0,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9763,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":804}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9681","reported_seq":9813,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:57.967419+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:57.967419+0000","last_peered":"2024-09-16T04:48:57.967419+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:57.967419+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9681,"log_dups_size":0,"ondisk_log_size":9681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23015314,"num_objects":2810,"num_object_clones":0,"num_object_copies":11240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9468","reported_seq":9560,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.098334+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:58.098334+0000","last_peered":"2024-09-16T04:48:58.098334+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:58.098334+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9468,"log_dups_size":0,"ondisk_log_size":9468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23452294,"num_objects":2863,"num_object_clones":0,"num_object_copies":11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9467,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":758}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9632","reported_seq":9720,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:56.907670+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:56.907670+0000","last_peered":"2024-09-16T04:48:56.907670+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:56.907670+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9632,"log_dups_size":0,"ondisk_log_size":9632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23763590,"num_objects":2901,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9631,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":680}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9819","reported_seq":9951,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.196598+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:58.196598+0000","last_peered":"2024-09-16T04:48:58.196598+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:58.196598+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9819,"log_dups_size":0,"ondisk_log_size":9819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24270092,"num_objects":2963,"num_object_clones":0,"num_object_copies":11852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9817,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":765}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9502","reported_seq":9605,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:56.904276+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:56.904276+0000","last_peered":"2024-09-16T04:48:56.904276+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:56.904276+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9502,"log_dups_size":0,"ondisk_log_size":9502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24139020,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9500,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":824}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9537","reported_seq":9630,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:57.414844+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:57.414844+0000","last_peered":"2024-09-16T04:48:57.414844+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:57.414844+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22689036,"num_objects":2770,"num_object_clones":0,"num_object_copies":11080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9535,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":542}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":380438208,"num_objects":46446,"num_object_clones":0,"num_object_copies":185784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46446,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155041,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5638,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":763699200,"data_stored":763699200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155068,"ondisk_log_size":155068,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722856,"kb_used_data":185816,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93648984,"statfs":{"total":96636764160,"available":95896559616,"internally_reserved":0,"allocated":190275584,"data_stored":190086382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549908175},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,1,5,3,4,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":786368,"kb_used_data":187376,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93585472,"statfs":{"total":96636764160,"available":95831523328,"internally_reserved":0,"allocated":191873024,"data_stored":191683822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613732,"kb_used_data":189140,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93758108,"statfs":{"total":96636764160,"available":96008302592,"internally_reserved":0,"allocated":193679360,"data_stored":193490158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434761423},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247913,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727228,"kb_used_data":186092,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93644612,"statfs":{"total":96636764160,"available":95892082688,"internally_reserved":0,"allocated":190558208,"data_stored":190369006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":554102478},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,2,9,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":190160896,"data_stored":190160896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":193114112,"data_stored":193114112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":190730240,"data_stored":190730240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":189693952,"data_stored":189693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:58.773 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:48:58.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9717", 'reported_seq': 9833, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:58.086362+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:48:58.086362+0000', 'last_peered': '2024-09-16T04:48:58.086362+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:48:58.086362+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9717, 'log_dups_size': 0, 'ondisk_log_size': 9717, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24059904, 'num_objects': 2937, 'num_object_clones': 0, 'num_object_copies': 11748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9717, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 751}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:58.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:48:58.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9707", 'reported_seq': 9795, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:57.487337+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:48:57.487337+0000', 'last_peered': '2024-09-16T04:48:57.487337+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:48:57.487337+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9707, 'log_dups_size': 0, 'ondisk_log_size': 9707, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24302860, 'num_objects': 2967, 'num_object_clones': 0, 'num_object_copies': 11868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9705, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 764}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:58.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:48:58.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9705", 'reported_seq': 9803, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:56.765586+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:48:56.765586+0000', 'last_peered': '2024-09-16T04:48:56.765586+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:48:56.765586+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9705, 'log_dups_size': 0, 'ondisk_log_size': 9705, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24287902, 'num_objects': 2966, 'num_object_clones': 0, 'num_object_copies': 11864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2966, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9704, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 747}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:58.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:48:58.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9777", 'reported_seq': 11396, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:57.590398+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:48:57.590398+0000', 'last_peered': '2024-09-16T04:48:57.590398+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:48:57.590398+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9777, 'log_dups_size': 0, 'ondisk_log_size': 9777, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23696652, 'num_objects': 2893, 'num_object_clones': 0, 'num_object_copies': 11572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1458, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2893, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9775, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 775, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1435}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:58.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:48:58.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9832", 'reported_seq': 9965, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:58.090887+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:48:58.090887+0000', 'last_peered': '2024-09-16T04:48:58.090887+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:48:58.090887+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9832, 'log_dups_size': 0, 'ondisk_log_size': 9832, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23827724, 'num_objects': 2909, 'num_object_clones': 0, 'num_object_copies': 11636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9830, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 661}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:58.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:48:58.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9494", 'reported_seq': 9604, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:58.047401+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:48:58.047401+0000', 'last_peered': '2024-09-16T04:48:58.047401+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:48:58.047401+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9494, 'log_dups_size': 0, 'ondisk_log_size': 9494, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23085056, 'num_objects': 2818, 'num_object_clones': 0, 'num_object_copies': 11272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9494, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 640}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:58.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:48:58.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9749", 'reported_seq': 9845, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:57.458088+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:48:57.458088+0000', 'last_peered': '2024-09-16T04:48:57.458088+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:48:57.458088+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9749, 'log_dups_size': 0, 'ondisk_log_size': 9749, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23352588, 'num_objects': 2851, 'num_object_clones': 0, 'num_object_copies': 11404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9747, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 638}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:58.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:48:58.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9765", 'reported_seq': 9872, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:56.923132+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:48:56.923132+0000', 'last_peered': '2024-09-16T04:48:56.923132+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:48:56.923132+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9765, 'log_dups_size': 0, 'ondisk_log_size': 9765, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24515852, 'num_objects': 2993, 'num_object_clones': 0, 'num_object_copies': 11972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2993, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9763, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 804}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:58.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:48:58.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9681", 'reported_seq': 9813, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:57.967419+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:48:57.967419+0000', 'last_peered': '2024-09-16T04:48:57.967419+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:48:57.967419+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9681, 'log_dups_size': 0, 'ondisk_log_size': 9681, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23015314, 'num_objects': 2810, 'num_object_clones': 0, 'num_object_copies': 11240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9678, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 655}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:58.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:48:58.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9468", 'reported_seq': 9560, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:58.098334+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:48:58.098334+0000', 'last_peered': '2024-09-16T04:48:58.098334+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:48:58.098334+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9468, 'log_dups_size': 0, 'ondisk_log_size': 9468, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23452294, 'num_objects': 2863, 'num_object_clones': 0, 'num_object_copies': 11452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9467, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 758}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:58.776 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:48:58.776 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9632", 'reported_seq': 9720, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:56.907670+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:48:56.907670+0000', 'last_peered': '2024-09-16T04:48:56.907670+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:48:56.907670+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9632, 'log_dups_size': 0, 'ondisk_log_size': 9632, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23763590, 'num_objects': 2901, 'num_object_clones': 0, 'num_object_copies': 11604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2901, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9631, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 680}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:58.776 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:48:58.776 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9819", 'reported_seq': 9951, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:58.196598+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:48:58.196598+0000', 'last_peered': '2024-09-16T04:48:58.196598+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:48:58.196598+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9819, 'log_dups_size': 0, 'ondisk_log_size': 9819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24270092, 'num_objects': 2963, 'num_object_clones': 0, 'num_object_copies': 11852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2963, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9817, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 765}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:48:58.776 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:48:58.776 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9502", 'reported_seq': 9605, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:56.904276+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:48:56.904276+0000', 'last_peered': '2024-09-16T04:48:56.904276+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:48:56.904276+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9502, 'log_dups_size': 0, 'ondisk_log_size': 9502, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24139020, 'num_objects': 2947, 'num_object_clones': 0, 'num_object_copies': 11788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9500, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 824}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:48:58.776 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:48:58.776 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9537", 'reported_seq': 9630, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:57.414844+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:48:57.414844+0000', 'last_peered': '2024-09-16T04:48:57.414844+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:48:57.414844+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9537, 'log_dups_size': 0, 'ondisk_log_size': 9537, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22689036, 'num_objects': 2770, 'num_object_clones': 0, 'num_object_copies': 11080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2770, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9535, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 542}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:48:58.777 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:48:59.516 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:48:59.516 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29944/185784 objects degraded (16.118%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":551,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":46446,"data_bytes":380438208,"bytes_used":2918588416,"bytes_avail":383628468224,"bytes_total":386547056640,"degraded_objects":29944,"degraded_total":185784,"degraded_ratio":0.16117641992851917,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":255},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:48:59.545 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:48:59.546 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:00.193 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:00.194 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:00.215 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-16T04:48:58.714648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":380438208,"num_objects":46446,"num_object_clones":0,"num_object_copies":185784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46446,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155041,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5638,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155068,"ondisk_log_size":155068,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2850184,"kb_used_data":748424,"kb_used_omap":14,"kb_used_meta":2101681,"kb_avail":374637176,"statfs":{"total":386547056640,"available":383628468224,"internally_reserved":0,"allocated":766386176,"data_stored":765629368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2152121905},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,3,11,15,26,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25075712,"num_objects":-3061,"num_object_clones":0,"num_object_copies":-12244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3061,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002919"},"pg_stats":[{"pgid":"1.f","version":"35'9717","reported_seq":9833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.086362+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:58.086362+0000","last_peered":"2024-09-16T04:48:58.086362+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:58.086362+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9717,"log_dups_size":0,"ondisk_log_size":9717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24059904,"num_objects":2937,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9717,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9707","reported_seq":9795,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:57.487337+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:48:57.487337+0000","last_peered":"2024-09-16T04:48:57.487337+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:48:57.487337+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24302860,"num_objects":2967,"num_object_clones":0,"num_object_copies":11868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9705,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9705","reported_seq":9803,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:56.765586+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:48:56.765586+0000","last_peered":"2024-09-16T04:48:56.765586+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:48:56.765586+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9705,"log_dups_size":0,"ondisk_log_size":9705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24287902,"num_objects":2966,"num_object_clones":0,"num_object_copies":11864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9704,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":747}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9777","reported_seq":11396,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:57.590398+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:48:57.590398+0000","last_peered":"2024-09-16T04:48:57.590398+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:48:57.590398+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9777,"log_dups_size":0,"ondisk_log_size":9777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23696652,"num_objects":2893,"num_object_clones":0,"num_object_copies":11572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9775,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":775,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1435}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9832","reported_seq":9965,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.090887+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:58.090887+0000","last_peered":"2024-09-16T04:48:58.090887+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:58.090887+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9832,"log_dups_size":0,"ondisk_log_size":9832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23827724,"num_objects":2909,"num_object_clones":0,"num_object_copies":11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":661}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9494","reported_seq":9604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.047401+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:58.047401+0000","last_peered":"2024-09-16T04:48:58.047401+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:58.047401+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9494,"log_dups_size":0,"ondisk_log_size":9494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23085056,"num_objects":2818,"num_object_clones":0,"num_object_copies":11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":640}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9749","reported_seq":9845,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:57.458088+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:48:57.458088+0000","last_peered":"2024-09-16T04:48:57.458088+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:48:57.458088+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9749,"log_dups_size":0,"ondisk_log_size":9749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23352588,"num_objects":2851,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9747,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":638}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9792","reported_seq":14354,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:54.879450+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:54.879450+0000","last_peered":"2024-09-16T04:48:54.879450+0000","last_clean":"2024-09-16T04:48:54.879450+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:54.879450+0000","last_undegraded":"2024-09-16T04:48:54.879450+0000","last_fullsized":"2024-09-16T04:48:54.879450+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9792,"log_dups_size":0,"ondisk_log_size":9792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:57.597397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24515852,"num_objects":2993,"num_object_clones":0,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9790,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9891","reported_seq":14648,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:58.110544+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:58.110544+0000","last_peered":"2024-09-16T04:48:58.110544+0000","last_clean":"2024-09-16T04:48:58.110544+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:58.110544+0000","last_undegraded":"2024-09-16T04:48:58.110544+0000","last_fullsized":"2024-09-16T04:48:58.110544+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:59.843797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23464472,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9888,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9765","reported_seq":9872,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:56.923132+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:48:56.923132+0000","last_peered":"2024-09-16T04:48:56.923132+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:48:56.923132+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9765,"log_dups_size":0,"ondisk_log_size":9765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24515852,"num_objects":2993,"num_object_clones":0,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9763,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":804}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9681","reported_seq":9813,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:57.967419+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:57.967419+0000","last_peered":"2024-09-16T04:48:57.967419+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:57.967419+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9681,"log_dups_size":0,"ondisk_log_size":9681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23015314,"num_objects":2810,"num_object_clones":0,"num_object_copies":11240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9468","reported_seq":9560,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.098334+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:58.098334+0000","last_peered":"2024-09-16T04:48:58.098334+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:58.098334+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9468,"log_dups_size":0,"ondisk_log_size":9468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23452294,"num_objects":2863,"num_object_clones":0,"num_object_copies":11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9467,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":758}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9632","reported_seq":9720,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:56.907670+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:48:56.907670+0000","last_peered":"2024-09-16T04:48:56.907670+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:48:56.907670+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9632,"log_dups_size":0,"ondisk_log_size":9632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23763590,"num_objects":2901,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9631,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":680}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9819","reported_seq":9951,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.196598+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:58.196598+0000","last_peered":"2024-09-16T04:48:58.196598+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:58.196598+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9819,"log_dups_size":0,"ondisk_log_size":9819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24270092,"num_objects":2963,"num_object_clones":0,"num_object_copies":11852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9817,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":765}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9502","reported_seq":9605,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:56.904276+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:48:56.904276+0000","last_peered":"2024-09-16T04:48:56.904276+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:48:56.904276+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9502,"log_dups_size":0,"ondisk_log_size":9502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24139020,"num_objects":2947,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9500,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":824}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9537","reported_seq":9630,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:57.414844+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:48:57.414844+0000","last_peered":"2024-09-16T04:48:57.414844+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:48:57.414844+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22689036,"num_objects":2770,"num_object_clones":0,"num_object_copies":11080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9535,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":542}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":380438208,"num_objects":46446,"num_object_clones":0,"num_object_copies":185784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46446,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155041,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5638,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":763699200,"data_stored":763699200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155068,"ondisk_log_size":155068,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722856,"kb_used_data":185816,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93648984,"statfs":{"total":96636764160,"available":95896559616,"internally_reserved":0,"allocated":190275584,"data_stored":190086382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549908175},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,1,5,3,4,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":786368,"kb_used_data":187376,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93585472,"statfs":{"total":96636764160,"available":95831523328,"internally_reserved":0,"allocated":191873024,"data_stored":191683822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613732,"kb_used_data":189140,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93758108,"statfs":{"total":96636764160,"available":96008302592,"internally_reserved":0,"allocated":193679360,"data_stored":193490158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434761423},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247913,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727228,"kb_used_data":186092,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93644612,"statfs":{"total":96636764160,"available":95892082688,"internally_reserved":0,"allocated":190558208,"data_stored":190369006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":554102478},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,2,9,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":190160896,"data_stored":190160896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":193114112,"data_stored":193114112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":190730240,"data_stored":190730240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":189693952,"data_stored":189693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:03.217 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:03.842 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:03.842 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:03.865 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-16T04:49:02.715644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376170176,"num_objects":45925,"num_object_clones":0,"num_object_copies":183700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45925,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155562,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5687,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155589,"ondisk_log_size":155589,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2833332,"kb_used_data":735692,"kb_used_omap":14,"kb_used_meta":2097521,"kb_avail":374654028,"statfs":{"total":386547056640,"available":383645724672,"internally_reserved":0,"allocated":753348608,"data_stored":752591800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2147862065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,6,2,16,11,14,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20905984,"num_objects":-2552,"num_object_clones":0,"num_object_copies":-10208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2552,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002924"},"pg_stats":[{"pgid":"1.f","version":"35'9717","reported_seq":9833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.086362+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:58.086362+0000","last_peered":"2024-09-16T04:48:58.086362+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:58.086362+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9717,"log_dups_size":0,"ondisk_log_size":9717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24059904,"num_objects":2937,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9717,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9770","reported_seq":9858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:02.311513+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:02.311513+0000","last_peered":"2024-09-16T04:49:02.311513+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:02.311513+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9770,"log_dups_size":0,"ondisk_log_size":9770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23786764,"num_objects":2904,"num_object_clones":0,"num_object_copies":11616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9768,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9771","reported_seq":9868,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:01.875453+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:01.875453+0000","last_peered":"2024-09-16T04:49:01.875453+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:01.875453+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9771,"log_dups_size":0,"ondisk_log_size":9771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23747230,"num_objects":2900,"num_object_clones":0,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9770,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":681}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9843","reported_seq":11558,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:02.672264+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:02.672264+0000","last_peered":"2024-09-16T04:49:02.672264+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:02.672264+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9843,"log_dups_size":0,"ondisk_log_size":9843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23155980,"num_objects":2827,"num_object_clones":0,"num_object_copies":11308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9841,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":824,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1418}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9832","reported_seq":9965,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.090887+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:58.090887+0000","last_peered":"2024-09-16T04:48:58.090887+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:58.090887+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9832,"log_dups_size":0,"ondisk_log_size":9832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23827724,"num_objects":2909,"num_object_clones":0,"num_object_copies":11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":661}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9494","reported_seq":9604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.047401+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:58.047401+0000","last_peered":"2024-09-16T04:48:58.047401+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:58.047401+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9494,"log_dups_size":0,"ondisk_log_size":9494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23085056,"num_objects":2818,"num_object_clones":0,"num_object_copies":11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":640}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9806","reported_seq":9904,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:02.628517+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:02.628517+0000","last_peered":"2024-09-16T04:49:02.628517+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:02.628517+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9806,"log_dups_size":0,"ondisk_log_size":9806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22885644,"num_objects":2794,"num_object_clones":0,"num_object_copies":11176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9804,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":581}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9848","reported_seq":14410,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:59.830643+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:59.830643+0000","last_peered":"2024-09-16T04:48:59.830643+0000","last_clean":"2024-09-16T04:48:59.830643+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:59.830643+0000","last_undegraded":"2024-09-16T04:48:59.830643+0000","last_fullsized":"2024-09-16T04:48:59.830643+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9848,"log_dups_size":0,"ondisk_log_size":9848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:02.644769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24057100,"num_objects":2937,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9846,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9891","reported_seq":14648,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:58.110544+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:58.110544+0000","last_peered":"2024-09-16T04:48:58.110544+0000","last_clean":"2024-09-16T04:48:58.110544+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:58.110544+0000","last_undegraded":"2024-09-16T04:48:58.110544+0000","last_fullsized":"2024-09-16T04:48:58.110544+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:59.843797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23464472,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9888,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9817","reported_seq":9924,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:01.962444+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:01.962444+0000","last_peered":"2024-09-16T04:49:01.962444+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:01.962444+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9817,"log_dups_size":0,"ondisk_log_size":9817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24089868,"num_objects":2941,"num_object_clones":0,"num_object_copies":11764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9815,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":752}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9681","reported_seq":9813,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:57.967419+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:57.967419+0000","last_peered":"2024-09-16T04:48:57.967419+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:57.967419+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9681,"log_dups_size":0,"ondisk_log_size":9681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23015314,"num_objects":2810,"num_object_clones":0,"num_object_copies":11240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9468","reported_seq":9560,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.098334+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:58.098334+0000","last_peered":"2024-09-16T04:48:58.098334+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:58.098334+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9468,"log_dups_size":0,"ondisk_log_size":9468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23452294,"num_objects":2863,"num_object_clones":0,"num_object_copies":11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9467,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":758}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9690","reported_seq":9778,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:01.880139+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:01.880139+0000","last_peered":"2024-09-16T04:49:01.880139+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:01.880139+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9690,"log_dups_size":0,"ondisk_log_size":9690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23288454,"num_objects":2843,"num_object_clones":0,"num_object_copies":11372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9689,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":622}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9819","reported_seq":9951,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.196598+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:58.196598+0000","last_peered":"2024-09-16T04:48:58.196598+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:58.196598+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9819,"log_dups_size":0,"ondisk_log_size":9819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24270092,"num_objects":2963,"num_object_clones":0,"num_object_copies":11852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9817,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":765}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9559","reported_seq":9661,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:01.684686+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:01.684686+0000","last_peered":"2024-09-16T04:49:01.684686+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:01.684686+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9559,"log_dups_size":0,"ondisk_log_size":9559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23672076,"num_objects":2890,"num_object_clones":0,"num_object_copies":11560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9557,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9583","reported_seq":9677,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:02.537718+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:02.537718+0000","last_peered":"2024-09-16T04:49:02.537718+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:02.537718+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9583,"log_dups_size":0,"ondisk_log_size":9583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22312204,"num_objects":2724,"num_object_clones":0,"num_object_copies":10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9581,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":376170176,"num_objects":45925,"num_object_clones":0,"num_object_copies":183700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45925,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155562,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5687,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":751366144,"data_stored":751366144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155589,"ondisk_log_size":155589,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722856,"kb_used_data":185816,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93648984,"statfs":{"total":96636764160,"available":95896559616,"internally_reserved":0,"allocated":190275584,"data_stored":190086382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549908175},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,1,5,3,4,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769112,"kb_used_data":182588,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93602728,"statfs":{"total":96636764160,"available":95849193472,"internally_reserved":0,"allocated":186970112,"data_stored":186780910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,5,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051675,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613876,"kb_used_data":185112,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93757964,"statfs":{"total":96636764160,"available":96008155136,"internally_reserved":0,"allocated":189554688,"data_stored":189365486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":439021263},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727488,"kb_used_data":182176,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93644352,"statfs":{"total":96636764160,"available":95891816448,"internally_reserved":0,"allocated":186548224,"data_stored":186359022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":558362318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":186064896,"data_stored":186064896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":189009920,"data_stored":189009920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186597376,"data_stored":186597376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":189693952,"data_stored":189693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:03.867 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:04.463 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:04.465 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:04.489 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-16T04:49:02.715644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376170176,"num_objects":45925,"num_object_clones":0,"num_object_copies":183700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45925,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155562,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5687,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155589,"ondisk_log_size":155589,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2833332,"kb_used_data":735692,"kb_used_omap":14,"kb_used_meta":2097521,"kb_avail":374654028,"statfs":{"total":386547056640,"available":383645724672,"internally_reserved":0,"allocated":753348608,"data_stored":752591800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2147862065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,6,2,16,11,14,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20905984,"num_objects":-2552,"num_object_clones":0,"num_object_copies":-10208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2552,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002924"},"pg_stats":[{"pgid":"1.f","version":"35'9717","reported_seq":9833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.086362+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:48:58.086362+0000","last_peered":"2024-09-16T04:48:58.086362+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:48:58.086362+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9717,"log_dups_size":0,"ondisk_log_size":9717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24059904,"num_objects":2937,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9717,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9770","reported_seq":9858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:02.311513+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:02.311513+0000","last_peered":"2024-09-16T04:49:02.311513+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:02.311513+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9770,"log_dups_size":0,"ondisk_log_size":9770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23786764,"num_objects":2904,"num_object_clones":0,"num_object_copies":11616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9768,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9771","reported_seq":9868,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:01.875453+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:01.875453+0000","last_peered":"2024-09-16T04:49:01.875453+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:01.875453+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9771,"log_dups_size":0,"ondisk_log_size":9771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23747230,"num_objects":2900,"num_object_clones":0,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9770,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":681}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9843","reported_seq":11558,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:02.672264+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:02.672264+0000","last_peered":"2024-09-16T04:49:02.672264+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:02.672264+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9843,"log_dups_size":0,"ondisk_log_size":9843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23155980,"num_objects":2827,"num_object_clones":0,"num_object_copies":11308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9841,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":824,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1418}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9832","reported_seq":9965,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.090887+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:48:58.090887+0000","last_peered":"2024-09-16T04:48:58.090887+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:48:58.090887+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9832,"log_dups_size":0,"ondisk_log_size":9832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23827724,"num_objects":2909,"num_object_clones":0,"num_object_copies":11636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":661}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9494","reported_seq":9604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.047401+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:48:58.047401+0000","last_peered":"2024-09-16T04:48:58.047401+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:48:58.047401+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9494,"log_dups_size":0,"ondisk_log_size":9494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23085056,"num_objects":2818,"num_object_clones":0,"num_object_copies":11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":640}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9806","reported_seq":9904,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:02.628517+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:02.628517+0000","last_peered":"2024-09-16T04:49:02.628517+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:02.628517+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9806,"log_dups_size":0,"ondisk_log_size":9806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22885644,"num_objects":2794,"num_object_clones":0,"num_object_copies":11176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9804,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":581}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9848","reported_seq":14410,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:59.830643+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:59.830643+0000","last_peered":"2024-09-16T04:48:59.830643+0000","last_clean":"2024-09-16T04:48:59.830643+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:59.830643+0000","last_undegraded":"2024-09-16T04:48:59.830643+0000","last_fullsized":"2024-09-16T04:48:59.830643+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9848,"log_dups_size":0,"ondisk_log_size":9848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:02.644769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24057100,"num_objects":2937,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9846,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9891","reported_seq":14648,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:58.110544+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:48:58.110544+0000","last_peered":"2024-09-16T04:48:58.110544+0000","last_clean":"2024-09-16T04:48:58.110544+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:48:58.110544+0000","last_undegraded":"2024-09-16T04:48:58.110544+0000","last_fullsized":"2024-09-16T04:48:58.110544+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:48:59.843797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23464472,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9888,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9817","reported_seq":9924,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:01.962444+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:01.962444+0000","last_peered":"2024-09-16T04:49:01.962444+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:01.962444+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9817,"log_dups_size":0,"ondisk_log_size":9817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24089868,"num_objects":2941,"num_object_clones":0,"num_object_copies":11764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9815,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":752}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9681","reported_seq":9813,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:57.967419+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:48:57.967419+0000","last_peered":"2024-09-16T04:48:57.967419+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:48:57.967419+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9681,"log_dups_size":0,"ondisk_log_size":9681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23015314,"num_objects":2810,"num_object_clones":0,"num_object_copies":11240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9468","reported_seq":9560,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.098334+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:48:58.098334+0000","last_peered":"2024-09-16T04:48:58.098334+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:48:58.098334+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9468,"log_dups_size":0,"ondisk_log_size":9468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23452294,"num_objects":2863,"num_object_clones":0,"num_object_copies":11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9467,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":758}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9690","reported_seq":9778,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:01.880139+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:01.880139+0000","last_peered":"2024-09-16T04:49:01.880139+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:01.880139+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9690,"log_dups_size":0,"ondisk_log_size":9690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23288454,"num_objects":2843,"num_object_clones":0,"num_object_copies":11372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9689,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":622}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9819","reported_seq":9951,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:48:58.196598+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:48:58.196598+0000","last_peered":"2024-09-16T04:48:58.196598+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:48:58.196598+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9819,"log_dups_size":0,"ondisk_log_size":9819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24270092,"num_objects":2963,"num_object_clones":0,"num_object_copies":11852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9817,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":765}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9559","reported_seq":9661,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:01.684686+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:01.684686+0000","last_peered":"2024-09-16T04:49:01.684686+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:01.684686+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9559,"log_dups_size":0,"ondisk_log_size":9559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23672076,"num_objects":2890,"num_object_clones":0,"num_object_copies":11560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9557,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9583","reported_seq":9677,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:02.537718+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:02.537718+0000","last_peered":"2024-09-16T04:49:02.537718+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:02.537718+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9583,"log_dups_size":0,"ondisk_log_size":9583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22312204,"num_objects":2724,"num_object_clones":0,"num_object_copies":10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9581,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":376170176,"num_objects":45925,"num_object_clones":0,"num_object_copies":183700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45925,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155562,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5687,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":751366144,"data_stored":751366144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155589,"ondisk_log_size":155589,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722856,"kb_used_data":185816,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93648984,"statfs":{"total":96636764160,"available":95896559616,"internally_reserved":0,"allocated":190275584,"data_stored":190086382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549908175},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,1,5,3,4,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769112,"kb_used_data":182588,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93602728,"statfs":{"total":96636764160,"available":95849193472,"internally_reserved":0,"allocated":186970112,"data_stored":186780910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,5,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051675,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613876,"kb_used_data":185112,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93757964,"statfs":{"total":96636764160,"available":96008155136,"internally_reserved":0,"allocated":189554688,"data_stored":189365486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":439021263},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727488,"kb_used_data":182176,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93644352,"statfs":{"total":96636764160,"available":95891816448,"internally_reserved":0,"allocated":186548224,"data_stored":186359022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":558362318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":186064896,"data_stored":186064896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":189009920,"data_stored":189009920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186597376,"data_stored":186597376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":189693952,"data_stored":189693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:04.490 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:49:04.490 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9717", 'reported_seq': 9833, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:58.086362+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:48:58.086362+0000', 'last_peered': '2024-09-16T04:48:58.086362+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:48:58.086362+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9717, 'log_dups_size': 0, 'ondisk_log_size': 9717, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24059904, 'num_objects': 2937, 'num_object_clones': 0, 'num_object_copies': 11748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9717, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 751}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:04.490 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:49:04.491 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9770", 'reported_seq': 9858, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:02.311513+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:49:02.311513+0000', 'last_peered': '2024-09-16T04:49:02.311513+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:49:02.311513+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9770, 'log_dups_size': 0, 'ondisk_log_size': 9770, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23786764, 'num_objects': 2904, 'num_object_clones': 0, 'num_object_copies': 11616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2904, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9768, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 701}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:04.491 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:49:04.491 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9771", 'reported_seq': 9868, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:01.875453+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:49:01.875453+0000', 'last_peered': '2024-09-16T04:49:01.875453+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:49:01.875453+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9771, 'log_dups_size': 0, 'ondisk_log_size': 9771, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23747230, 'num_objects': 2900, 'num_object_clones': 0, 'num_object_copies': 11600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2900, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9770, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 681}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:04.491 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:49:04.491 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9843", 'reported_seq': 11558, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:02.672264+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:49:02.672264+0000', 'last_peered': '2024-09-16T04:49:02.672264+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:49:02.672264+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9843, 'log_dups_size': 0, 'ondisk_log_size': 9843, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23155980, 'num_objects': 2827, 'num_object_clones': 0, 'num_object_copies': 11308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1409, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2827, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9841, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 824, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1418}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:04.491 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:49:04.491 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9832", 'reported_seq': 9965, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:58.090887+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:48:58.090887+0000', 'last_peered': '2024-09-16T04:48:58.090887+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:48:58.090887+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9832, 'log_dups_size': 0, 'ondisk_log_size': 9832, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23827724, 'num_objects': 2909, 'num_object_clones': 0, 'num_object_copies': 11636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9830, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 661}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:04.491 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:49:04.491 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9494", 'reported_seq': 9604, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:58.047401+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:48:58.047401+0000', 'last_peered': '2024-09-16T04:48:58.047401+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:48:58.047401+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9494, 'log_dups_size': 0, 'ondisk_log_size': 9494, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23085056, 'num_objects': 2818, 'num_object_clones': 0, 'num_object_copies': 11272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9494, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 640}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:04.492 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:49:04.492 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9806", 'reported_seq': 9904, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:02.628517+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:49:02.628517+0000', 'last_peered': '2024-09-16T04:49:02.628517+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:49:02.628517+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9806, 'log_dups_size': 0, 'ondisk_log_size': 9806, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22885644, 'num_objects': 2794, 'num_object_clones': 0, 'num_object_copies': 11176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2794, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9804, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 581}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:04.492 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:49:04.492 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9817", 'reported_seq': 9924, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:01.962444+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:49:01.962444+0000', 'last_peered': '2024-09-16T04:49:01.962444+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:49:01.962444+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9817, 'log_dups_size': 0, 'ondisk_log_size': 9817, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24089868, 'num_objects': 2941, 'num_object_clones': 0, 'num_object_copies': 11764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9815, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 752}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:04.492 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:49:04.492 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9681", 'reported_seq': 9813, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:57.967419+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:48:57.967419+0000', 'last_peered': '2024-09-16T04:48:57.967419+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:48:57.967419+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9681, 'log_dups_size': 0, 'ondisk_log_size': 9681, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23015314, 'num_objects': 2810, 'num_object_clones': 0, 'num_object_copies': 11240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9678, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 655}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:04.492 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:49:04.492 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9468", 'reported_seq': 9560, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:58.098334+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:48:58.098334+0000', 'last_peered': '2024-09-16T04:48:58.098334+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:48:58.098334+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9468, 'log_dups_size': 0, 'ondisk_log_size': 9468, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23452294, 'num_objects': 2863, 'num_object_clones': 0, 'num_object_copies': 11452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9467, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 758}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:04.492 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:49:04.492 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9690", 'reported_seq': 9778, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:01.880139+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:49:01.880139+0000', 'last_peered': '2024-09-16T04:49:01.880139+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:49:01.880139+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9690, 'log_dups_size': 0, 'ondisk_log_size': 9690, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23288454, 'num_objects': 2843, 'num_object_clones': 0, 'num_object_copies': 11372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9689, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 622}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:04.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:49:04.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9819", 'reported_seq': 9951, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:48:58.196598+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:48:58.196598+0000', 'last_peered': '2024-09-16T04:48:58.196598+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:48:58.196598+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9819, 'log_dups_size': 0, 'ondisk_log_size': 9819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24270092, 'num_objects': 2963, 'num_object_clones': 0, 'num_object_copies': 11852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2963, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9817, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 765}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:04.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:49:04.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9559", 'reported_seq': 9661, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:01.684686+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:49:01.684686+0000', 'last_peered': '2024-09-16T04:49:01.684686+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:49:01.684686+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9559, 'log_dups_size': 0, 'ondisk_log_size': 9559, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23672076, 'num_objects': 2890, 'num_object_clones': 0, 'num_object_copies': 11560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9557, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 767}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:04.493 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:49:04.493 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9583", 'reported_seq': 9677, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:02.537718+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:49:02.537718+0000', 'last_peered': '2024-09-16T04:49:02.537718+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:49:02.537718+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9583, 'log_dups_size': 0, 'ondisk_log_size': 9583, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22312204, 'num_objects': 2724, 'num_object_clones': 0, 'num_object_copies': 10896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9581, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 496}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:04.493 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:49:05.205 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:05.206 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29895/183700 objects degraded (16.274%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":557,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":45925,"data_bytes":376170176,"bytes_used":2901331968,"bytes_avail":383645724672,"bytes_total":386547056640,"degraded_objects":29895,"degraded_total":183700,"degraded_ratio":0.16273816004354927,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":212},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:49:05.243 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:49:05.243 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:05.863 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:05.863 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:05.898 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-16T04:49:04.716154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":372451008,"num_objects":45471,"num_object_clones":0,"num_object_copies":181884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156016,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5687,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156043,"ondisk_log_size":156043,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2833512,"kb_used_data":731696,"kb_used_omap":14,"kb_used_meta":2101681,"kb_avail":374653848,"statfs":{"total":386547056640,"available":383645540352,"internally_reserved":0,"allocated":749256704,"data_stored":748499896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2152121905},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,1,11,13,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20389888,"num_objects":-2489,"num_object_clones":0,"num_object_copies":-9956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2489,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002963"},"pg_stats":[{"pgid":"1.f","version":"35'9774","reported_seq":9889,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:03.169853+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:03.169853+0000","last_peered":"2024-09-16T04:49:03.169853+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:03.169853+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9774,"log_dups_size":0,"ondisk_log_size":9774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23592960,"num_objects":2880,"num_object_clones":0,"num_object_copies":11520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9774,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9770","reported_seq":9858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:02.311513+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:02.311513+0000","last_peered":"2024-09-16T04:49:02.311513+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:02.311513+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9770,"log_dups_size":0,"ondisk_log_size":9770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23786764,"num_objects":2904,"num_object_clones":0,"num_object_copies":11616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9768,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9771","reported_seq":9868,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:01.875453+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:01.875453+0000","last_peered":"2024-09-16T04:49:01.875453+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:01.875453+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9771,"log_dups_size":0,"ondisk_log_size":9771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23747230,"num_objects":2900,"num_object_clones":0,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9770,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":681}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9843","reported_seq":11558,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:02.672264+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:02.672264+0000","last_peered":"2024-09-16T04:49:02.672264+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:02.672264+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9843,"log_dups_size":0,"ondisk_log_size":9843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23155980,"num_objects":2827,"num_object_clones":0,"num_object_copies":11308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9841,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":824,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1418}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9889","reported_seq":10022,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:03.155346+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:03.155346+0000","last_peered":"2024-09-16T04:49:03.155346+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:03.155346+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9889,"log_dups_size":0,"ondisk_log_size":9889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23360780,"num_objects":2852,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9887,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9577","reported_seq":9687,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:03.090856+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:03.090856+0000","last_peered":"2024-09-16T04:49:03.090856+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:03.090856+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9577,"log_dups_size":0,"ondisk_log_size":9577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22405120,"num_objects":2735,"num_object_clones":0,"num_object_copies":10940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9577,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":557}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9806","reported_seq":9904,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:02.628517+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:02.628517+0000","last_peered":"2024-09-16T04:49:02.628517+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:02.628517+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9806,"log_dups_size":0,"ondisk_log_size":9806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22885644,"num_objects":2794,"num_object_clones":0,"num_object_copies":11176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9804,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":581}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9848","reported_seq":14410,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:48:59.830643+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:48:59.830643+0000","last_peered":"2024-09-16T04:48:59.830643+0000","last_clean":"2024-09-16T04:48:59.830643+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:48:59.830643+0000","last_undegraded":"2024-09-16T04:48:59.830643+0000","last_fullsized":"2024-09-16T04:48:59.830643+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9848,"log_dups_size":0,"ondisk_log_size":9848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:02.644769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24057100,"num_objects":2937,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9846,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'9947","reported_seq":14704,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:03.205506+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:03.205506+0000","last_peered":"2024-09-16T04:49:03.205506+0000","last_clean":"2024-09-16T04:49:03.205506+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:03.205506+0000","last_undegraded":"2024-09-16T04:49:03.205506+0000","last_fullsized":"2024-09-16T04:49:03.205506+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9947,"log_dups_size":0,"ondisk_log_size":9947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:05.874125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23005720,"num_objects":2809,"num_object_clones":0,"num_object_copies":11236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9944,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9817","reported_seq":9924,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:01.962444+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:01.962444+0000","last_peered":"2024-09-16T04:49:01.962444+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:01.962444+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9817,"log_dups_size":0,"ondisk_log_size":9817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24089868,"num_objects":2941,"num_object_clones":0,"num_object_copies":11764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9815,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":752}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9732","reported_seq":9864,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:03.184762+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:03.184762+0000","last_peered":"2024-09-16T04:49:03.184762+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:03.184762+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9732,"log_dups_size":0,"ondisk_log_size":9732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22597522,"num_objects":2759,"num_object_clones":0,"num_object_copies":11036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9729,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9539","reported_seq":9631,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:03.165718+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:03.165718+0000","last_peered":"2024-09-16T04:49:03.165718+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:03.165718+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9539,"log_dups_size":0,"ondisk_log_size":9539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22870662,"num_objects":2792,"num_object_clones":0,"num_object_copies":11168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9538,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":687}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9690","reported_seq":9778,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:01.880139+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:01.880139+0000","last_peered":"2024-09-16T04:49:01.880139+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:01.880139+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9690,"log_dups_size":0,"ondisk_log_size":9690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23288454,"num_objects":2843,"num_object_clones":0,"num_object_copies":11372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9689,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":622}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9898","reported_seq":10030,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:03.156410+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:03.156410+0000","last_peered":"2024-09-16T04:49:03.156410+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:03.156410+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9898,"log_dups_size":0,"ondisk_log_size":9898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23622924,"num_objects":2884,"num_object_clones":0,"num_object_copies":11536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9896,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":686}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9559","reported_seq":9661,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:01.684686+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:01.684686+0000","last_peered":"2024-09-16T04:49:01.684686+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:01.684686+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9559,"log_dups_size":0,"ondisk_log_size":9559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23672076,"num_objects":2890,"num_object_clones":0,"num_object_copies":11560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9557,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9583","reported_seq":9677,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:02.537718+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:02.537718+0000","last_peered":"2024-09-16T04:49:02.537718+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:02.537718+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9583,"log_dups_size":0,"ondisk_log_size":9583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22312204,"num_objects":2724,"num_object_clones":0,"num_object_copies":10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9581,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":372451008,"num_objects":45471,"num_object_clones":0,"num_object_copies":181884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156016,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5687,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":747208704,"data_stored":747208704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156043,"ondisk_log_size":156043,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723036,"kb_used_data":181820,"kb_used_omap":4,"kb_used_meta":541179,"kb_avail":93648804,"statfs":{"total":96636764160,"available":95896375296,"internally_reserved":0,"allocated":186183680,"data_stored":185994478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554168015},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769112,"kb_used_data":182588,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93602728,"statfs":{"total":96636764160,"available":95849193472,"internally_reserved":0,"allocated":186970112,"data_stored":186780910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,5,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051675,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613876,"kb_used_data":185112,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93757964,"statfs":{"total":96636764160,"available":96008155136,"internally_reserved":0,"allocated":189554688,"data_stored":189365486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":439021263},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727488,"kb_used_data":182176,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93644352,"statfs":{"total":96636764160,"available":95891816448,"internally_reserved":0,"allocated":186548224,"data_stored":186359022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":558362318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":186064896,"data_stored":186064896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":189009920,"data_stored":189009920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186597376,"data_stored":186597376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":185536512,"data_stored":185536512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:08.901 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:09.550 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:09.551 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:09.578 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-09-16T04:49:08.717122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":363931328,"num_objects":44431,"num_object_clones":0,"num_object_copies":177724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44431,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157056,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5737,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157083,"ondisk_log_size":157083,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2925176,"kb_used_data":716500,"kb_used_omap":14,"kb_used_meta":2208561,"kb_avail":374562184,"statfs":{"total":386547056640,"available":383551676416,"internally_reserved":0,"allocated":733696000,"data_stored":732939192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2261567025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,11,14,9,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24592384,"num_objects":-3002,"num_object_clones":0,"num_object_copies":-12008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3002,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002948"},"pg_stats":[{"pgid":"1.f","version":"35'9842","reported_seq":9957,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:08.193955+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:08.193955+0000","last_peered":"2024-09-16T04:49:08.193955+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:08.193955+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9842,"log_dups_size":0,"ondisk_log_size":9842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23035904,"num_objects":2812,"num_object_clones":0,"num_object_copies":11248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9842,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9836","reported_seq":9924,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.528010+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:07.528010+0000","last_peered":"2024-09-16T04:49:07.528010+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:07.528010+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9836,"log_dups_size":0,"ondisk_log_size":9836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23246092,"num_objects":2838,"num_object_clones":0,"num_object_copies":11352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9834,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":635}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9850","reported_seq":9948,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.010322+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:07.010322+0000","last_peered":"2024-09-16T04:49:07.010322+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:07.010322+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23100062,"num_objects":2821,"num_object_clones":0,"num_object_copies":11284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2821,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9849,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":602}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9904","reported_seq":11722,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.630230+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:07.630230+0000","last_peered":"2024-09-16T04:49:07.630230+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:07.630230+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22656268,"num_objects":2766,"num_object_clones":0,"num_object_copies":11064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9902,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1407}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9951","reported_seq":10084,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.937531+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:07.937531+0000","last_peered":"2024-09-16T04:49:07.937531+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:07.937531+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9951,"log_dups_size":0,"ondisk_log_size":9951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22852876,"num_objects":2790,"num_object_clones":0,"num_object_copies":11160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9949,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9628","reported_seq":9737,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:08.300463+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:08.300463+0000","last_peered":"2024-09-16T04:49:08.300463+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:08.300463+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9628,"log_dups_size":0,"ondisk_log_size":9628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21987328,"num_objects":2684,"num_object_clones":0,"num_object_copies":10736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9628,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":506}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9868","reported_seq":9966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.624518+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:07.624518+0000","last_peered":"2024-09-16T04:49:07.624518+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:07.624518+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9868,"log_dups_size":0,"ondisk_log_size":9868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22377740,"num_objects":2732,"num_object_clones":0,"num_object_copies":10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9866,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":519}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9905","reported_seq":14467,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:04.902154+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:04.902154+0000","last_peered":"2024-09-16T04:49:04.902154+0000","last_clean":"2024-09-16T04:49:04.902154+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:04.902154+0000","last_undegraded":"2024-09-16T04:49:04.902154+0000","last_fullsized":"2024-09-16T04:49:04.902154+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9905,"log_dups_size":0,"ondisk_log_size":9905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:07.659398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23590156,"num_objects":2880,"num_object_clones":0,"num_object_copies":11520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9903,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10017","reported_seq":14774,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:08.304048+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:08.304048+0000","last_peered":"2024-09-16T04:49:08.304048+0000","last_clean":"2024-09-16T04:49:08.304048+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:08.304048+0000","last_undegraded":"2024-09-16T04:49:08.304048+0000","last_fullsized":"2024-09-16T04:49:08.304048+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:11.816781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22432280,"num_objects":2739,"num_object_clones":0,"num_object_copies":10956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10014,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9884","reported_seq":9990,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.011549+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:07.011549+0000","last_peered":"2024-09-16T04:49:07.011549+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:07.011549+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9884,"log_dups_size":0,"ondisk_log_size":9884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23541004,"num_objects":2874,"num_object_clones":0,"num_object_copies":11496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9882,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":685}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9793","reported_seq":9925,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:08.274932+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:08.274932+0000","last_peered":"2024-09-16T04:49:08.274932+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:08.274932+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9793,"log_dups_size":0,"ondisk_log_size":9793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22097810,"num_objects":2698,"num_object_clones":0,"num_object_copies":10792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9790,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9609","reported_seq":9701,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:08.194039+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:08.194039+0000","last_peered":"2024-09-16T04:49:08.194039+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:08.194039+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9609,"log_dups_size":0,"ondisk_log_size":9609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22297222,"num_objects":2722,"num_object_clones":0,"num_object_copies":10888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9608,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":617}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9767","reported_seq":9855,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:06.813656+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:06.813656+0000","last_peered":"2024-09-16T04:49:06.813656+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:06.813656+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9767,"log_dups_size":0,"ondisk_log_size":9767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22657670,"num_objects":2766,"num_object_clones":0,"num_object_copies":11064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":545}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9970","reported_seq":10099,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:08.346992+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:08.346992+0000","last_peered":"2024-09-16T04:49:08.346992+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:08.346992+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9970,"log_dups_size":0,"ondisk_log_size":9970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23033100,"num_objects":2812,"num_object_clones":0,"num_object_copies":11248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9968,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":614}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9617","reported_seq":9720,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.019083+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:07.019083+0000","last_peered":"2024-09-16T04:49:07.019083+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:07.019083+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9617,"log_dups_size":0,"ondisk_log_size":9617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23196940,"num_objects":2832,"num_object_clones":0,"num_object_copies":11328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9615,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":709}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9642","reported_seq":9736,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.577551+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:07.577551+0000","last_peered":"2024-09-16T04:49:07.577551+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:07.577551+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9642,"log_dups_size":0,"ondisk_log_size":9642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21828876,"num_objects":2665,"num_object_clones":0,"num_object_copies":10660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9640,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":437}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":363931328,"num_objects":44431,"num_object_clones":0,"num_object_copies":177724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44431,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157056,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5737,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":730742784,"data_stored":730742784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157083,"ondisk_log_size":157083,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759376,"kb_used_data":177920,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93612464,"statfs":{"total":96636764160,"available":95859163136,"internally_reserved":0,"allocated":182190080,"data_stored":182000878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595390159},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,6,3,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":770032,"kb_used_data":179360,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93601808,"statfs":{"total":96636764160,"available":95848251392,"internally_reserved":0,"allocated":183664640,"data_stored":183475438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,9,2,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051676,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613936,"kb_used_data":180992,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93757904,"statfs":{"total":96636764160,"available":96008093696,"internally_reserved":0,"allocated":185335808,"data_stored":185146606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443281103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":781832,"kb_used_data":178228,"kb_used_omap":4,"kb_used_meta":603579,"kb_avail":93590008,"statfs":{"total":96636764160,"available":95836168192,"internally_reserved":0,"allocated":182505472,"data_stored":182316270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":618065614},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":181993472,"data_stored":181993472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184823808,"data_stored":184823808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182497280,"data_stored":182497280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":181428224,"data_stored":181428224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:09.580 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:10.226 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:10.227 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:10.272 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-09-16T04:49:08.717122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":363931328,"num_objects":44431,"num_object_clones":0,"num_object_copies":177724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44431,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157056,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5737,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157083,"ondisk_log_size":157083,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2925176,"kb_used_data":716500,"kb_used_omap":14,"kb_used_meta":2208561,"kb_avail":374562184,"statfs":{"total":386547056640,"available":383551676416,"internally_reserved":0,"allocated":733696000,"data_stored":732939192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2261567025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,11,14,9,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24592384,"num_objects":-3002,"num_object_clones":0,"num_object_copies":-12008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3002,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002948"},"pg_stats":[{"pgid":"1.f","version":"35'9842","reported_seq":9957,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:08.193955+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:08.193955+0000","last_peered":"2024-09-16T04:49:08.193955+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:08.193955+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9842,"log_dups_size":0,"ondisk_log_size":9842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23035904,"num_objects":2812,"num_object_clones":0,"num_object_copies":11248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9842,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9836","reported_seq":9924,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.528010+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:07.528010+0000","last_peered":"2024-09-16T04:49:07.528010+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:07.528010+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9836,"log_dups_size":0,"ondisk_log_size":9836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23246092,"num_objects":2838,"num_object_clones":0,"num_object_copies":11352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9834,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":635}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9850","reported_seq":9948,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.010322+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:07.010322+0000","last_peered":"2024-09-16T04:49:07.010322+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:07.010322+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23100062,"num_objects":2821,"num_object_clones":0,"num_object_copies":11284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2821,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9849,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":602}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9904","reported_seq":11722,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.630230+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:07.630230+0000","last_peered":"2024-09-16T04:49:07.630230+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:07.630230+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22656268,"num_objects":2766,"num_object_clones":0,"num_object_copies":11064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9902,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1407}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9951","reported_seq":10084,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.937531+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:07.937531+0000","last_peered":"2024-09-16T04:49:07.937531+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:07.937531+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9951,"log_dups_size":0,"ondisk_log_size":9951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22852876,"num_objects":2790,"num_object_clones":0,"num_object_copies":11160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9949,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9628","reported_seq":9737,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:08.300463+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:08.300463+0000","last_peered":"2024-09-16T04:49:08.300463+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:08.300463+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9628,"log_dups_size":0,"ondisk_log_size":9628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21987328,"num_objects":2684,"num_object_clones":0,"num_object_copies":10736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9628,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":506}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9868","reported_seq":9966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.624518+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:07.624518+0000","last_peered":"2024-09-16T04:49:07.624518+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:07.624518+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9868,"log_dups_size":0,"ondisk_log_size":9868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22377740,"num_objects":2732,"num_object_clones":0,"num_object_copies":10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9866,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":519}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9905","reported_seq":14467,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:04.902154+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:04.902154+0000","last_peered":"2024-09-16T04:49:04.902154+0000","last_clean":"2024-09-16T04:49:04.902154+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:04.902154+0000","last_undegraded":"2024-09-16T04:49:04.902154+0000","last_fullsized":"2024-09-16T04:49:04.902154+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9905,"log_dups_size":0,"ondisk_log_size":9905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:07.659398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23590156,"num_objects":2880,"num_object_clones":0,"num_object_copies":11520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9903,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10017","reported_seq":14774,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:08.304048+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:08.304048+0000","last_peered":"2024-09-16T04:49:08.304048+0000","last_clean":"2024-09-16T04:49:08.304048+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:08.304048+0000","last_undegraded":"2024-09-16T04:49:08.304048+0000","last_fullsized":"2024-09-16T04:49:08.304048+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:11.816781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22432280,"num_objects":2739,"num_object_clones":0,"num_object_copies":10956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10014,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9884","reported_seq":9990,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.011549+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:07.011549+0000","last_peered":"2024-09-16T04:49:07.011549+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:07.011549+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9884,"log_dups_size":0,"ondisk_log_size":9884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23541004,"num_objects":2874,"num_object_clones":0,"num_object_copies":11496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9882,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":685}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9793","reported_seq":9925,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:08.274932+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:08.274932+0000","last_peered":"2024-09-16T04:49:08.274932+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:08.274932+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9793,"log_dups_size":0,"ondisk_log_size":9793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22097810,"num_objects":2698,"num_object_clones":0,"num_object_copies":10792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9790,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9609","reported_seq":9701,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:08.194039+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:08.194039+0000","last_peered":"2024-09-16T04:49:08.194039+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:08.194039+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9609,"log_dups_size":0,"ondisk_log_size":9609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22297222,"num_objects":2722,"num_object_clones":0,"num_object_copies":10888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9608,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":617}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9767","reported_seq":9855,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:06.813656+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:06.813656+0000","last_peered":"2024-09-16T04:49:06.813656+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:06.813656+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9767,"log_dups_size":0,"ondisk_log_size":9767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22657670,"num_objects":2766,"num_object_clones":0,"num_object_copies":11064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":545}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9970","reported_seq":10099,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:08.346992+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:08.346992+0000","last_peered":"2024-09-16T04:49:08.346992+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:08.346992+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9970,"log_dups_size":0,"ondisk_log_size":9970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23033100,"num_objects":2812,"num_object_clones":0,"num_object_copies":11248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9968,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":614}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9617","reported_seq":9720,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.019083+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:07.019083+0000","last_peered":"2024-09-16T04:49:07.019083+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:07.019083+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9617,"log_dups_size":0,"ondisk_log_size":9617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23196940,"num_objects":2832,"num_object_clones":0,"num_object_copies":11328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9615,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":709}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9642","reported_seq":9736,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.577551+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:07.577551+0000","last_peered":"2024-09-16T04:49:07.577551+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:07.577551+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9642,"log_dups_size":0,"ondisk_log_size":9642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21828876,"num_objects":2665,"num_object_clones":0,"num_object_copies":10660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9640,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":437}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":363931328,"num_objects":44431,"num_object_clones":0,"num_object_copies":177724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44431,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157056,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5737,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":730742784,"data_stored":730742784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157083,"ondisk_log_size":157083,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759376,"kb_used_data":177920,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93612464,"statfs":{"total":96636764160,"available":95859163136,"internally_reserved":0,"allocated":182190080,"data_stored":182000878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595390159},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,6,3,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":770032,"kb_used_data":179360,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93601808,"statfs":{"total":96636764160,"available":95848251392,"internally_reserved":0,"allocated":183664640,"data_stored":183475438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,9,2,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051676,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613936,"kb_used_data":180992,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93757904,"statfs":{"total":96636764160,"available":96008093696,"internally_reserved":0,"allocated":185335808,"data_stored":185146606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443281103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":781832,"kb_used_data":178228,"kb_used_omap":4,"kb_used_meta":603579,"kb_avail":93590008,"statfs":{"total":96636764160,"available":95836168192,"internally_reserved":0,"allocated":182505472,"data_stored":182316270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":618065614},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":181993472,"data_stored":181993472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184823808,"data_stored":184823808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182497280,"data_stored":182497280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":181428224,"data_stored":181428224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:10.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:49:10.273 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9842", 'reported_seq': 9957, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:08.193955+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:49:08.193955+0000', 'last_peered': '2024-09-16T04:49:08.193955+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:49:08.193955+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9842, 'log_dups_size': 0, 'ondisk_log_size': 9842, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23035904, 'num_objects': 2812, 'num_object_clones': 0, 'num_object_copies': 11248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2812, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9842, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 626}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:10.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:49:10.273 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9836", 'reported_seq': 9924, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:07.528010+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:49:07.528010+0000', 'last_peered': '2024-09-16T04:49:07.528010+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:49:07.528010+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9836, 'log_dups_size': 0, 'ondisk_log_size': 9836, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23246092, 'num_objects': 2838, 'num_object_clones': 0, 'num_object_copies': 11352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2838, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9834, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 635}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:10.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:49:10.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9850", 'reported_seq': 9948, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:07.010322+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:49:07.010322+0000', 'last_peered': '2024-09-16T04:49:07.010322+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:49:07.010322+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9850, 'log_dups_size': 0, 'ondisk_log_size': 9850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23100062, 'num_objects': 2821, 'num_object_clones': 0, 'num_object_copies': 11284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2821, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9849, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 602}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:10.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:49:10.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9904", 'reported_seq': 11722, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:07.630230+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:49:07.630230+0000', 'last_peered': '2024-09-16T04:49:07.630230+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:49:07.630230+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9904, 'log_dups_size': 0, 'ondisk_log_size': 9904, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22656268, 'num_objects': 2766, 'num_object_clones': 0, 'num_object_copies': 11064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1359, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9902, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 874, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1407}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:10.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:49:10.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9951", 'reported_seq': 10084, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:07.937531+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:49:07.937531+0000', 'last_peered': '2024-09-16T04:49:07.937531+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:49:07.937531+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9951, 'log_dups_size': 0, 'ondisk_log_size': 9951, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22852876, 'num_objects': 2790, 'num_object_clones': 0, 'num_object_copies': 11160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2790, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9949, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 542}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:10.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:49:10.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9628", 'reported_seq': 9737, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:08.300463+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:49:08.300463+0000', 'last_peered': '2024-09-16T04:49:08.300463+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:49:08.300463+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9628, 'log_dups_size': 0, 'ondisk_log_size': 9628, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21987328, 'num_objects': 2684, 'num_object_clones': 0, 'num_object_copies': 10736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9628, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 506}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:10.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:49:10.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9868", 'reported_seq': 9966, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:07.624518+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:49:07.624518+0000', 'last_peered': '2024-09-16T04:49:07.624518+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:49:07.624518+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9868, 'log_dups_size': 0, 'ondisk_log_size': 9868, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22377740, 'num_objects': 2732, 'num_object_clones': 0, 'num_object_copies': 10928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9866, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 519}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:10.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:49:10.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9884", 'reported_seq': 9990, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:07.011549+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:49:07.011549+0000', 'last_peered': '2024-09-16T04:49:07.011549+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:49:07.011549+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9884, 'log_dups_size': 0, 'ondisk_log_size': 9884, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23541004, 'num_objects': 2874, 'num_object_clones': 0, 'num_object_copies': 11496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2874, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9882, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 685}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:10.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:49:10.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9793", 'reported_seq': 9925, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:08.274932+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:49:08.274932+0000', 'last_peered': '2024-09-16T04:49:08.274932+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:49:08.274932+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9793, 'log_dups_size': 0, 'ondisk_log_size': 9793, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22097810, 'num_objects': 2698, 'num_object_clones': 0, 'num_object_copies': 10792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9790, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 543}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:10.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:49:10.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9609", 'reported_seq': 9701, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:08.194039+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:49:08.194039+0000', 'last_peered': '2024-09-16T04:49:08.194039+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:49:08.194039+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9609, 'log_dups_size': 0, 'ondisk_log_size': 9609, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22297222, 'num_objects': 2722, 'num_object_clones': 0, 'num_object_copies': 10888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9608, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 617}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:10.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:49:10.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9767", 'reported_seq': 9855, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:06.813656+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:49:06.813656+0000', 'last_peered': '2024-09-16T04:49:06.813656+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:49:06.813656+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9767, 'log_dups_size': 0, 'ondisk_log_size': 9767, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22657670, 'num_objects': 2766, 'num_object_clones': 0, 'num_object_copies': 11064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9766, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 545}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:10.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:49:10.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9970", 'reported_seq': 10099, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:08.346992+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:49:08.346992+0000', 'last_peered': '2024-09-16T04:49:08.346992+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:49:08.346992+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9970, 'log_dups_size': 0, 'ondisk_log_size': 9970, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23033100, 'num_objects': 2812, 'num_object_clones': 0, 'num_object_copies': 11248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2812, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9968, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 614}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:10.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:49:10.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9617", 'reported_seq': 9720, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:07.019083+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:49:07.019083+0000', 'last_peered': '2024-09-16T04:49:07.019083+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:49:07.019083+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9617, 'log_dups_size': 0, 'ondisk_log_size': 9617, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23196940, 'num_objects': 2832, 'num_object_clones': 0, 'num_object_copies': 11328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9615, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 709}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:10.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:49:10.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9642", 'reported_seq': 9736, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:07.577551+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:49:07.577551+0000', 'last_peered': '2024-09-16T04:49:07.577551+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:49:07.577551+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9642, 'log_dups_size': 0, 'ondisk_log_size': 9642, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21828876, 'num_objects': 2665, 'num_object_clones': 0, 'num_object_copies': 10660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9640, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 437}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:10.276 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:49:10.983 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:10.984 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29845/177724 objects degraded (16.793%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":563,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":44431,"data_bytes":363931328,"bytes_used":2995380224,"bytes_avail":383551676416,"bytes_total":386547056640,"degraded_objects":29845,"degraded_total":177724,"degraded_ratio":0.16792892349935853,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":250},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:49:11.009 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:49:11.009 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:11.623 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:11.624 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:11.655 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-09-16T04:49:10.717547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":363480768,"num_objects":44376,"num_object_clones":0,"num_object_copies":177504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44376,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157111,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5737,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157138,"ondisk_log_size":157138,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2925296,"kb_used_data":712492,"kb_used_omap":14,"kb_used_meta":2212721,"kb_avail":374562064,"statfs":{"total":386547056640,"available":383551553536,"internally_reserved":0,"allocated":729591808,"data_stored":728835000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2265826865},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,11,11,17,8,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16957440,"num_objects":-2070,"num_object_clones":0,"num_object_copies":-8280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2070,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002898"},"pg_stats":[{"pgid":"1.f","version":"35'9842","reported_seq":9957,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:08.193955+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:08.193955+0000","last_peered":"2024-09-16T04:49:08.193955+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:08.193955+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9842,"log_dups_size":0,"ondisk_log_size":9842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23035904,"num_objects":2812,"num_object_clones":0,"num_object_copies":11248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9842,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9836","reported_seq":9924,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.528010+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:07.528010+0000","last_peered":"2024-09-16T04:49:07.528010+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:07.528010+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9836,"log_dups_size":0,"ondisk_log_size":9836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23246092,"num_objects":2838,"num_object_clones":0,"num_object_copies":11352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9834,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":635}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9850","reported_seq":9948,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.010322+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:07.010322+0000","last_peered":"2024-09-16T04:49:07.010322+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:07.010322+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23100062,"num_objects":2821,"num_object_clones":0,"num_object_copies":11284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2821,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9849,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":602}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9904","reported_seq":11722,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.630230+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:07.630230+0000","last_peered":"2024-09-16T04:49:07.630230+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:07.630230+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22656268,"num_objects":2766,"num_object_clones":0,"num_object_copies":11064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9902,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1407}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9951","reported_seq":10084,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.937531+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:07.937531+0000","last_peered":"2024-09-16T04:49:07.937531+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:07.937531+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9951,"log_dups_size":0,"ondisk_log_size":9951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22852876,"num_objects":2790,"num_object_clones":0,"num_object_copies":11160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9949,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9628","reported_seq":9737,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:08.300463+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:08.300463+0000","last_peered":"2024-09-16T04:49:08.300463+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:08.300463+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9628,"log_dups_size":0,"ondisk_log_size":9628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21987328,"num_objects":2684,"num_object_clones":0,"num_object_copies":10736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9628,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":506}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9868","reported_seq":9966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.624518+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:07.624518+0000","last_peered":"2024-09-16T04:49:07.624518+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:07.624518+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9868,"log_dups_size":0,"ondisk_log_size":9868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22377740,"num_objects":2732,"num_object_clones":0,"num_object_copies":10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9866,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":519}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9960","reported_seq":14522,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:10.088971+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:10.088971+0000","last_peered":"2024-09-16T04:49:10.088971+0000","last_clean":"2024-09-16T04:49:10.088971+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:10.088971+0000","last_undegraded":"2024-09-16T04:49:10.088971+0000","last_fullsized":"2024-09-16T04:49:10.088971+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9960,"log_dups_size":0,"ondisk_log_size":9960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:12.854800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23139596,"num_objects":2825,"num_object_clones":0,"num_object_copies":11300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9958,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10017","reported_seq":14774,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:08.304048+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:08.304048+0000","last_peered":"2024-09-16T04:49:08.304048+0000","last_clean":"2024-09-16T04:49:08.304048+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:08.304048+0000","last_undegraded":"2024-09-16T04:49:08.304048+0000","last_fullsized":"2024-09-16T04:49:08.304048+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:11.816781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22432280,"num_objects":2739,"num_object_clones":0,"num_object_copies":10956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10014,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9884","reported_seq":9990,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.011549+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:07.011549+0000","last_peered":"2024-09-16T04:49:07.011549+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:07.011549+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9884,"log_dups_size":0,"ondisk_log_size":9884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23541004,"num_objects":2874,"num_object_clones":0,"num_object_copies":11496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9882,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":685}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9793","reported_seq":9925,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:08.274932+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:08.274932+0000","last_peered":"2024-09-16T04:49:08.274932+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:08.274932+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9793,"log_dups_size":0,"ondisk_log_size":9793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22097810,"num_objects":2698,"num_object_clones":0,"num_object_copies":10792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9790,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9609","reported_seq":9701,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:08.194039+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:08.194039+0000","last_peered":"2024-09-16T04:49:08.194039+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:08.194039+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9609,"log_dups_size":0,"ondisk_log_size":9609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22297222,"num_objects":2722,"num_object_clones":0,"num_object_copies":10888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9608,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":617}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9767","reported_seq":9855,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:06.813656+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:06.813656+0000","last_peered":"2024-09-16T04:49:06.813656+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:06.813656+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9767,"log_dups_size":0,"ondisk_log_size":9767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22657670,"num_objects":2766,"num_object_clones":0,"num_object_copies":11064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":545}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9970","reported_seq":10099,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:08.346992+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:08.346992+0000","last_peered":"2024-09-16T04:49:08.346992+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:08.346992+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9970,"log_dups_size":0,"ondisk_log_size":9970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23033100,"num_objects":2812,"num_object_clones":0,"num_object_copies":11248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9968,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":614}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9617","reported_seq":9720,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.019083+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:07.019083+0000","last_peered":"2024-09-16T04:49:07.019083+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:07.019083+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9617,"log_dups_size":0,"ondisk_log_size":9617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23196940,"num_objects":2832,"num_object_clones":0,"num_object_copies":11328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9615,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":709}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9642","reported_seq":9736,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:07.577551+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:07.577551+0000","last_peered":"2024-09-16T04:49:07.577551+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:07.577551+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9642,"log_dups_size":0,"ondisk_log_size":9642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21828876,"num_objects":2665,"num_object_clones":0,"num_object_copies":10660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9640,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":437}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":363480768,"num_objects":44376,"num_object_clones":0,"num_object_copies":177504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44376,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157111,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5737,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":726609920,"data_stored":726609920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157138,"ondisk_log_size":157138,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759376,"kb_used_data":177920,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93612464,"statfs":{"total":96636764160,"available":95859163136,"internally_reserved":0,"allocated":182190080,"data_stored":182000878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595390159},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,6,3,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":770032,"kb_used_data":179360,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93601808,"statfs":{"total":96636764160,"available":95848251392,"internally_reserved":0,"allocated":183664640,"data_stored":183475438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,9,2,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051677,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614056,"kb_used_data":176984,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93757784,"statfs":{"total":96636764160,"available":96007970816,"internally_reserved":0,"allocated":181231616,"data_stored":181042414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447540943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,12,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":781832,"kb_used_data":178228,"kb_used_omap":4,"kb_used_meta":603579,"kb_avail":93590008,"statfs":{"total":96636764160,"available":95836168192,"internally_reserved":0,"allocated":182505472,"data_stored":182316270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":618065614},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":181993472,"data_stored":181993472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180690944,"data_stored":180690944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182497280,"data_stored":182497280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":181428224,"data_stored":181428224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:14.657 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:15.294 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:15.295 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:15.316 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-16T04:49:14.718484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":356099776,"num_objects":43475,"num_object_clones":0,"num_object_copies":173900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43475,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158012,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5813,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158039,"ondisk_log_size":158039,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2910736,"kb_used_data":700688,"kb_used_omap":14,"kb_used_meta":2209969,"kb_avail":374576624,"statfs":{"total":386547056640,"available":383566462976,"internally_reserved":0,"allocated":717504512,"data_stored":716747704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2263008817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,18,18,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20070400,"num_objects":-2450,"num_object_clones":0,"num_object_copies":-9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2450,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002840"},"pg_stats":[{"pgid":"1.f","version":"35'9897","reported_seq":10013,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:13.140748+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:13.140748+0000","last_peered":"2024-09-16T04:49:13.140748+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:13.140748+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22585344,"num_objects":2757,"num_object_clones":0,"num_object_copies":11028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9897,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":571}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9902","reported_seq":9990,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.651408+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:12.651408+0000","last_peered":"2024-09-16T04:49:12.651408+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:12.651408+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9902,"log_dups_size":0,"ondisk_log_size":9902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22705420,"num_objects":2772,"num_object_clones":0,"num_object_copies":11088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":569}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9913","reported_seq":10011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.136437+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:12.136437+0000","last_peered":"2024-09-16T04:49:12.136437+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:12.136437+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9913,"log_dups_size":0,"ondisk_log_size":9913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22583966,"num_objects":2758,"num_object_clones":0,"num_object_copies":11032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2758,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9912,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":539}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9963","reported_seq":11933,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.711281+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:12.711281+0000","last_peered":"2024-09-16T04:49:12.711281+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:12.711281+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9963,"log_dups_size":0,"ondisk_log_size":9963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22172940,"num_objects":2707,"num_object_clones":0,"num_object_copies":10828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9961,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":950,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10016","reported_seq":10148,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:13.399296+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:13.399296+0000","last_peered":"2024-09-16T04:49:13.399296+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:13.399296+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22320396,"num_objects":2725,"num_object_clones":0,"num_object_copies":10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10014,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9707","reported_seq":9816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:13.228405+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:13.228405+0000","last_peered":"2024-09-16T04:49:13.228405+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:13.228405+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21340160,"num_objects":2605,"num_object_clones":0,"num_object_copies":10420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":427}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9935","reported_seq":10033,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.512806+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:12.512806+0000","last_peered":"2024-09-16T04:49:12.512806+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:12.512806+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9935,"log_dups_size":0,"ondisk_log_size":9935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21828876,"num_objects":2665,"num_object_clones":0,"num_object_copies":10660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9933,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":452}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9960","reported_seq":14522,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:10.088971+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:10.088971+0000","last_peered":"2024-09-16T04:49:10.088971+0000","last_clean":"2024-09-16T04:49:10.088971+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:10.088971+0000","last_undegraded":"2024-09-16T04:49:10.088971+0000","last_fullsized":"2024-09-16T04:49:10.088971+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9960,"log_dups_size":0,"ondisk_log_size":9960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:12.854800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23139596,"num_objects":2825,"num_object_clones":0,"num_object_copies":11300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9958,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10065","reported_seq":14823,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:13.280033+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:13.280033+0000","last_peered":"2024-09-16T04:49:13.280033+0000","last_clean":"2024-09-16T04:49:13.280033+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:13.280033+0000","last_undegraded":"2024-09-16T04:49:13.280033+0000","last_fullsized":"2024-09-16T04:49:13.280033+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:16.955562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22039064,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10062,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9935","reported_seq":10042,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.032635+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:12.032635+0000","last_peered":"2024-09-16T04:49:12.032635+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:12.032635+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9935,"log_dups_size":0,"ondisk_log_size":9935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23123212,"num_objects":2823,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9933,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9848","reported_seq":9981,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:13.360929+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:13.360929+0000","last_peered":"2024-09-16T04:49:13.360929+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:13.360929+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9848,"log_dups_size":0,"ondisk_log_size":9848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21647250,"num_objects":2643,"num_object_clones":0,"num_object_copies":10572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9845,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":488}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9657","reported_seq":9747,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:13.409906+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:13.409906+0000","last_peered":"2024-09-16T04:49:13.409906+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:13.409906+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9657,"log_dups_size":0,"ondisk_log_size":9657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21904006,"num_objects":2674,"num_object_clones":0,"num_object_copies":10696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9656,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":569}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9824","reported_seq":9911,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:11.976796+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:11.976796+0000","last_peered":"2024-09-16T04:49:11.976796+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:11.976796+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9824,"log_dups_size":0,"ondisk_log_size":9824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22190726,"num_objects":2709,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9823,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":488}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10028","reported_seq":10160,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.973860+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:12.973860+0000","last_peered":"2024-09-16T04:49:12.973860+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:12.973860+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22557964,"num_objects":2754,"num_object_clones":0,"num_object_copies":11016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10026,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":556}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9680","reported_seq":9784,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.186624+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:12.186624+0000","last_peered":"2024-09-16T04:49:12.186624+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:12.186624+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9680,"log_dups_size":0,"ondisk_log_size":9680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22680844,"num_objects":2769,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":646}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9709","reported_seq":9802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.703048+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:12.703048+0000","last_peered":"2024-09-16T04:49:12.703048+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:12.703048+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21280012,"num_objects":2598,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":356099776,"num_objects":43475,"num_object_clones":0,"num_object_copies":173900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43475,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158012,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5813,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":714670080,"data_stored":714670080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158039,"ondisk_log_size":158039,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761008,"kb_used_data":173984,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93610832,"statfs":{"total":96636764160,"available":95857491968,"internally_reserved":0,"allocated":178159616,"data_stored":177970414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601091791},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,1,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":770212,"kb_used_data":175380,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93601628,"statfs":{"total":96636764160,"available":95848067072,"internally_reserved":0,"allocated":179589120,"data_stored":179399918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051677,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614056,"kb_used_data":176984,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93757784,"statfs":{"total":96636764160,"available":96007970816,"internally_reserved":0,"allocated":181231616,"data_stored":181042414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447540943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,12,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765460,"kb_used_data":174340,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93606380,"statfs":{"total":96636764160,"available":95852933120,"internally_reserved":0,"allocated":178524160,"data_stored":178334958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":605286094},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":178044928,"data_stored":178044928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180690944,"data_stored":180690944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":178511872,"data_stored":178511872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":177422336,"data_stored":177422336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:15.318 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:15.941 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:15.942 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:15.966 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-16T04:49:14.718484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":356099776,"num_objects":43475,"num_object_clones":0,"num_object_copies":173900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43475,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158012,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5813,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158039,"ondisk_log_size":158039,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2910736,"kb_used_data":700688,"kb_used_omap":14,"kb_used_meta":2209969,"kb_avail":374576624,"statfs":{"total":386547056640,"available":383566462976,"internally_reserved":0,"allocated":717504512,"data_stored":716747704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2263008817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,18,18,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20070400,"num_objects":-2450,"num_object_clones":0,"num_object_copies":-9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2450,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002840"},"pg_stats":[{"pgid":"1.f","version":"35'9897","reported_seq":10013,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:13.140748+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:13.140748+0000","last_peered":"2024-09-16T04:49:13.140748+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:13.140748+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22585344,"num_objects":2757,"num_object_clones":0,"num_object_copies":11028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9897,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":571}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9902","reported_seq":9990,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.651408+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:12.651408+0000","last_peered":"2024-09-16T04:49:12.651408+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:12.651408+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9902,"log_dups_size":0,"ondisk_log_size":9902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22705420,"num_objects":2772,"num_object_clones":0,"num_object_copies":11088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":569}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9913","reported_seq":10011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.136437+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:12.136437+0000","last_peered":"2024-09-16T04:49:12.136437+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:12.136437+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9913,"log_dups_size":0,"ondisk_log_size":9913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22583966,"num_objects":2758,"num_object_clones":0,"num_object_copies":11032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2758,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9912,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":539}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9963","reported_seq":11933,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.711281+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:12.711281+0000","last_peered":"2024-09-16T04:49:12.711281+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:12.711281+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9963,"log_dups_size":0,"ondisk_log_size":9963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22172940,"num_objects":2707,"num_object_clones":0,"num_object_copies":10828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9961,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":950,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10016","reported_seq":10148,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:13.399296+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:13.399296+0000","last_peered":"2024-09-16T04:49:13.399296+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:13.399296+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22320396,"num_objects":2725,"num_object_clones":0,"num_object_copies":10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10014,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9707","reported_seq":9816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:13.228405+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:13.228405+0000","last_peered":"2024-09-16T04:49:13.228405+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:13.228405+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21340160,"num_objects":2605,"num_object_clones":0,"num_object_copies":10420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":427}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9935","reported_seq":10033,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.512806+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:12.512806+0000","last_peered":"2024-09-16T04:49:12.512806+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:12.512806+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9935,"log_dups_size":0,"ondisk_log_size":9935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21828876,"num_objects":2665,"num_object_clones":0,"num_object_copies":10660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9933,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":452}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9960","reported_seq":14522,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:10.088971+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:10.088971+0000","last_peered":"2024-09-16T04:49:10.088971+0000","last_clean":"2024-09-16T04:49:10.088971+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:10.088971+0000","last_undegraded":"2024-09-16T04:49:10.088971+0000","last_fullsized":"2024-09-16T04:49:10.088971+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9960,"log_dups_size":0,"ondisk_log_size":9960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:12.854800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23139596,"num_objects":2825,"num_object_clones":0,"num_object_copies":11300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9958,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10065","reported_seq":14823,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:13.280033+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:13.280033+0000","last_peered":"2024-09-16T04:49:13.280033+0000","last_clean":"2024-09-16T04:49:13.280033+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:13.280033+0000","last_undegraded":"2024-09-16T04:49:13.280033+0000","last_fullsized":"2024-09-16T04:49:13.280033+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:16.955562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22039064,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10062,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9935","reported_seq":10042,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.032635+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:12.032635+0000","last_peered":"2024-09-16T04:49:12.032635+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:12.032635+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9935,"log_dups_size":0,"ondisk_log_size":9935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23123212,"num_objects":2823,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9933,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9848","reported_seq":9981,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:13.360929+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:13.360929+0000","last_peered":"2024-09-16T04:49:13.360929+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:13.360929+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9848,"log_dups_size":0,"ondisk_log_size":9848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21647250,"num_objects":2643,"num_object_clones":0,"num_object_copies":10572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9845,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":488}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9657","reported_seq":9747,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:13.409906+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:13.409906+0000","last_peered":"2024-09-16T04:49:13.409906+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:13.409906+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9657,"log_dups_size":0,"ondisk_log_size":9657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21904006,"num_objects":2674,"num_object_clones":0,"num_object_copies":10696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9656,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":569}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9824","reported_seq":9911,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:11.976796+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:11.976796+0000","last_peered":"2024-09-16T04:49:11.976796+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:11.976796+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9824,"log_dups_size":0,"ondisk_log_size":9824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22190726,"num_objects":2709,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9823,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":488}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10028","reported_seq":10160,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.973860+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:12.973860+0000","last_peered":"2024-09-16T04:49:12.973860+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:12.973860+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22557964,"num_objects":2754,"num_object_clones":0,"num_object_copies":11016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10026,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":556}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9680","reported_seq":9784,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.186624+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:12.186624+0000","last_peered":"2024-09-16T04:49:12.186624+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:12.186624+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9680,"log_dups_size":0,"ondisk_log_size":9680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22680844,"num_objects":2769,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":646}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9709","reported_seq":9802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.703048+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:12.703048+0000","last_peered":"2024-09-16T04:49:12.703048+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:12.703048+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21280012,"num_objects":2598,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":356099776,"num_objects":43475,"num_object_clones":0,"num_object_copies":173900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43475,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158012,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5813,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":714670080,"data_stored":714670080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158039,"ondisk_log_size":158039,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761008,"kb_used_data":173984,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93610832,"statfs":{"total":96636764160,"available":95857491968,"internally_reserved":0,"allocated":178159616,"data_stored":177970414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601091791},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,1,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":770212,"kb_used_data":175380,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93601628,"statfs":{"total":96636764160,"available":95848067072,"internally_reserved":0,"allocated":179589120,"data_stored":179399918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051677,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614056,"kb_used_data":176984,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93757784,"statfs":{"total":96636764160,"available":96007970816,"internally_reserved":0,"allocated":181231616,"data_stored":181042414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447540943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,12,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765460,"kb_used_data":174340,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93606380,"statfs":{"total":96636764160,"available":95852933120,"internally_reserved":0,"allocated":178524160,"data_stored":178334958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":605286094},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":178044928,"data_stored":178044928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180690944,"data_stored":180690944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":178511872,"data_stored":178511872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":177422336,"data_stored":177422336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:15.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:49:15.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9897", 'reported_seq': 10013, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:13.140748+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:49:13.140748+0000', 'last_peered': '2024-09-16T04:49:13.140748+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:49:13.140748+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9897, 'log_dups_size': 0, 'ondisk_log_size': 9897, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22585344, 'num_objects': 2757, 'num_object_clones': 0, 'num_object_copies': 11028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9897, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 571}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:15.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:49:15.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9902", 'reported_seq': 9990, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:12.651408+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:49:12.651408+0000', 'last_peered': '2024-09-16T04:49:12.651408+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:49:12.651408+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9902, 'log_dups_size': 0, 'ondisk_log_size': 9902, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22705420, 'num_objects': 2772, 'num_object_clones': 0, 'num_object_copies': 11088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9900, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 569}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:15.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:49:15.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9913", 'reported_seq': 10011, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:12.136437+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:49:12.136437+0000', 'last_peered': '2024-09-16T04:49:12.136437+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:49:12.136437+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9913, 'log_dups_size': 0, 'ondisk_log_size': 9913, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22583966, 'num_objects': 2758, 'num_object_clones': 0, 'num_object_copies': 11032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2758, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9912, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 539}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:15.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:49:15.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9963", 'reported_seq': 11933, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:12.711281+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:49:12.711281+0000', 'last_peered': '2024-09-16T04:49:12.711281+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:49:12.711281+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9963, 'log_dups_size': 0, 'ondisk_log_size': 9963, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22172940, 'num_objects': 2707, 'num_object_clones': 0, 'num_object_copies': 10828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9961, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 950, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1424}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:15.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:49:15.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10016", 'reported_seq': 10148, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:13.399296+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:49:13.399296+0000', 'last_peered': '2024-09-16T04:49:13.399296+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:49:13.399296+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22320396, 'num_objects': 2725, 'num_object_clones': 0, 'num_object_copies': 10900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10014, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 477}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:15.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:49:15.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9707", 'reported_seq': 9816, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:13.228405+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:49:13.228405+0000', 'last_peered': '2024-09-16T04:49:13.228405+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:49:13.228405+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9707, 'log_dups_size': 0, 'ondisk_log_size': 9707, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21340160, 'num_objects': 2605, 'num_object_clones': 0, 'num_object_copies': 10420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2605, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9707, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 427}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:15.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:49:15.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9935", 'reported_seq': 10033, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:12.512806+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:49:12.512806+0000', 'last_peered': '2024-09-16T04:49:12.512806+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:49:12.512806+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9935, 'log_dups_size': 0, 'ondisk_log_size': 9935, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21828876, 'num_objects': 2665, 'num_object_clones': 0, 'num_object_copies': 10660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9933, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 452}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:15.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:49:15.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9935", 'reported_seq': 10042, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:12.032635+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:49:12.032635+0000', 'last_peered': '2024-09-16T04:49:12.032635+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:49:12.032635+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9935, 'log_dups_size': 0, 'ondisk_log_size': 9935, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23123212, 'num_objects': 2823, 'num_object_clones': 0, 'num_object_copies': 11292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2823, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9933, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 634}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:15.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:49:15.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9848", 'reported_seq': 9981, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:13.360929+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:49:13.360929+0000', 'last_peered': '2024-09-16T04:49:13.360929+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:49:13.360929+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9848, 'log_dups_size': 0, 'ondisk_log_size': 9848, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21647250, 'num_objects': 2643, 'num_object_clones': 0, 'num_object_copies': 10572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2643, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9845, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 488}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:15.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:49:15.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9657", 'reported_seq': 9747, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:13.409906+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:49:13.409906+0000', 'last_peered': '2024-09-16T04:49:13.409906+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:49:13.409906+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9657, 'log_dups_size': 0, 'ondisk_log_size': 9657, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21904006, 'num_objects': 2674, 'num_object_clones': 0, 'num_object_copies': 10696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2674, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9656, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 569}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:15.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:49:15.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9824", 'reported_seq': 9911, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:11.976796+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:49:11.976796+0000', 'last_peered': '2024-09-16T04:49:11.976796+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:49:11.976796+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9824, 'log_dups_size': 0, 'ondisk_log_size': 9824, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22190726, 'num_objects': 2709, 'num_object_clones': 0, 'num_object_copies': 10836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9823, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 488}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:15.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:49:15.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10028", 'reported_seq': 10160, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:12.973860+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:49:12.973860+0000', 'last_peered': '2024-09-16T04:49:12.973860+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:49:12.973860+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22557964, 'num_objects': 2754, 'num_object_clones': 0, 'num_object_copies': 11016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2754, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10026, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 556}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:15.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:49:15.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9680", 'reported_seq': 9784, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:12.186624+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:49:12.186624+0000', 'last_peered': '2024-09-16T04:49:12.186624+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:49:12.186624+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9680, 'log_dups_size': 0, 'ondisk_log_size': 9680, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22680844, 'num_objects': 2769, 'num_object_clones': 0, 'num_object_copies': 11076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9678, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 646}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:15.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:49:15.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9709", 'reported_seq': 9802, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:12.703048+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:49:12.703048+0000', 'last_peered': '2024-09-16T04:49:12.703048+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:49:12.703048+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9709, 'log_dups_size': 0, 'ondisk_log_size': 9709, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21280012, 'num_objects': 2598, 'num_object_clones': 0, 'num_object_copies': 10392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9707, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 370}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:15.971 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:49:16.698 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:16.698 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29769/173900 objects degraded (17.118%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":569,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":43475,"data_bytes":356099776,"bytes_used":2980593664,"bytes_avail":383566462976,"bytes_total":386547056640,"degraded_objects":29769,"degraded_total":173900,"degraded_ratio":0.17118458884416332,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":204},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:49:16.721 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:49:16.721 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:17.329 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:17.331 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:17.360 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-16T04:49:16.718995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":355567296,"num_objects":43410,"num_object_clones":0,"num_object_copies":173640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43410,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158077,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5813,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158104,"ondisk_log_size":158104,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2894232,"kb_used_data":696664,"kb_used_omap":14,"kb_used_meta":2197489,"kb_avail":374593128,"statfs":{"total":386547056640,"available":383583363072,"internally_reserved":0,"allocated":713383936,"data_stored":712627128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2250229297},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,2,15,7,14,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16883712,"num_objects":-2061,"num_object_clones":0,"num_object_copies":-8244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2061,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002841"},"pg_stats":[{"pgid":"1.f","version":"35'9897","reported_seq":10013,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:13.140748+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:13.140748+0000","last_peered":"2024-09-16T04:49:13.140748+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:13.140748+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22585344,"num_objects":2757,"num_object_clones":0,"num_object_copies":11028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9897,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":571}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9902","reported_seq":9990,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.651408+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:12.651408+0000","last_peered":"2024-09-16T04:49:12.651408+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:12.651408+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9902,"log_dups_size":0,"ondisk_log_size":9902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22705420,"num_objects":2772,"num_object_clones":0,"num_object_copies":11088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":569}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9913","reported_seq":10011,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.136437+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:12.136437+0000","last_peered":"2024-09-16T04:49:12.136437+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:12.136437+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9913,"log_dups_size":0,"ondisk_log_size":9913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22583966,"num_objects":2758,"num_object_clones":0,"num_object_copies":11032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2758,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9912,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":539}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9963","reported_seq":11933,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.711281+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:12.711281+0000","last_peered":"2024-09-16T04:49:12.711281+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:12.711281+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9963,"log_dups_size":0,"ondisk_log_size":9963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22172940,"num_objects":2707,"num_object_clones":0,"num_object_copies":10828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9961,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":950,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10016","reported_seq":10148,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:13.399296+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:13.399296+0000","last_peered":"2024-09-16T04:49:13.399296+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:13.399296+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22320396,"num_objects":2725,"num_object_clones":0,"num_object_copies":10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10014,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9707","reported_seq":9816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:13.228405+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:13.228405+0000","last_peered":"2024-09-16T04:49:13.228405+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:13.228405+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21340160,"num_objects":2605,"num_object_clones":0,"num_object_copies":10420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":427}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9935","reported_seq":10033,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.512806+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:12.512806+0000","last_peered":"2024-09-16T04:49:12.512806+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:12.512806+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9935,"log_dups_size":0,"ondisk_log_size":9935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21828876,"num_objects":2665,"num_object_clones":0,"num_object_copies":10660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9933,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":452}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10025","reported_seq":14586,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:15.315367+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:15.315367+0000","last_peered":"2024-09-16T04:49:15.315367+0000","last_clean":"2024-09-16T04:49:15.315367+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:15.315367+0000","last_undegraded":"2024-09-16T04:49:15.315367+0000","last_fullsized":"2024-09-16T04:49:15.315367+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:18.884246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22607116,"num_objects":2760,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10023,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10065","reported_seq":14823,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:13.280033+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:13.280033+0000","last_peered":"2024-09-16T04:49:13.280033+0000","last_clean":"2024-09-16T04:49:13.280033+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:13.280033+0000","last_undegraded":"2024-09-16T04:49:13.280033+0000","last_fullsized":"2024-09-16T04:49:13.280033+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:16.955562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22039064,"num_objects":2691,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10062,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9935","reported_seq":10042,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.032635+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:12.032635+0000","last_peered":"2024-09-16T04:49:12.032635+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:12.032635+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9935,"log_dups_size":0,"ondisk_log_size":9935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23123212,"num_objects":2823,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9933,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9848","reported_seq":9981,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:13.360929+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:13.360929+0000","last_peered":"2024-09-16T04:49:13.360929+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:13.360929+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9848,"log_dups_size":0,"ondisk_log_size":9848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21647250,"num_objects":2643,"num_object_clones":0,"num_object_copies":10572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9845,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":488}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9657","reported_seq":9747,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:13.409906+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:13.409906+0000","last_peered":"2024-09-16T04:49:13.409906+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:13.409906+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9657,"log_dups_size":0,"ondisk_log_size":9657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21904006,"num_objects":2674,"num_object_clones":0,"num_object_copies":10696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9656,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":569}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9824","reported_seq":9911,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:11.976796+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:11.976796+0000","last_peered":"2024-09-16T04:49:11.976796+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:11.976796+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9824,"log_dups_size":0,"ondisk_log_size":9824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22190726,"num_objects":2709,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9823,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":488}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10028","reported_seq":10160,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.973860+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:12.973860+0000","last_peered":"2024-09-16T04:49:12.973860+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:12.973860+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22557964,"num_objects":2754,"num_object_clones":0,"num_object_copies":11016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10026,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":556}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9680","reported_seq":9784,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.186624+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:12.186624+0000","last_peered":"2024-09-16T04:49:12.186624+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:12.186624+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9680,"log_dups_size":0,"ondisk_log_size":9680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22680844,"num_objects":2769,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":646}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9709","reported_seq":9802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:12.703048+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:12.703048+0000","last_peered":"2024-09-16T04:49:12.703048+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:12.703048+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21280012,"num_objects":2598,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":355567296,"num_objects":43410,"num_object_clones":0,"num_object_copies":173640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43410,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158077,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5813,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":710414336,"data_stored":710414336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158104,"ondisk_log_size":158104,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761008,"kb_used_data":173984,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93610832,"statfs":{"total":96636764160,"available":95857491968,"internally_reserved":0,"allocated":178159616,"data_stored":177970414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601091791},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,1,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":770212,"kb_used_data":175380,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93601628,"statfs":{"total":96636764160,"available":95848067072,"internally_reserved":0,"allocated":179589120,"data_stored":179399918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051678,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597552,"kb_used_data":172960,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93774288,"statfs":{"total":96636764160,"available":96024870912,"internally_reserved":0,"allocated":177111040,"data_stored":176921838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434761423},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,0,11,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765460,"kb_used_data":174340,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93606380,"statfs":{"total":96636764160,"available":95852933120,"internally_reserved":0,"allocated":178524160,"data_stored":178334958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":605286094},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":178044928,"data_stored":178044928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176435200,"data_stored":176435200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":178511872,"data_stored":178511872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":177422336,"data_stored":177422336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:20.361 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:20.958 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:20.959 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:20.993 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-16T04:49:20.720044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":347489984,"num_objects":42424,"num_object_clones":0,"num_object_copies":169696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42424,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159063,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5862,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158990,"ondisk_log_size":158990,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2889900,"kb_used_data":679820,"kb_used_omap":14,"kb_used_meta":2209969,"kb_avail":374597460,"statfs":{"total":386547056640,"available":383587799040,"internally_reserved":0,"allocated":696135680,"data_stored":695378872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2263008817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,5,5,9,13,11,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16441344,"num_objects":-2007,"num_object_clones":0,"num_object_copies":-8028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2007,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002921"},"pg_stats":[{"pgid":"1.f","version":"35'9949","reported_seq":10064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:18.385249+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:18.385249+0000","last_peered":"2024-09-16T04:49:18.385249+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:18.385249+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22159360,"num_objects":2705,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9949,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":519}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9968","reported_seq":10056,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.612101+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:17.612101+0000","last_peered":"2024-09-16T04:49:17.612101+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:17.612101+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9968,"log_dups_size":0,"ondisk_log_size":9968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22164748,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9966,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9987","reported_seq":10086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.293372+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:17.293372+0000","last_peered":"2024-09-16T04:49:17.293372+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:17.293372+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9987,"log_dups_size":0,"ondisk_log_size":9987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21977758,"num_objects":2684,"num_object_clones":0,"num_object_copies":10736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9986,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":465}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10029","reported_seq":12097,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.593698+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:17.593698+0000","last_peered":"2024-09-16T04:49:17.593698+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:17.593698+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21632268,"num_objects":2641,"num_object_clones":0,"num_object_copies":10564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10027,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1407}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10071","reported_seq":10202,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:18.470872+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:18.470872+0000","last_peered":"2024-09-16T04:49:18.470872+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:18.470872+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21869836,"num_objects":2670,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10069,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9773","reported_seq":9883,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.989725+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:17.989725+0000","last_peered":"2024-09-16T04:49:17.989725+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:17.989725+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9773,"log_dups_size":0,"ondisk_log_size":9773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20799488,"num_objects":2539,"num_object_clones":0,"num_object_copies":10156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9773,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9998","reported_seq":10096,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.601297+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:17.601297+0000","last_peered":"2024-09-16T04:49:17.601297+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:17.601297+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9998,"log_dups_size":0,"ondisk_log_size":9998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21312780,"num_objects":2602,"num_object_clones":0,"num_object_copies":10408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9996,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":389}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10085","reported_seq":14647,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:20.255416+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:20.255416+0000","last_peered":"2024-09-16T04:49:20.255416+0000","last_clean":"2024-09-16T04:49:20.255416+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:20.255416+0000","last_undegraded":"2024-09-16T04:49:20.255416+0000","last_fullsized":"2024-09-16T04:49:20.255416+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:23.970542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22115596,"num_objects":2700,"num_object_clones":0,"num_object_copies":10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10083,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10110","reported_seq":14868,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:18.408048+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:18.408048+0000","last_peered":"2024-09-16T04:49:18.408048+0000","last_clean":"2024-09-16T04:49:18.408048+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:18.408048+0000","last_undegraded":"2024-09-16T04:49:18.408048+0000","last_fullsized":"2024-09-16T04:49:18.408048+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:22.079404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21670424,"num_objects":2646,"num_object_clones":0,"num_object_copies":10584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10107,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9995","reported_seq":10102,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.080200+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:17.080200+0000","last_peered":"2024-09-16T04:49:17.080200+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:17.080200+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9995,"log_dups_size":0,"ondisk_log_size":9995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22631692,"num_objects":2763,"num_object_clones":0,"num_object_copies":11052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9993,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":574}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9902","reported_seq":10034,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:18.466832+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:18.466832+0000","last_peered":"2024-09-16T04:49:18.466832+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:18.466832+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9902,"log_dups_size":0,"ondisk_log_size":9902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21204882,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9899,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9724","reported_seq":9816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:18.299873+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:18.299873+0000","last_peered":"2024-09-16T04:49:18.299873+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:18.299873+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9724,"log_dups_size":0,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21355142,"num_objects":2607,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9723,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":502}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9889","reported_seq":9978,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.138627+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:17.138627+0000","last_peered":"2024-09-16T04:49:17.138627+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:17.138627+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9889,"log_dups_size":0,"ondisk_log_size":9889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21658246,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9888,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10095","reported_seq":10227,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:18.452493+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:18.452493+0000","last_peered":"2024-09-16T04:49:18.452493+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:18.452493+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22009100,"num_objects":2687,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10093,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":489}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9747","reported_seq":9851,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.091354+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:17.091354+0000","last_peered":"2024-09-16T04:49:17.091354+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:17.091354+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9747,"log_dups_size":0,"ondisk_log_size":9747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22131980,"num_objects":2702,"num_object_clones":0,"num_object_copies":10808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":579}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9768","reported_seq":9862,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.473909+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:17.473909+0000","last_peered":"2024-09-16T04:49:17.473909+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:17.473909+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9768,"log_dups_size":0,"ondisk_log_size":9768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20796684,"num_objects":2539,"num_object_clones":0,"num_object_copies":10156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":311}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":347489984,"num_objects":42424,"num_object_clones":0,"num_object_copies":169696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42424,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159063,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5862,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":693927936,"data_stored":693927936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158990,"ondisk_log_size":158990,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761132,"kb_used_data":169948,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93610708,"statfs":{"total":96636764160,"available":95857364992,"internally_reserved":0,"allocated":174026752,"data_stored":173837550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605351631},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,0,0,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769588,"kb_used_data":170580,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93602252,"statfs":{"total":96636764160,"available":95848706048,"internally_reserved":0,"allocated":174673920,"data_stored":174484718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051679,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597684,"kb_used_data":168932,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93774156,"statfs":{"total":96636764160,"available":96024735744,"internally_reserved":0,"allocated":172986368,"data_stored":172797166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":439021263},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,6,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761496,"kb_used_data":170360,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93610344,"statfs":{"total":96636764160,"available":95856992256,"internally_reserved":0,"allocated":174448640,"data_stored":174259438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":605286094},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":173981696,"data_stored":173981696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172335104,"data_stored":172335104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174272512,"data_stored":174272512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":173338624,"data_stored":173338624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:20.995 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:21.611 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:21.611 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:21.637 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-16T04:49:20.720044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":347489984,"num_objects":42424,"num_object_clones":0,"num_object_copies":169696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42424,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159063,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5862,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158990,"ondisk_log_size":158990,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2889900,"kb_used_data":679820,"kb_used_omap":14,"kb_used_meta":2209969,"kb_avail":374597460,"statfs":{"total":386547056640,"available":383587799040,"internally_reserved":0,"allocated":696135680,"data_stored":695378872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2263008817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,5,5,9,13,11,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16441344,"num_objects":-2007,"num_object_clones":0,"num_object_copies":-8028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2007,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002921"},"pg_stats":[{"pgid":"1.f","version":"35'9949","reported_seq":10064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:18.385249+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:18.385249+0000","last_peered":"2024-09-16T04:49:18.385249+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:18.385249+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22159360,"num_objects":2705,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9949,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":519}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9968","reported_seq":10056,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.612101+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:17.612101+0000","last_peered":"2024-09-16T04:49:17.612101+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:17.612101+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9968,"log_dups_size":0,"ondisk_log_size":9968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22164748,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9966,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9987","reported_seq":10086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.293372+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:17.293372+0000","last_peered":"2024-09-16T04:49:17.293372+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:17.293372+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9987,"log_dups_size":0,"ondisk_log_size":9987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21977758,"num_objects":2684,"num_object_clones":0,"num_object_copies":10736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9986,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":465}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10029","reported_seq":12097,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.593698+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:17.593698+0000","last_peered":"2024-09-16T04:49:17.593698+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:17.593698+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21632268,"num_objects":2641,"num_object_clones":0,"num_object_copies":10564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10027,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1407}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10071","reported_seq":10202,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:18.470872+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:18.470872+0000","last_peered":"2024-09-16T04:49:18.470872+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:18.470872+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21869836,"num_objects":2670,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10069,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9773","reported_seq":9883,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.989725+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:17.989725+0000","last_peered":"2024-09-16T04:49:17.989725+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:17.989725+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9773,"log_dups_size":0,"ondisk_log_size":9773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20799488,"num_objects":2539,"num_object_clones":0,"num_object_copies":10156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9773,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9998","reported_seq":10096,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.601297+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:17.601297+0000","last_peered":"2024-09-16T04:49:17.601297+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:17.601297+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9998,"log_dups_size":0,"ondisk_log_size":9998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21312780,"num_objects":2602,"num_object_clones":0,"num_object_copies":10408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9996,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":389}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10085","reported_seq":14647,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:20.255416+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:20.255416+0000","last_peered":"2024-09-16T04:49:20.255416+0000","last_clean":"2024-09-16T04:49:20.255416+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:20.255416+0000","last_undegraded":"2024-09-16T04:49:20.255416+0000","last_fullsized":"2024-09-16T04:49:20.255416+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:23.970542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22115596,"num_objects":2700,"num_object_clones":0,"num_object_copies":10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10083,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10110","reported_seq":14868,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:18.408048+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:18.408048+0000","last_peered":"2024-09-16T04:49:18.408048+0000","last_clean":"2024-09-16T04:49:18.408048+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:18.408048+0000","last_undegraded":"2024-09-16T04:49:18.408048+0000","last_fullsized":"2024-09-16T04:49:18.408048+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:22.079404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21670424,"num_objects":2646,"num_object_clones":0,"num_object_copies":10584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10107,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9995","reported_seq":10102,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.080200+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:17.080200+0000","last_peered":"2024-09-16T04:49:17.080200+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:17.080200+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9995,"log_dups_size":0,"ondisk_log_size":9995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22631692,"num_objects":2763,"num_object_clones":0,"num_object_copies":11052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9993,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":574}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9902","reported_seq":10034,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:18.466832+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:18.466832+0000","last_peered":"2024-09-16T04:49:18.466832+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:18.466832+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9902,"log_dups_size":0,"ondisk_log_size":9902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21204882,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9899,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9724","reported_seq":9816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:18.299873+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:18.299873+0000","last_peered":"2024-09-16T04:49:18.299873+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:18.299873+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9724,"log_dups_size":0,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21355142,"num_objects":2607,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9723,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":502}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9889","reported_seq":9978,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.138627+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:17.138627+0000","last_peered":"2024-09-16T04:49:17.138627+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:17.138627+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9889,"log_dups_size":0,"ondisk_log_size":9889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21658246,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9888,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10095","reported_seq":10227,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:18.452493+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:18.452493+0000","last_peered":"2024-09-16T04:49:18.452493+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:18.452493+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22009100,"num_objects":2687,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10093,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":489}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9747","reported_seq":9851,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.091354+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:17.091354+0000","last_peered":"2024-09-16T04:49:17.091354+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:17.091354+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9747,"log_dups_size":0,"ondisk_log_size":9747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22131980,"num_objects":2702,"num_object_clones":0,"num_object_copies":10808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":579}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9768","reported_seq":9862,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.473909+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:17.473909+0000","last_peered":"2024-09-16T04:49:17.473909+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:17.473909+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9768,"log_dups_size":0,"ondisk_log_size":9768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20796684,"num_objects":2539,"num_object_clones":0,"num_object_copies":10156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":311}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":347489984,"num_objects":42424,"num_object_clones":0,"num_object_copies":169696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42424,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159063,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5862,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":693927936,"data_stored":693927936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158990,"ondisk_log_size":158990,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761132,"kb_used_data":169948,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93610708,"statfs":{"total":96636764160,"available":95857364992,"internally_reserved":0,"allocated":174026752,"data_stored":173837550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605351631},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,0,0,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769588,"kb_used_data":170580,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93602252,"statfs":{"total":96636764160,"available":95848706048,"internally_reserved":0,"allocated":174673920,"data_stored":174484718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051679,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597684,"kb_used_data":168932,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93774156,"statfs":{"total":96636764160,"available":96024735744,"internally_reserved":0,"allocated":172986368,"data_stored":172797166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":439021263},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,6,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761496,"kb_used_data":170360,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93610344,"statfs":{"total":96636764160,"available":95856992256,"internally_reserved":0,"allocated":174448640,"data_stored":174259438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":605286094},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":173981696,"data_stored":173981696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172335104,"data_stored":172335104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174272512,"data_stored":174272512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":173338624,"data_stored":173338624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:21.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:49:21.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9949", 'reported_seq': 10064, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:18.385249+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:49:18.385249+0000', 'last_peered': '2024-09-16T04:49:18.385249+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:49:18.385249+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9949, 'log_dups_size': 0, 'ondisk_log_size': 9949, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22159360, 'num_objects': 2705, 'num_object_clones': 0, 'num_object_copies': 10820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9949, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 519}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:21.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:49:21.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9968", 'reported_seq': 10056, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:17.612101+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:49:17.612101+0000', 'last_peered': '2024-09-16T04:49:17.612101+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:49:17.612101+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9968, 'log_dups_size': 0, 'ondisk_log_size': 9968, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22164748, 'num_objects': 2706, 'num_object_clones': 0, 'num_object_copies': 10824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9966, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 503}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:21.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:49:21.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9987", 'reported_seq': 10086, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:17.293372+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:49:17.293372+0000', 'last_peered': '2024-09-16T04:49:17.293372+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:49:17.293372+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9987, 'log_dups_size': 0, 'ondisk_log_size': 9987, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21977758, 'num_objects': 2684, 'num_object_clones': 0, 'num_object_copies': 10736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2684, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9986, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 465}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:21.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:49:21.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10029", 'reported_seq': 12097, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:17.593698+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:49:17.593698+0000', 'last_peered': '2024-09-16T04:49:17.593698+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:49:17.593698+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21632268, 'num_objects': 2641, 'num_object_clones': 0, 'num_object_copies': 10564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1234, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2641, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10027, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 999, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1407}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:21.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:49:21.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10071", 'reported_seq': 10202, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:18.470872+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:49:18.470872+0000', 'last_peered': '2024-09-16T04:49:18.470872+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:49:18.470872+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21869836, 'num_objects': 2670, 'num_object_clones': 0, 'num_object_copies': 10680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2670, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10069, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 422}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:21.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:49:21.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9773", 'reported_seq': 9883, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:17.989725+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:49:17.989725+0000', 'last_peered': '2024-09-16T04:49:17.989725+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:49:17.989725+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9773, 'log_dups_size': 0, 'ondisk_log_size': 9773, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20799488, 'num_objects': 2539, 'num_object_clones': 0, 'num_object_copies': 10156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2539, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9773, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 361}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:21.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:49:21.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9998", 'reported_seq': 10096, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:17.601297+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:49:17.601297+0000', 'last_peered': '2024-09-16T04:49:17.601297+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:49:17.601297+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9998, 'log_dups_size': 0, 'ondisk_log_size': 9998, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21312780, 'num_objects': 2602, 'num_object_clones': 0, 'num_object_copies': 10408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9996, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 389}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:21.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:49:21.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9995", 'reported_seq': 10102, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:17.080200+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:49:17.080200+0000', 'last_peered': '2024-09-16T04:49:17.080200+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:49:17.080200+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9995, 'log_dups_size': 0, 'ondisk_log_size': 9995, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22631692, 'num_objects': 2763, 'num_object_clones': 0, 'num_object_copies': 11052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2763, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9993, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 574}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:21.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:49:21.639 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9902", 'reported_seq': 10034, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:18.466832+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:49:18.466832+0000', 'last_peered': '2024-09-16T04:49:18.466832+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:49:18.466832+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9902, 'log_dups_size': 0, 'ondisk_log_size': 9902, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21204882, 'num_objects': 2589, 'num_object_clones': 0, 'num_object_copies': 10356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9899, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 434}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:21.639 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:49:21.640 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9724", 'reported_seq': 9816, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:18.299873+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:49:18.299873+0000', 'last_peered': '2024-09-16T04:49:18.299873+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:49:18.299873+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9724, 'log_dups_size': 0, 'ondisk_log_size': 9724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21355142, 'num_objects': 2607, 'num_object_clones': 0, 'num_object_copies': 10428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2607, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9723, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 502}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:21.640 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:49:21.640 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9889", 'reported_seq': 9978, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:17.138627+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:49:17.138627+0000', 'last_peered': '2024-09-16T04:49:17.138627+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:49:17.138627+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9889, 'log_dups_size': 0, 'ondisk_log_size': 9889, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21658246, 'num_objects': 2644, 'num_object_clones': 0, 'num_object_copies': 10576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2644, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9888, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 423}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:21.640 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:49:21.640 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10095", 'reported_seq': 10227, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:18.452493+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:49:18.452493+0000', 'last_peered': '2024-09-16T04:49:18.452493+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:49:18.452493+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22009100, 'num_objects': 2687, 'num_object_clones': 0, 'num_object_copies': 10748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2687, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10093, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 489}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:21.640 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:49:21.640 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9747", 'reported_seq': 9851, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:17.091354+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:49:17.091354+0000', 'last_peered': '2024-09-16T04:49:17.091354+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:49:17.091354+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9747, 'log_dups_size': 0, 'ondisk_log_size': 9747, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22131980, 'num_objects': 2702, 'num_object_clones': 0, 'num_object_copies': 10808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2702, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9745, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 579}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:21.640 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:49:21.640 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9768", 'reported_seq': 9862, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:17.473909+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:49:17.473909+0000', 'last_peered': '2024-09-16T04:49:17.473909+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:49:17.473909+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9768, 'log_dups_size': 0, 'ondisk_log_size': 9768, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20796684, 'num_objects': 2539, 'num_object_clones': 0, 'num_object_copies': 10156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2539, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9766, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 311}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:21.641 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:49:22.396 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:22.396 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29720/169696 objects degraded (17.514%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":574,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":42424,"data_bytes":347489984,"bytes_used":2959257600,"bytes_avail":383587799040,"bytes_total":386547056640,"degraded_objects":29720,"degraded_total":169696,"degraded_ratio":0.17513671506694323,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":167},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:49:22.425 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:49:22.425 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:23.085 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:23.085 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:23.109 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-09-16T04:49:22.720505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":345458368,"num_objects":42176,"num_object_clones":0,"num_object_copies":168704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42176,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159311,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5862,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159238,"ondisk_log_size":159238,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2874084,"kb_used_data":676500,"kb_used_omap":14,"kb_used_meta":2197489,"kb_avail":374613276,"statfs":{"total":386547056640,"available":383603994624,"internally_reserved":0,"allocated":692736000,"data_stored":691979192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2250229297},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,7,4,11,18,9,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18022400,"num_objects":-2200,"num_object_clones":0,"num_object_copies":-8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2200,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002957"},"pg_stats":[{"pgid":"1.f","version":"35'9949","reported_seq":10064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:18.385249+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:18.385249+0000","last_peered":"2024-09-16T04:49:18.385249+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:18.385249+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22159360,"num_objects":2705,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9949,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":519}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9968","reported_seq":10056,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.612101+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:17.612101+0000","last_peered":"2024-09-16T04:49:17.612101+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:17.612101+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9968,"log_dups_size":0,"ondisk_log_size":9968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22164748,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9966,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10040","reported_seq":10139,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.291696+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:22.291696+0000","last_peered":"2024-09-16T04:49:22.291696+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:22.291696+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21543582,"num_objects":2631,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10039,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":412}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10029","reported_seq":12097,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.593698+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:17.593698+0000","last_peered":"2024-09-16T04:49:17.593698+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:17.593698+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21632268,"num_objects":2641,"num_object_clones":0,"num_object_copies":10564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10027,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1407}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10071","reported_seq":10202,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:18.470872+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:18.470872+0000","last_peered":"2024-09-16T04:49:18.470872+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:18.470872+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21869836,"num_objects":2670,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10069,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9773","reported_seq":9883,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.989725+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:17.989725+0000","last_peered":"2024-09-16T04:49:17.989725+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:17.989725+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9773,"log_dups_size":0,"ondisk_log_size":9773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20799488,"num_objects":2539,"num_object_clones":0,"num_object_copies":10156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9773,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9998","reported_seq":10096,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.601297+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:17.601297+0000","last_peered":"2024-09-16T04:49:17.601297+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:17.601297+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9998,"log_dups_size":0,"ondisk_log_size":9998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21312780,"num_objects":2602,"num_object_clones":0,"num_object_copies":10408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9996,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":389}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10085","reported_seq":14647,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:20.255416+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:20.255416+0000","last_peered":"2024-09-16T04:49:20.255416+0000","last_clean":"2024-09-16T04:49:20.255416+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:20.255416+0000","last_undegraded":"2024-09-16T04:49:20.255416+0000","last_fullsized":"2024-09-16T04:49:20.255416+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:23.970542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22115596,"num_objects":2700,"num_object_clones":0,"num_object_copies":10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10083,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10110","reported_seq":14868,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:18.408048+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:18.408048+0000","last_peered":"2024-09-16T04:49:18.408048+0000","last_clean":"2024-09-16T04:49:18.408048+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:18.408048+0000","last_undegraded":"2024-09-16T04:49:18.408048+0000","last_fullsized":"2024-09-16T04:49:18.408048+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:22.079404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21670424,"num_objects":2646,"num_object_clones":0,"num_object_copies":10584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10107,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10059","reported_seq":10166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.205643+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:22.205643+0000","last_peered":"2024-09-16T04:49:22.205643+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:22.205643+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22107404,"num_objects":2699,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9902","reported_seq":10034,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:18.466832+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:18.466832+0000","last_peered":"2024-09-16T04:49:18.466832+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:18.466832+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9902,"log_dups_size":0,"ondisk_log_size":9902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21204882,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9899,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9724","reported_seq":9816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:18.299873+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:18.299873+0000","last_peered":"2024-09-16T04:49:18.299873+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:18.299873+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9724,"log_dups_size":0,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21355142,"num_objects":2607,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9723,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":502}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9954","reported_seq":10043,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.131738+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:22.131738+0000","last_peered":"2024-09-16T04:49:22.131738+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:22.131738+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9954,"log_dups_size":0,"ondisk_log_size":9954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21125766,"num_objects":2579,"num_object_clones":0,"num_object_copies":10316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9953,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10095","reported_seq":10227,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:18.452493+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:18.452493+0000","last_peered":"2024-09-16T04:49:18.452493+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:18.452493+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22009100,"num_objects":2687,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10093,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":489}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9813","reported_seq":9916,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.006939+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:22.006939+0000","last_peered":"2024-09-16T04:49:22.006939+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:22.006939+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21591308,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":513}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9768","reported_seq":9862,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:17.473909+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:17.473909+0000","last_peered":"2024-09-16T04:49:17.473909+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:17.473909+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9768,"log_dups_size":0,"ondisk_log_size":9768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20796684,"num_objects":2539,"num_object_clones":0,"num_object_copies":10156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":311}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":345458368,"num_objects":42176,"num_object_clones":0,"num_object_copies":168704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42176,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159311,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5862,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":689917952,"data_stored":689917952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159238,"ondisk_log_size":159238,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761132,"kb_used_data":169948,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93610708,"statfs":{"total":96636764160,"available":95857364992,"internally_reserved":0,"allocated":174026752,"data_stored":173837550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605351631},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,0,0,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753772,"kb_used_data":167260,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93618068,"statfs":{"total":96636764160,"available":95864901632,"internally_reserved":0,"allocated":171274240,"data_stored":171085038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,6,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051679,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597684,"kb_used_data":168932,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93774156,"statfs":{"total":96636764160,"available":96024735744,"internally_reserved":0,"allocated":172986368,"data_stored":172797166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":439021263},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,6,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761496,"kb_used_data":170360,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93610344,"statfs":{"total":96636764160,"available":95856992256,"internally_reserved":0,"allocated":174448640,"data_stored":174259438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":605286094},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":173981696,"data_stored":173981696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172335104,"data_stored":172335104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170262528,"data_stored":170262528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":173338624,"data_stored":173338624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:26.111 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:26.695 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:26.696 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:26.707 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-09-16T04:49:24.721037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":339863232,"num_objects":41493,"num_object_clones":0,"num_object_copies":165972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41493,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159994,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5922,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159721,"ondisk_log_size":159721,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2874660,"kb_used_data":668772,"kb_used_omap":14,"kb_used_meta":2205809,"kb_avail":374612700,"statfs":{"total":386547056640,"available":383603404800,"internally_reserved":0,"allocated":684822528,"data_stored":684065720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2258748977},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,8,12,8,21,16,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21700608,"num_objects":-2649,"num_object_clones":0,"num_object_copies":-10596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2649,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":185,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003026"},"pg_stats":[{"pgid":"1.f","version":"35'10016","reported_seq":10132,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.472108+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:23.472108+0000","last_peered":"2024-09-16T04:49:23.472108+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:23.472108+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21610496,"num_objects":2638,"num_object_clones":0,"num_object_copies":10552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10016,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10037","reported_seq":10125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.554083+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:22.554083+0000","last_peered":"2024-09-16T04:49:22.554083+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:22.554083+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21599500,"num_objects":2637,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10040","reported_seq":10139,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.291696+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:22.291696+0000","last_peered":"2024-09-16T04:49:22.291696+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:22.291696+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21543582,"num_objects":2631,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10039,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":412}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10088","reported_seq":12274,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.677531+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:22.677531+0000","last_peered":"2024-09-16T04:49:22.677531+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:22.677531+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21148940,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10086,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1059,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1408}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10124","reported_seq":10257,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.172965+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:23.172965+0000","last_peered":"2024-09-16T04:49:23.172965+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:23.172965+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21435660,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10122,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9830","reported_seq":9940,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.472693+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:23.472693+0000","last_peered":"2024-09-16T04:49:23.472693+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:23.472693+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9830,"log_dups_size":0,"ondisk_log_size":9830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20332544,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":304}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10065","reported_seq":10163,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.748649+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:22.748649+0000","last_peered":"2024-09-16T04:49:22.748649+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:22.748649+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20763916,"num_objects":2535,"num_object_clones":0,"num_object_copies":10140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10063,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":322}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10085","reported_seq":14647,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:20.255416+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:20.255416+0000","last_peered":"2024-09-16T04:49:20.255416+0000","last_clean":"2024-09-16T04:49:20.255416+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:20.255416+0000","last_undegraded":"2024-09-16T04:49:20.255416+0000","last_fullsized":"2024-09-16T04:49:20.255416+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:23.970542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22115596,"num_objects":2700,"num_object_clones":0,"num_object_copies":10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10083,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10176","reported_seq":14933,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:23.430860+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:23.430860+0000","last_peered":"2024-09-16T04:49:23.430860+0000","last_clean":"2024-09-16T04:49:23.430860+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:23.430860+0000","last_undegraded":"2024-09-16T04:49:23.430860+0000","last_fullsized":"2024-09-16T04:49:23.430860+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:27.091090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21129752,"num_objects":2580,"num_object_clones":0,"num_object_copies":10320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10173,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10059","reported_seq":10166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.205643+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:22.205643+0000","last_peered":"2024-09-16T04:49:22.205643+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:22.205643+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22107404,"num_objects":2699,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9963","reported_seq":10096,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.391512+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:23.391512+0000","last_peered":"2024-09-16T04:49:23.391512+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:23.391512+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9963,"log_dups_size":0,"ondisk_log_size":9963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20705170,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9960,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":373}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9782","reported_seq":9874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.289259+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:23.289259+0000","last_peered":"2024-09-16T04:49:23.289259+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:23.289259+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20880006,"num_objects":2549,"num_object_clones":0,"num_object_copies":10196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9781,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":444}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9954","reported_seq":10043,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.131738+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:22.131738+0000","last_peered":"2024-09-16T04:49:22.131738+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:22.131738+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9954,"log_dups_size":0,"ondisk_log_size":9954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21125766,"num_objects":2579,"num_object_clones":0,"num_object_copies":10316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9953,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10158","reported_seq":10290,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.474200+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:23.474200+0000","last_peered":"2024-09-16T04:49:23.474200+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:23.474200+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21493004,"num_objects":2624,"num_object_clones":0,"num_object_copies":10496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10156,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":426}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9813","reported_seq":9916,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.006939+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:22.006939+0000","last_peered":"2024-09-16T04:49:22.006939+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:22.006939+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21591308,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":513}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9831","reported_seq":9925,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.540914+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:22.540914+0000","last_peered":"2024-09-16T04:49:22.540914+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:22.540914+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9831,"log_dups_size":0,"ondisk_log_size":9831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20280588,"num_objects":2476,"num_object_clones":0,"num_object_copies":9904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9829,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":339863232,"num_objects":41493,"num_object_clones":0,"num_object_copies":165972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41493,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159994,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5922,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":681873408,"data_stored":681873408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159721,"ondisk_log_size":159721,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761452,"kb_used_data":166108,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93610388,"statfs":{"total":96636764160,"available":95857037312,"internally_reserved":0,"allocated":170094592,"data_stored":169905390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609611471},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,6,3,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753772,"kb_used_data":167260,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93618068,"statfs":{"total":96636764160,"available":95864901632,"internally_reserved":0,"allocated":171274240,"data_stored":171085038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,6,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051679,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597684,"kb_used_data":168932,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93774156,"statfs":{"total":96636764160,"available":96024735744,"internally_reserved":0,"allocated":172986368,"data_stored":172797166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":439021263},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,6,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761752,"kb_used_data":166472,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93610088,"statfs":{"total":96636764160,"available":95856730112,"internally_reserved":0,"allocated":170467328,"data_stored":170278126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":609545934},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,1,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":169910272,"data_stored":169910272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172335104,"data_stored":172335104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170262528,"data_stored":170262528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":169365504,"data_stored":169365504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:26.708 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:27.312 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:27.312 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:27.338 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-16T04:49:26.721538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":339330752,"num_objects":41428,"num_object_clones":0,"num_object_copies":165712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41428,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160059,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5922,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159686,"ondisk_log_size":159686,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2874924,"kb_used_data":664876,"kb_used_omap":14,"kb_used_meta":2209969,"kb_avail":374612436,"statfs":{"total":386547056640,"available":383603134464,"internally_reserved":0,"allocated":680833024,"data_stored":680076216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2263008817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,7,13,9,18,18,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16769024,"num_objects":-2047,"num_object_clones":0,"num_object_copies":-8188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2047,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003053"},"pg_stats":[{"pgid":"1.f","version":"35'10016","reported_seq":10132,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.472108+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:23.472108+0000","last_peered":"2024-09-16T04:49:23.472108+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:23.472108+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21610496,"num_objects":2638,"num_object_clones":0,"num_object_copies":10552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10016,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10037","reported_seq":10125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.554083+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:22.554083+0000","last_peered":"2024-09-16T04:49:22.554083+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:22.554083+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21599500,"num_objects":2637,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10040","reported_seq":10139,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.291696+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:22.291696+0000","last_peered":"2024-09-16T04:49:22.291696+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:22.291696+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21543582,"num_objects":2631,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10039,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":412}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10088","reported_seq":12274,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.677531+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:22.677531+0000","last_peered":"2024-09-16T04:49:22.677531+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:22.677531+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21148940,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10086,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1059,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1408}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10124","reported_seq":10257,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.172965+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:23.172965+0000","last_peered":"2024-09-16T04:49:23.172965+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:23.172965+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21435660,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10122,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9830","reported_seq":9940,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.472693+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:23.472693+0000","last_peered":"2024-09-16T04:49:23.472693+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:23.472693+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9830,"log_dups_size":0,"ondisk_log_size":9830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20332544,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":304}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10065","reported_seq":10163,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.748649+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:22.748649+0000","last_peered":"2024-09-16T04:49:22.748649+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:22.748649+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20763916,"num_objects":2535,"num_object_clones":0,"num_object_copies":10140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10063,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":322}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10150","reported_seq":14712,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:25.167007+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:25.167007+0000","last_peered":"2024-09-16T04:49:25.167007+0000","last_clean":"2024-09-16T04:49:25.167007+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:25.167007+0000","last_undegraded":"2024-09-16T04:49:25.167007+0000","last_fullsized":"2024-09-16T04:49:25.167007+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:29.018808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21583116,"num_objects":2635,"num_object_clones":0,"num_object_copies":10540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10148,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10176","reported_seq":14933,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:23.430860+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:23.430860+0000","last_peered":"2024-09-16T04:49:23.430860+0000","last_clean":"2024-09-16T04:49:23.430860+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:23.430860+0000","last_undegraded":"2024-09-16T04:49:23.430860+0000","last_fullsized":"2024-09-16T04:49:23.430860+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:27.091090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21129752,"num_objects":2580,"num_object_clones":0,"num_object_copies":10320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10173,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10059","reported_seq":10166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.205643+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:22.205643+0000","last_peered":"2024-09-16T04:49:22.205643+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:22.205643+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22107404,"num_objects":2699,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9963","reported_seq":10096,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.391512+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:23.391512+0000","last_peered":"2024-09-16T04:49:23.391512+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:23.391512+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9963,"log_dups_size":0,"ondisk_log_size":9963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20705170,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9960,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":373}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9782","reported_seq":9874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.289259+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:23.289259+0000","last_peered":"2024-09-16T04:49:23.289259+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:23.289259+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20880006,"num_objects":2549,"num_object_clones":0,"num_object_copies":10196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9781,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":444}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9954","reported_seq":10043,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.131738+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:22.131738+0000","last_peered":"2024-09-16T04:49:22.131738+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:22.131738+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9954,"log_dups_size":0,"ondisk_log_size":9954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21125766,"num_objects":2579,"num_object_clones":0,"num_object_copies":10316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9953,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10158","reported_seq":10290,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.474200+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:23.474200+0000","last_peered":"2024-09-16T04:49:23.474200+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:23.474200+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21493004,"num_objects":2624,"num_object_clones":0,"num_object_copies":10496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10156,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":426}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9813","reported_seq":9916,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.006939+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:22.006939+0000","last_peered":"2024-09-16T04:49:22.006939+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:22.006939+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21591308,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":513}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9831","reported_seq":9925,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.540914+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:22.540914+0000","last_peered":"2024-09-16T04:49:22.540914+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:22.540914+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9831,"log_dups_size":0,"ondisk_log_size":9831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20280588,"num_objects":2476,"num_object_clones":0,"num_object_copies":9904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9829,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":339330752,"num_objects":41428,"num_object_clones":0,"num_object_copies":165712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41428,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160059,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5922,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":677830656,"data_stored":677830656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159686,"ondisk_log_size":159686,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761452,"kb_used_data":166108,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93610388,"statfs":{"total":96636764160,"available":95857037312,"internally_reserved":0,"allocated":170094592,"data_stored":169905390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609611471},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,6,3,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753772,"kb_used_data":167260,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93618068,"statfs":{"total":96636764160,"available":95864901632,"internally_reserved":0,"allocated":171274240,"data_stored":171085038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,6,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051680,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597948,"kb_used_data":165036,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93773892,"statfs":{"total":96636764160,"available":96024465408,"internally_reserved":0,"allocated":168996864,"data_stored":168807662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443281103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,3,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761752,"kb_used_data":166472,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93610088,"statfs":{"total":96636764160,"available":95856730112,"internally_reserved":0,"allocated":170467328,"data_stored":170278126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":609545934},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,1,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":169910272,"data_stored":169910272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168292352,"data_stored":168292352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170262528,"data_stored":170262528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":169365504,"data_stored":169365504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:27.339 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:49:27.340 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10016", 'reported_seq': 10132, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:23.472108+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:49:23.472108+0000', 'last_peered': '2024-09-16T04:49:23.472108+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:49:23.472108+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21610496, 'num_objects': 2638, 'num_object_clones': 0, 'num_object_copies': 10552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2638, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10016, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 452}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:27.340 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:49:27.340 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10037", 'reported_seq': 10125, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:22.554083+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:49:22.554083+0000', 'last_peered': '2024-09-16T04:49:22.554083+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:49:22.554083+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21599500, 'num_objects': 2637, 'num_object_clones': 0, 'num_object_copies': 10548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2637, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10035, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 434}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:27.340 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:49:27.340 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10040", 'reported_seq': 10139, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:22.291696+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:49:22.291696+0000', 'last_peered': '2024-09-16T04:49:22.291696+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:49:22.291696+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21543582, 'num_objects': 2631, 'num_object_clones': 0, 'num_object_copies': 10524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2631, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10039, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 412}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:27.340 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:49:27.340 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10088", 'reported_seq': 12274, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:22.677531+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:49:22.677531+0000', 'last_peered': '2024-09-16T04:49:22.677531+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:49:22.677531+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21148940, 'num_objects': 2582, 'num_object_clones': 0, 'num_object_copies': 10328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1174, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2582, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10086, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1059, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1408}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:27.340 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:49:27.340 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10124", 'reported_seq': 10257, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:23.172965+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:49:23.172965+0000', 'last_peered': '2024-09-16T04:49:23.172965+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:49:23.172965+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21435660, 'num_objects': 2617, 'num_object_clones': 0, 'num_object_copies': 10468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2617, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10122, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 369}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:27.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:49:27.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9830", 'reported_seq': 9940, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:23.472693+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:49:23.472693+0000', 'last_peered': '2024-09-16T04:49:23.472693+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:49:23.472693+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9830, 'log_dups_size': 0, 'ondisk_log_size': 9830, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20332544, 'num_objects': 2482, 'num_object_clones': 0, 'num_object_copies': 9928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9830, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 304}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:27.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:49:27.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10065", 'reported_seq': 10163, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:22.748649+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:49:22.748649+0000', 'last_peered': '2024-09-16T04:49:22.748649+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:49:22.748649+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20763916, 'num_objects': 2535, 'num_object_clones': 0, 'num_object_copies': 10140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2535, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10063, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 322}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:27.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:49:27.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10059", 'reported_seq': 10166, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:22.205643+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:49:22.205643+0000', 'last_peered': '2024-09-16T04:49:22.205643+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:49:22.205643+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22107404, 'num_objects': 2699, 'num_object_clones': 0, 'num_object_copies': 10796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10057, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 510}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:27.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:49:27.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9963", 'reported_seq': 10096, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:23.391512+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:49:23.391512+0000', 'last_peered': '2024-09-16T04:49:23.391512+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:49:23.391512+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9963, 'log_dups_size': 0, 'ondisk_log_size': 9963, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20705170, 'num_objects': 2528, 'num_object_clones': 0, 'num_object_copies': 10112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2528, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9960, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 373}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:27.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:49:27.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9782", 'reported_seq': 9874, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:23.289259+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:49:23.289259+0000', 'last_peered': '2024-09-16T04:49:23.289259+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:49:23.289259+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9782, 'log_dups_size': 0, 'ondisk_log_size': 9782, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20880006, 'num_objects': 2549, 'num_object_clones': 0, 'num_object_copies': 10196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2549, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9781, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 444}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:27.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:49:27.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9954", 'reported_seq': 10043, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:22.131738+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:49:22.131738+0000', 'last_peered': '2024-09-16T04:49:22.131738+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:49:22.131738+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9954, 'log_dups_size': 0, 'ondisk_log_size': 9954, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21125766, 'num_objects': 2579, 'num_object_clones': 0, 'num_object_copies': 10316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2579, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9953, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 358}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:27.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:49:27.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10158", 'reported_seq': 10290, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:23.474200+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:49:23.474200+0000', 'last_peered': '2024-09-16T04:49:23.474200+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:49:23.474200+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21493004, 'num_objects': 2624, 'num_object_clones': 0, 'num_object_copies': 10496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10156, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 426}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:27.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:49:27.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9813", 'reported_seq': 9916, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:22.006939+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:49:22.006939+0000', 'last_peered': '2024-09-16T04:49:22.006939+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:49:22.006939+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9813, 'log_dups_size': 0, 'ondisk_log_size': 9813, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21591308, 'num_objects': 2636, 'num_object_clones': 0, 'num_object_copies': 10544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9811, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 513}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:27.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:49:27.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9831", 'reported_seq': 9925, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:22.540914+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:49:22.540914+0000', 'last_peered': '2024-09-16T04:49:22.540914+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:49:22.540914+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9831, 'log_dups_size': 0, 'ondisk_log_size': 9831, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20280588, 'num_objects': 2476, 'num_object_clones': 0, 'num_object_copies': 9904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2476, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9829, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 248}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:27.342 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:49:28.057 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:28.058 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29660/165712 objects degraded (17.899%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":580,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":41428,"data_bytes":339330752,"bytes_used":2943922176,"bytes_avail":383603134464,"bytes_total":386547056640,"degraded_objects":29660,"degraded_total":165712,"degraded_ratio":0.17898522738244665,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":170},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:49:28.091 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:49:28.091 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:28.708 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:28.709 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:28.738 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-16T04:49:26.721538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":339330752,"num_objects":41428,"num_object_clones":0,"num_object_copies":165712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41428,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160059,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5922,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159686,"ondisk_log_size":159686,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2874924,"kb_used_data":664876,"kb_used_omap":14,"kb_used_meta":2209969,"kb_avail":374612436,"statfs":{"total":386547056640,"available":383603134464,"internally_reserved":0,"allocated":680833024,"data_stored":680076216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2263008817},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,7,13,9,18,18,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16769024,"num_objects":-2047,"num_object_clones":0,"num_object_copies":-8188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2047,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003053"},"pg_stats":[{"pgid":"1.f","version":"35'10016","reported_seq":10132,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.472108+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:23.472108+0000","last_peered":"2024-09-16T04:49:23.472108+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:23.472108+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21610496,"num_objects":2638,"num_object_clones":0,"num_object_copies":10552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10016,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":452}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10037","reported_seq":10125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.554083+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:22.554083+0000","last_peered":"2024-09-16T04:49:22.554083+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:22.554083+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21599500,"num_objects":2637,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10040","reported_seq":10139,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.291696+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:22.291696+0000","last_peered":"2024-09-16T04:49:22.291696+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:22.291696+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21543582,"num_objects":2631,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10039,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":412}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10088","reported_seq":12274,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.677531+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:22.677531+0000","last_peered":"2024-09-16T04:49:22.677531+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:22.677531+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21148940,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10086,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1059,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1408}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10124","reported_seq":10257,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.172965+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:23.172965+0000","last_peered":"2024-09-16T04:49:23.172965+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:23.172965+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21435660,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10122,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9830","reported_seq":9940,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.472693+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:23.472693+0000","last_peered":"2024-09-16T04:49:23.472693+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:23.472693+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9830,"log_dups_size":0,"ondisk_log_size":9830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20332544,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":304}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10065","reported_seq":10163,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.748649+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:22.748649+0000","last_peered":"2024-09-16T04:49:22.748649+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:22.748649+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20763916,"num_objects":2535,"num_object_clones":0,"num_object_copies":10140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10063,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":322}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10150","reported_seq":14712,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:25.167007+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:25.167007+0000","last_peered":"2024-09-16T04:49:25.167007+0000","last_clean":"2024-09-16T04:49:25.167007+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:25.167007+0000","last_undegraded":"2024-09-16T04:49:25.167007+0000","last_fullsized":"2024-09-16T04:49:25.167007+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:29.018808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21583116,"num_objects":2635,"num_object_clones":0,"num_object_copies":10540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10148,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10176","reported_seq":14933,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:23.430860+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:23.430860+0000","last_peered":"2024-09-16T04:49:23.430860+0000","last_clean":"2024-09-16T04:49:23.430860+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:23.430860+0000","last_undegraded":"2024-09-16T04:49:23.430860+0000","last_fullsized":"2024-09-16T04:49:23.430860+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:27.091090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21129752,"num_objects":2580,"num_object_clones":0,"num_object_copies":10320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10173,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10059","reported_seq":10166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.205643+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:22.205643+0000","last_peered":"2024-09-16T04:49:22.205643+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:22.205643+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22107404,"num_objects":2699,"num_object_clones":0,"num_object_copies":10796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9963","reported_seq":10096,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.391512+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:23.391512+0000","last_peered":"2024-09-16T04:49:23.391512+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:23.391512+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9963,"log_dups_size":0,"ondisk_log_size":9963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20705170,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9960,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":373}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9782","reported_seq":9874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.289259+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:23.289259+0000","last_peered":"2024-09-16T04:49:23.289259+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:23.289259+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20880006,"num_objects":2549,"num_object_clones":0,"num_object_copies":10196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9781,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":444}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9954","reported_seq":10043,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.131738+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:22.131738+0000","last_peered":"2024-09-16T04:49:22.131738+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:22.131738+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9954,"log_dups_size":0,"ondisk_log_size":9954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21125766,"num_objects":2579,"num_object_clones":0,"num_object_copies":10316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9953,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10158","reported_seq":10290,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:23.474200+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:23.474200+0000","last_peered":"2024-09-16T04:49:23.474200+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:23.474200+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21493004,"num_objects":2624,"num_object_clones":0,"num_object_copies":10496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10156,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":426}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9813","reported_seq":9916,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.006939+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:22.006939+0000","last_peered":"2024-09-16T04:49:22.006939+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:22.006939+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21591308,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":513}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9831","reported_seq":9925,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:22.540914+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:22.540914+0000","last_peered":"2024-09-16T04:49:22.540914+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:22.540914+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9831,"log_dups_size":0,"ondisk_log_size":9831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20280588,"num_objects":2476,"num_object_clones":0,"num_object_copies":9904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9829,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":339330752,"num_objects":41428,"num_object_clones":0,"num_object_copies":165712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41428,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160059,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5922,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":677830656,"data_stored":677830656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159686,"ondisk_log_size":159686,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761452,"kb_used_data":166108,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93610388,"statfs":{"total":96636764160,"available":95857037312,"internally_reserved":0,"allocated":170094592,"data_stored":169905390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609611471},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,6,3,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753772,"kb_used_data":167260,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93618068,"statfs":{"total":96636764160,"available":95864901632,"internally_reserved":0,"allocated":171274240,"data_stored":171085038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,6,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051680,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597948,"kb_used_data":165036,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93773892,"statfs":{"total":96636764160,"available":96024465408,"internally_reserved":0,"allocated":168996864,"data_stored":168807662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443281103},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,3,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761752,"kb_used_data":166472,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93610088,"statfs":{"total":96636764160,"available":95856730112,"internally_reserved":0,"allocated":170467328,"data_stored":170278126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":609545934},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,1,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":169910272,"data_stored":169910272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168292352,"data_stored":168292352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170262528,"data_stored":170262528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":169365504,"data_stored":169365504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:31.740 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:32.343 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:32.344 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:32.367 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-09-16T04:49:30.722615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":331269824,"num_objects":40444,"num_object_clones":0,"num_object_copies":161776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161043,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5974,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159970,"ondisk_log_size":159970,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2875964,"kb_used_data":649272,"kb_used_omap":14,"kb_used_meta":2226609,"kb_avail":374611396,"statfs":{"total":386547056640,"available":383602069504,"internally_reserved":0,"allocated":664854528,"data_stored":664097720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2280048177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,1,2,17,12,11,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16711680,"num_objects":-2040,"num_object_clones":0,"num_object_copies":-8160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2040,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003102"},"pg_stats":[{"pgid":"1.f","version":"35'10074","reported_seq":10190,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.493737+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:28.493737+0000","last_peered":"2024-09-16T04:49:28.493737+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:28.493737+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21135360,"num_objects":2580,"num_object_clones":0,"num_object_copies":10320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10104","reported_seq":10192,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.912881+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:27.912881+0000","last_peered":"2024-09-16T04:49:27.912881+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:27.912881+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21050636,"num_objects":2570,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10102,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10095","reported_seq":10193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.321575+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:27.321575+0000","last_peered":"2024-09-16T04:49:27.321575+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:27.321575+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21093022,"num_objects":2576,"num_object_clones":0,"num_object_copies":10304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2576,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10094,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":357}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10149","reported_seq":12441,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.960491+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:27.960491+0000","last_peered":"2024-09-16T04:49:27.960491+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:27.960491+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20649228,"num_objects":2521,"num_object_clones":0,"num_object_copies":10084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10147,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1399}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10191","reported_seq":10323,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.586802+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:28.586802+0000","last_peered":"2024-09-16T04:49:28.586802+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:28.586802+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20886796,"num_objects":2550,"num_object_clones":0,"num_object_copies":10200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10189,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9891","reported_seq":10001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.443545+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:28.443545+0000","last_peered":"2024-09-16T04:49:28.443545+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:28.443545+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19832832,"num_objects":2421,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":243}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10116","reported_seq":10214,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.857278+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:27.857278+0000","last_peered":"2024-09-16T04:49:27.857278+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:27.857278+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20346124,"num_objects":2484,"num_object_clones":0,"num_object_copies":9936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10114,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":271}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10206","reported_seq":14768,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:30.452911+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:30.452911+0000","last_peered":"2024-09-16T04:49:30.452911+0000","last_clean":"2024-09-16T04:49:30.452911+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:30.452911+0000","last_undegraded":"2024-09-16T04:49:30.452911+0000","last_fullsized":"2024-09-16T04:49:30.452911+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:35.001803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21124364,"num_objects":2579,"num_object_clones":0,"num_object_copies":10316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10204,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10237","reported_seq":14995,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:28.470381+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:28.470381+0000","last_peered":"2024-09-16T04:49:28.470381+0000","last_clean":"2024-09-16T04:49:28.470381+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:28.470381+0000","last_undegraded":"2024-09-16T04:49:28.470381+0000","last_fullsized":"2024-09-16T04:49:28.470381+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:32.148492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20630040,"num_objects":2519,"num_object_clones":0,"num_object_copies":10076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10234,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10127","reported_seq":10234,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.263360+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:27.263360+0000","last_peered":"2024-09-16T04:49:27.263360+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:27.263360+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21550348,"num_objects":2631,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10125,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":442}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10019","reported_seq":10152,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.499140+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:28.499140+0000","last_peered":"2024-09-16T04:49:28.499140+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:28.499140+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20246418,"num_objects":2472,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10016,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":317}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9851","reported_seq":9941,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.634928+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:28.634928+0000","last_peered":"2024-09-16T04:49:28.634928+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:28.634928+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9851,"log_dups_size":0,"ondisk_log_size":9851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20314758,"num_objects":2480,"num_object_clones":0,"num_object_copies":9920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9850,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":375}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10022","reported_seq":10111,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.227904+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:27.227904+0000","last_peered":"2024-09-16T04:49:27.227904+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:27.227904+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20568710,"num_objects":2511,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10021,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":290}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10215","reported_seq":10347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.356949+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:28.356949+0000","last_peered":"2024-09-16T04:49:28.356949+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:28.356949+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21026060,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10213,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":369}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9880","reported_seq":9984,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.338325+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:27.338325+0000","last_peered":"2024-09-16T04:49:27.338325+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:27.338325+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9880,"log_dups_size":0,"ondisk_log_size":9880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21042444,"num_objects":2569,"num_object_clones":0,"num_object_copies":10276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9878,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":446}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9893","reported_seq":9985,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.980618+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:27.980618+0000","last_peered":"2024-09-16T04:49:27.980618+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:27.980618+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9893,"log_dups_size":0,"ondisk_log_size":9893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19772684,"num_objects":2414,"num_object_clones":0,"num_object_copies":9656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":331269824,"num_objects":40444,"num_object_clones":0,"num_object_copies":161776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161043,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5974,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":661626880,"data_stored":661626880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159970,"ondisk_log_size":159970,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761660,"kb_used_data":162156,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93610180,"statfs":{"total":96636764160,"available":95856824320,"internally_reserved":0,"allocated":166047744,"data_stored":165858542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613871311},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,7,7,2,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753988,"kb_used_data":163316,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93617852,"statfs":{"total":96636764160,"available":95864680448,"internally_reserved":0,"allocated":167235584,"data_stored":167046382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051681,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598296,"kb_used_data":161224,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93773544,"statfs":{"total":96636764160,"available":96024109056,"internally_reserved":0,"allocated":165093376,"data_stored":164904174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447540943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762020,"kb_used_data":162576,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93609820,"statfs":{"total":96636764160,"available":95856455680,"internally_reserved":0,"allocated":166477824,"data_stored":166288622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":613805774},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,0,3,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":165806080,"data_stored":165806080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":164196352,"data_stored":164196352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":166264832,"data_stored":166264832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165359616,"data_stored":165359616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:32.368 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:32.991 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:32.992 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:33.035 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-09-16T04:49:32.723095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":329205440,"num_objects":40192,"num_object_clones":0,"num_object_copies":160768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161295,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5974,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160122,"ondisk_log_size":160122,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2876228,"kb_used_data":645376,"kb_used_omap":14,"kb_used_meta":2230769,"kb_avail":374611132,"statfs":{"total":386547056640,"available":383601799168,"internally_reserved":0,"allocated":660865024,"data_stored":660108216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2284308017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,1,1,17,19,18,17,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18284544,"num_objects":-2232,"num_object_clones":0,"num_object_copies":-8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2232,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003051"},"pg_stats":[{"pgid":"1.f","version":"35'10074","reported_seq":10190,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.493737+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:28.493737+0000","last_peered":"2024-09-16T04:49:28.493737+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:28.493737+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21135360,"num_objects":2580,"num_object_clones":0,"num_object_copies":10320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10104","reported_seq":10192,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.912881+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:27.912881+0000","last_peered":"2024-09-16T04:49:27.912881+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:27.912881+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21050636,"num_objects":2570,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10102,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10155","reported_seq":10254,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.186108+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:32.186108+0000","last_peered":"2024-09-16T04:49:32.186108+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:32.186108+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20601502,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10154,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":297}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10149","reported_seq":12441,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.960491+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:27.960491+0000","last_peered":"2024-09-16T04:49:27.960491+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:27.960491+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20649228,"num_objects":2521,"num_object_clones":0,"num_object_copies":10084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10147,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1399}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10191","reported_seq":10323,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.586802+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:28.586802+0000","last_peered":"2024-09-16T04:49:28.586802+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:28.586802+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20886796,"num_objects":2550,"num_object_clones":0,"num_object_copies":10200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10189,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9891","reported_seq":10001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.443545+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:28.443545+0000","last_peered":"2024-09-16T04:49:28.443545+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:28.443545+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19832832,"num_objects":2421,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":243}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10116","reported_seq":10214,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.857278+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:27.857278+0000","last_peered":"2024-09-16T04:49:27.857278+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:27.857278+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20346124,"num_objects":2484,"num_object_clones":0,"num_object_copies":9936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10114,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":271}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10206","reported_seq":14768,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:30.452911+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:30.452911+0000","last_peered":"2024-09-16T04:49:30.452911+0000","last_clean":"2024-09-16T04:49:30.452911+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:30.452911+0000","last_undegraded":"2024-09-16T04:49:30.452911+0000","last_fullsized":"2024-09-16T04:49:30.452911+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:35.001803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21124364,"num_objects":2579,"num_object_clones":0,"num_object_copies":10316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10204,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10237","reported_seq":14995,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:28.470381+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:28.470381+0000","last_peered":"2024-09-16T04:49:28.470381+0000","last_clean":"2024-09-16T04:49:28.470381+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:28.470381+0000","last_undegraded":"2024-09-16T04:49:28.470381+0000","last_fullsized":"2024-09-16T04:49:28.470381+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:32.148492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20630040,"num_objects":2519,"num_object_clones":0,"num_object_copies":10076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10234,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10198","reported_seq":10303,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.079503+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:32.079503+0000","last_peered":"2024-09-16T04:49:32.079503+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:32.079503+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20968716,"num_objects":2560,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10196,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":371}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10019","reported_seq":10152,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.499140+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:28.499140+0000","last_peered":"2024-09-16T04:49:28.499140+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:28.499140+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20246418,"num_objects":2472,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10016,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":317}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9851","reported_seq":9941,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.634928+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:28.634928+0000","last_peered":"2024-09-16T04:49:28.634928+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:28.634928+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9851,"log_dups_size":0,"ondisk_log_size":9851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20314758,"num_objects":2480,"num_object_clones":0,"num_object_copies":9920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9850,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":375}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10080","reported_seq":10169,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.213938+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:32.213938+0000","last_peered":"2024-09-16T04:49:32.213938+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:32.213938+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20093574,"num_objects":2453,"num_object_clones":0,"num_object_copies":9812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10079,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":232}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10215","reported_seq":10347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.356949+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:28.356949+0000","last_peered":"2024-09-16T04:49:28.356949+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:28.356949+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21026060,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10213,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":369}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9943","reported_seq":10046,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.185202+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:32.185202+0000","last_peered":"2024-09-16T04:49:32.185202+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:32.185202+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9943,"log_dups_size":0,"ondisk_log_size":9943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20526348,"num_objects":2506,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9941,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9893","reported_seq":9985,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.980618+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:27.980618+0000","last_peered":"2024-09-16T04:49:27.980618+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:27.980618+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9893,"log_dups_size":0,"ondisk_log_size":9893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19772684,"num_objects":2414,"num_object_clones":0,"num_object_copies":9656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":329205440,"num_objects":40192,"num_object_clones":0,"num_object_copies":160768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161295,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5974,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":657547264,"data_stored":657547264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160122,"ondisk_log_size":160122,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761660,"kb_used_data":162156,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93610180,"statfs":{"total":96636764160,"available":95856824320,"internally_reserved":0,"allocated":166047744,"data_stored":165858542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613871311},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,7,7,2,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754252,"kb_used_data":159420,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93617588,"statfs":{"total":96636764160,"available":95864410112,"internally_reserved":0,"allocated":163246080,"data_stored":163056878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,7,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051681,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598296,"kb_used_data":161224,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93773544,"statfs":{"total":96636764160,"available":96024109056,"internally_reserved":0,"allocated":165093376,"data_stored":164904174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447540943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762020,"kb_used_data":162576,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93609820,"statfs":{"total":96636764160,"available":95856455680,"internally_reserved":0,"allocated":166477824,"data_stored":166288622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":613805774},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,0,3,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":165806080,"data_stored":165806080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":164196352,"data_stored":164196352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162185216,"data_stored":162185216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165359616,"data_stored":165359616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:33.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:49:33.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10074", 'reported_seq': 10190, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:28.493737+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:49:28.493737+0000', 'last_peered': '2024-09-16T04:49:28.493737+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:49:28.493737+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21135360, 'num_objects': 2580, 'num_object_clones': 0, 'num_object_copies': 10320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2580, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10074, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 394}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:33.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:49:33.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10104", 'reported_seq': 10192, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:27.912881+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:49:27.912881+0000', 'last_peered': '2024-09-16T04:49:27.912881+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:49:27.912881+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21050636, 'num_objects': 2570, 'num_object_clones': 0, 'num_object_copies': 10280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10102, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 367}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:33.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:49:33.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10155", 'reported_seq': 10254, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:32.186108+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:49:32.186108+0000', 'last_peered': '2024-09-16T04:49:32.186108+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:49:32.186108+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20601502, 'num_objects': 2516, 'num_object_clones': 0, 'num_object_copies': 10064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2516, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10154, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 297}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:33.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:49:33.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10149", 'reported_seq': 12441, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:27.960491+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:49:27.960491+0000', 'last_peered': '2024-09-16T04:49:27.960491+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:49:27.960491+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20649228, 'num_objects': 2521, 'num_object_clones': 0, 'num_object_copies': 10084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1122, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10147, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1399}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:33.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:49:33.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10191", 'reported_seq': 10323, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:28.586802+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:49:28.586802+0000', 'last_peered': '2024-09-16T04:49:28.586802+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:49:28.586802+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20886796, 'num_objects': 2550, 'num_object_clones': 0, 'num_object_copies': 10200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2550, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10189, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 302}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:33.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:49:33.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9891", 'reported_seq': 10001, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:28.443545+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:49:28.443545+0000', 'last_peered': '2024-09-16T04:49:28.443545+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:49:28.443545+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9891, 'log_dups_size': 0, 'ondisk_log_size': 9891, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19832832, 'num_objects': 2421, 'num_object_clones': 0, 'num_object_copies': 9684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9891, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 243}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:33.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:49:33.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10116", 'reported_seq': 10214, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:27.857278+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:49:27.857278+0000', 'last_peered': '2024-09-16T04:49:27.857278+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:49:27.857278+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20346124, 'num_objects': 2484, 'num_object_clones': 0, 'num_object_copies': 9936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10114, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 271}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:33.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:49:33.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10198", 'reported_seq': 10303, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:32.079503+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:49:32.079503+0000', 'last_peered': '2024-09-16T04:49:32.079503+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:49:32.079503+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20968716, 'num_objects': 2560, 'num_object_clones': 0, 'num_object_copies': 10240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2560, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10196, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 371}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:33.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:49:33.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10019", 'reported_seq': 10152, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:28.499140+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:49:28.499140+0000', 'last_peered': '2024-09-16T04:49:28.499140+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:49:28.499140+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20246418, 'num_objects': 2472, 'num_object_clones': 0, 'num_object_copies': 9888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10016, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 317}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:33.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:49:33.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9851", 'reported_seq': 9941, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:28.634928+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:49:28.634928+0000', 'last_peered': '2024-09-16T04:49:28.634928+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:49:28.634928+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9851, 'log_dups_size': 0, 'ondisk_log_size': 9851, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20314758, 'num_objects': 2480, 'num_object_clones': 0, 'num_object_copies': 9920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9850, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 375}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:33.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:49:33.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10080", 'reported_seq': 10169, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:32.213938+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:49:32.213938+0000', 'last_peered': '2024-09-16T04:49:32.213938+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:49:32.213938+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20093574, 'num_objects': 2453, 'num_object_clones': 0, 'num_object_copies': 9812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10079, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 232}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:33.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:49:33.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10215", 'reported_seq': 10347, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:28.356949+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:49:28.356949+0000', 'last_peered': '2024-09-16T04:49:28.356949+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:49:28.356949+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21026060, 'num_objects': 2567, 'num_object_clones': 0, 'num_object_copies': 10268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2567, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10213, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 369}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:33.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:49:33.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9943", 'reported_seq': 10046, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:32.185202+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:49:32.185202+0000', 'last_peered': '2024-09-16T04:49:32.185202+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:49:32.185202+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9943, 'log_dups_size': 0, 'ondisk_log_size': 9943, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20526348, 'num_objects': 2506, 'num_object_clones': 0, 'num_object_copies': 10024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2506, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9941, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 383}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:33.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:49:33.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9893", 'reported_seq': 9985, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:27.980618+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:49:27.980618+0000', 'last_peered': '2024-09-16T04:49:27.980618+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:49:27.980618+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9893, 'log_dups_size': 0, 'ondisk_log_size': 9893, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19772684, 'num_objects': 2414, 'num_object_clones': 0, 'num_object_copies': 9656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9891, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 186}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:33.039 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:49:33.764 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:33.764 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29608/160768 objects degraded (18.417%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":586,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40192,"data_bytes":329205440,"bytes_used":2945257472,"bytes_avail":383601799168,"bytes_total":386547056640,"degraded_objects":29608,"degraded_total":160768,"degraded_ratio":0.18416600318471338,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":185},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:49:33.790 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:49:33.791 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:34.354 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:34.354 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:34.381 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-09-16T04:49:32.723095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":329205440,"num_objects":40192,"num_object_clones":0,"num_object_copies":160768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161295,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5974,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160122,"ondisk_log_size":160122,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2876228,"kb_used_data":645376,"kb_used_omap":14,"kb_used_meta":2230769,"kb_avail":374611132,"statfs":{"total":386547056640,"available":383601799168,"internally_reserved":0,"allocated":660865024,"data_stored":660108216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2284308017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,1,1,17,19,18,17,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18284544,"num_objects":-2232,"num_object_clones":0,"num_object_copies":-8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2232,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003051"},"pg_stats":[{"pgid":"1.f","version":"35'10074","reported_seq":10190,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.493737+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:28.493737+0000","last_peered":"2024-09-16T04:49:28.493737+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:28.493737+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21135360,"num_objects":2580,"num_object_clones":0,"num_object_copies":10320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10104","reported_seq":10192,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.912881+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:27.912881+0000","last_peered":"2024-09-16T04:49:27.912881+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:27.912881+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21050636,"num_objects":2570,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10102,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10155","reported_seq":10254,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.186108+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:32.186108+0000","last_peered":"2024-09-16T04:49:32.186108+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:32.186108+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20601502,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10154,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":297}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10149","reported_seq":12441,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.960491+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:27.960491+0000","last_peered":"2024-09-16T04:49:27.960491+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:27.960491+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20649228,"num_objects":2521,"num_object_clones":0,"num_object_copies":10084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10147,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1399}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10191","reported_seq":10323,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.586802+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:28.586802+0000","last_peered":"2024-09-16T04:49:28.586802+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:28.586802+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20886796,"num_objects":2550,"num_object_clones":0,"num_object_copies":10200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10189,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9891","reported_seq":10001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.443545+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:28.443545+0000","last_peered":"2024-09-16T04:49:28.443545+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:28.443545+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19832832,"num_objects":2421,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":243}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10116","reported_seq":10214,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.857278+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:27.857278+0000","last_peered":"2024-09-16T04:49:27.857278+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:27.857278+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20346124,"num_objects":2484,"num_object_clones":0,"num_object_copies":9936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10114,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":271}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10206","reported_seq":14768,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:30.452911+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:30.452911+0000","last_peered":"2024-09-16T04:49:30.452911+0000","last_clean":"2024-09-16T04:49:30.452911+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:30.452911+0000","last_undegraded":"2024-09-16T04:49:30.452911+0000","last_fullsized":"2024-09-16T04:49:30.452911+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:35.001803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21124364,"num_objects":2579,"num_object_clones":0,"num_object_copies":10316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10204,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10237","reported_seq":14995,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:28.470381+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:28.470381+0000","last_peered":"2024-09-16T04:49:28.470381+0000","last_clean":"2024-09-16T04:49:28.470381+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:28.470381+0000","last_undegraded":"2024-09-16T04:49:28.470381+0000","last_fullsized":"2024-09-16T04:49:28.470381+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:32.148492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20630040,"num_objects":2519,"num_object_clones":0,"num_object_copies":10076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10234,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10198","reported_seq":10303,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.079503+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:32.079503+0000","last_peered":"2024-09-16T04:49:32.079503+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:32.079503+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20968716,"num_objects":2560,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10196,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":371}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10019","reported_seq":10152,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.499140+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:28.499140+0000","last_peered":"2024-09-16T04:49:28.499140+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:28.499140+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20246418,"num_objects":2472,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10016,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":317}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9851","reported_seq":9941,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.634928+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:28.634928+0000","last_peered":"2024-09-16T04:49:28.634928+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:28.634928+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9851,"log_dups_size":0,"ondisk_log_size":9851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20314758,"num_objects":2480,"num_object_clones":0,"num_object_copies":9920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9850,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":375}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10080","reported_seq":10169,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.213938+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:32.213938+0000","last_peered":"2024-09-16T04:49:32.213938+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:32.213938+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20093574,"num_objects":2453,"num_object_clones":0,"num_object_copies":9812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10079,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":232}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10215","reported_seq":10347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:28.356949+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:28.356949+0000","last_peered":"2024-09-16T04:49:28.356949+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:28.356949+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21026060,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10213,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":369}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9943","reported_seq":10046,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.185202+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:32.185202+0000","last_peered":"2024-09-16T04:49:32.185202+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:32.185202+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9943,"log_dups_size":0,"ondisk_log_size":9943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20526348,"num_objects":2506,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9941,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9893","reported_seq":9985,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:27.980618+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:27.980618+0000","last_peered":"2024-09-16T04:49:27.980618+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:27.980618+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9893,"log_dups_size":0,"ondisk_log_size":9893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19772684,"num_objects":2414,"num_object_clones":0,"num_object_copies":9656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":329205440,"num_objects":40192,"num_object_clones":0,"num_object_copies":160768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161295,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5974,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":657547264,"data_stored":657547264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160122,"ondisk_log_size":160122,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761660,"kb_used_data":162156,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93610180,"statfs":{"total":96636764160,"available":95856824320,"internally_reserved":0,"allocated":166047744,"data_stored":165858542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613871311},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,7,7,2,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754252,"kb_used_data":159420,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93617588,"statfs":{"total":96636764160,"available":95864410112,"internally_reserved":0,"allocated":163246080,"data_stored":163056878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,7,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051681,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598296,"kb_used_data":161224,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93773544,"statfs":{"total":96636764160,"available":96024109056,"internally_reserved":0,"allocated":165093376,"data_stored":164904174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447540943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762020,"kb_used_data":162576,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93609820,"statfs":{"total":96636764160,"available":95856455680,"internally_reserved":0,"allocated":166477824,"data_stored":166288622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":613805774},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,0,3,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":165806080,"data_stored":165806080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":164196352,"data_stored":164196352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162185216,"data_stored":162185216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165359616,"data_stored":165359616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:37.383 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:38.006 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:38.007 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:38.038 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-09-16T04:49:36.724130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":323135168,"num_objects":39451,"num_object_clones":0,"num_object_copies":157804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162036,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6031,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160563,"ondisk_log_size":160563,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2861812,"kb_used_data":633712,"kb_used_omap":14,"kb_used_meta":2228017,"kb_avail":374625548,"statfs":{"total":386547056640,"available":383616561152,"internally_reserved":0,"allocated":648921088,"data_stored":648164280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2281489969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,11,16,12,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16728064,"num_objects":-2042,"num_object_clones":0,"num_object_copies":-8168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2042,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003093"},"pg_stats":[{"pgid":"1.f","version":"35'10134","reported_seq":10250,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.482962+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:33.482962+0000","last_peered":"2024-09-16T04:49:33.482962+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:33.482962+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20643840,"num_objects":2520,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10134,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10179","reported_seq":10267,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.084221+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:33.084221+0000","last_peered":"2024-09-16T04:49:33.084221+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:33.084221+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20436236,"num_objects":2495,"num_object_clones":0,"num_object_copies":9980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10177,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":292}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10155","reported_seq":10254,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.186108+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:32.186108+0000","last_peered":"2024-09-16T04:49:32.186108+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:32.186108+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20601502,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10154,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":297}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10199","reported_seq":12605,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.246296+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:33.246296+0000","last_peered":"2024-09-16T04:49:33.246296+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:33.246296+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20239628,"num_objects":2471,"num_object_clones":0,"num_object_copies":9884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10197,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1168,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10252","reported_seq":10385,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.436451+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:33.436451+0000","last_peered":"2024-09-16T04:49:33.436451+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:33.436451+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20387084,"num_objects":2489,"num_object_clones":0,"num_object_copies":9956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10250,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":241}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9955","reported_seq":10062,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.555532+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:33.555532+0000","last_peered":"2024-09-16T04:49:33.555532+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:33.555532+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9955,"log_dups_size":0,"ondisk_log_size":9955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19308544,"num_objects":2357,"num_object_clones":0,"num_object_copies":9428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9955,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":179}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10166","reported_seq":10264,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.112044+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:33.112044+0000","last_peered":"2024-09-16T04:49:33.112044+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:33.112044+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19936524,"num_objects":2434,"num_object_clones":0,"num_object_copies":9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10164,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":221}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10273","reported_seq":14832,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:35.713257+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:35.713257+0000","last_peered":"2024-09-16T04:49:35.713257+0000","last_clean":"2024-09-16T04:49:35.713257+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:35.713257+0000","last_undegraded":"2024-09-16T04:49:35.713257+0000","last_fullsized":"2024-09-16T04:49:35.713257+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:40.009134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20575500,"num_objects":2512,"num_object_clones":0,"num_object_copies":10048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10271,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10306","reported_seq":15062,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:33.628518+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:33.628518+0000","last_peered":"2024-09-16T04:49:33.628518+0000","last_clean":"2024-09-16T04:49:33.628518+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:33.628518+0000","last_undegraded":"2024-09-16T04:49:33.628518+0000","last_fullsized":"2024-09-16T04:49:33.628518+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:38.084090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20064792,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10303,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10198","reported_seq":10303,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.079503+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:32.079503+0000","last_peered":"2024-09-16T04:49:32.079503+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:32.079503+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20968716,"num_objects":2560,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10196,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":371}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10079","reported_seq":10212,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.576576+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:33.576576+0000","last_peered":"2024-09-16T04:49:33.576576+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:33.576576+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19754898,"num_objects":2412,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10076,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9912","reported_seq":10002,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.770224+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:33.770224+0000","last_peered":"2024-09-16T04:49:33.770224+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:33.770224+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9912,"log_dups_size":0,"ondisk_log_size":9912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19815046,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9911,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":314}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10080","reported_seq":10169,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.213938+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:32.213938+0000","last_peered":"2024-09-16T04:49:32.213938+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:32.213938+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20093574,"num_objects":2453,"num_object_clones":0,"num_object_copies":9812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10079,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":232}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10279","reported_seq":10411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.646862+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:33.646862+0000","last_peered":"2024-09-16T04:49:33.646862+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:33.646862+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20501772,"num_objects":2503,"num_object_clones":0,"num_object_copies":10012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10277,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":305}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9943","reported_seq":10046,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.185202+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:32.185202+0000","last_peered":"2024-09-16T04:49:32.185202+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:32.185202+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9943,"log_dups_size":0,"ondisk_log_size":9943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20526348,"num_objects":2506,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9941,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9953","reported_seq":10047,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.068751+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:33.068751+0000","last_peered":"2024-09-16T04:49:33.068751+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:33.068751+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9953,"log_dups_size":0,"ondisk_log_size":9953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19281164,"num_objects":2354,"num_object_clones":0,"num_object_copies":9416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9951,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":126}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":323135168,"num_objects":39451,"num_object_clones":0,"num_object_copies":157804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162036,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6031,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":645165056,"data_stored":645165056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160563,"ondisk_log_size":160563,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745280,"kb_used_data":158252,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93626560,"statfs":{"total":96636764160,"available":95873597440,"internally_reserved":0,"allocated":162050048,"data_stored":161860846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601091791},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754252,"kb_used_data":159420,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93617588,"statfs":{"total":96636764160,"available":95864410112,"internally_reserved":0,"allocated":163246080,"data_stored":163056878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,7,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616672,"kb_used_data":157392,"kb_used_omap":4,"kb_used_meta":459259,"kb_avail":93755168,"statfs":{"total":96636764160,"available":96005292032,"internally_reserved":0,"allocated":161169408,"data_stored":160980206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":470281935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745608,"kb_used_data":158648,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93626232,"statfs":{"total":96636764160,"available":95873261568,"internally_reserved":0,"allocated":162455552,"data_stored":162266350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":601026254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161603584,"data_stored":161603584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":160145408,"data_stored":160145408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162185216,"data_stored":162185216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":161230848,"data_stored":161230848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:38.039 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:38.641 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:38.646 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:38.672 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-09-16T04:49:36.724130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":323135168,"num_objects":39451,"num_object_clones":0,"num_object_copies":157804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162036,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6031,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160563,"ondisk_log_size":160563,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2861812,"kb_used_data":633712,"kb_used_omap":14,"kb_used_meta":2228017,"kb_avail":374625548,"statfs":{"total":386547056640,"available":383616561152,"internally_reserved":0,"allocated":648921088,"data_stored":648164280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2281489969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,11,16,12,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16728064,"num_objects":-2042,"num_object_clones":0,"num_object_copies":-8168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2042,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003093"},"pg_stats":[{"pgid":"1.f","version":"35'10134","reported_seq":10250,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.482962+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:33.482962+0000","last_peered":"2024-09-16T04:49:33.482962+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:33.482962+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20643840,"num_objects":2520,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10134,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10179","reported_seq":10267,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.084221+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:33.084221+0000","last_peered":"2024-09-16T04:49:33.084221+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:33.084221+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20436236,"num_objects":2495,"num_object_clones":0,"num_object_copies":9980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10177,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":292}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10155","reported_seq":10254,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.186108+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:32.186108+0000","last_peered":"2024-09-16T04:49:32.186108+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:32.186108+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20601502,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10154,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":297}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10199","reported_seq":12605,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.246296+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:33.246296+0000","last_peered":"2024-09-16T04:49:33.246296+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:33.246296+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20239628,"num_objects":2471,"num_object_clones":0,"num_object_copies":9884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10197,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1168,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10252","reported_seq":10385,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.436451+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:33.436451+0000","last_peered":"2024-09-16T04:49:33.436451+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:33.436451+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20387084,"num_objects":2489,"num_object_clones":0,"num_object_copies":9956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10250,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":241}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9955","reported_seq":10062,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.555532+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:33.555532+0000","last_peered":"2024-09-16T04:49:33.555532+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:33.555532+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9955,"log_dups_size":0,"ondisk_log_size":9955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19308544,"num_objects":2357,"num_object_clones":0,"num_object_copies":9428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9955,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":179}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10166","reported_seq":10264,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.112044+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:33.112044+0000","last_peered":"2024-09-16T04:49:33.112044+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:33.112044+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19936524,"num_objects":2434,"num_object_clones":0,"num_object_copies":9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10164,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":221}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10273","reported_seq":14832,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:35.713257+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:35.713257+0000","last_peered":"2024-09-16T04:49:35.713257+0000","last_clean":"2024-09-16T04:49:35.713257+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:35.713257+0000","last_undegraded":"2024-09-16T04:49:35.713257+0000","last_fullsized":"2024-09-16T04:49:35.713257+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:40.009134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20575500,"num_objects":2512,"num_object_clones":0,"num_object_copies":10048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10271,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10306","reported_seq":15062,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:33.628518+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:33.628518+0000","last_peered":"2024-09-16T04:49:33.628518+0000","last_clean":"2024-09-16T04:49:33.628518+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:33.628518+0000","last_undegraded":"2024-09-16T04:49:33.628518+0000","last_fullsized":"2024-09-16T04:49:33.628518+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:38.084090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20064792,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10303,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10198","reported_seq":10303,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.079503+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:32.079503+0000","last_peered":"2024-09-16T04:49:32.079503+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:32.079503+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20968716,"num_objects":2560,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10196,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":371}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10079","reported_seq":10212,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.576576+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:33.576576+0000","last_peered":"2024-09-16T04:49:33.576576+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:33.576576+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19754898,"num_objects":2412,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10076,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9912","reported_seq":10002,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.770224+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:33.770224+0000","last_peered":"2024-09-16T04:49:33.770224+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:33.770224+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9912,"log_dups_size":0,"ondisk_log_size":9912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19815046,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9911,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":314}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10080","reported_seq":10169,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.213938+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:32.213938+0000","last_peered":"2024-09-16T04:49:32.213938+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:32.213938+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20093574,"num_objects":2453,"num_object_clones":0,"num_object_copies":9812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10079,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":232}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10279","reported_seq":10411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.646862+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:33.646862+0000","last_peered":"2024-09-16T04:49:33.646862+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:33.646862+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20501772,"num_objects":2503,"num_object_clones":0,"num_object_copies":10012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10277,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":305}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9943","reported_seq":10046,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:32.185202+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:32.185202+0000","last_peered":"2024-09-16T04:49:32.185202+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:32.185202+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9943,"log_dups_size":0,"ondisk_log_size":9943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20526348,"num_objects":2506,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9941,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9953","reported_seq":10047,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.068751+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:33.068751+0000","last_peered":"2024-09-16T04:49:33.068751+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:33.068751+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9953,"log_dups_size":0,"ondisk_log_size":9953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19281164,"num_objects":2354,"num_object_clones":0,"num_object_copies":9416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9951,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":126}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":323135168,"num_objects":39451,"num_object_clones":0,"num_object_copies":157804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162036,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6031,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":645165056,"data_stored":645165056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160563,"ondisk_log_size":160563,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745280,"kb_used_data":158252,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93626560,"statfs":{"total":96636764160,"available":95873597440,"internally_reserved":0,"allocated":162050048,"data_stored":161860846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601091791},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754252,"kb_used_data":159420,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93617588,"statfs":{"total":96636764160,"available":95864410112,"internally_reserved":0,"allocated":163246080,"data_stored":163056878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,7,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616672,"kb_used_data":157392,"kb_used_omap":4,"kb_used_meta":459259,"kb_avail":93755168,"statfs":{"total":96636764160,"available":96005292032,"internally_reserved":0,"allocated":161169408,"data_stored":160980206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":470281935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745608,"kb_used_data":158648,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93626232,"statfs":{"total":96636764160,"available":95873261568,"internally_reserved":0,"allocated":162455552,"data_stored":162266350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":601026254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161603584,"data_stored":161603584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":160145408,"data_stored":160145408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162185216,"data_stored":162185216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":161230848,"data_stored":161230848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:38.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:49:38.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10134", 'reported_seq': 10250, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:33.482962+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:49:33.482962+0000', 'last_peered': '2024-09-16T04:49:33.482962+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:49:33.482962+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20643840, 'num_objects': 2520, 'num_object_clones': 0, 'num_object_copies': 10080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2520, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10134, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 334}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:38.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:49:38.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10179", 'reported_seq': 10267, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:33.084221+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:49:33.084221+0000', 'last_peered': '2024-09-16T04:49:33.084221+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:49:33.084221+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20436236, 'num_objects': 2495, 'num_object_clones': 0, 'num_object_copies': 9980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2495, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10177, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 292}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:38.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:49:38.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10155", 'reported_seq': 10254, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:32.186108+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:49:32.186108+0000', 'last_peered': '2024-09-16T04:49:32.186108+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:49:32.186108+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20601502, 'num_objects': 2516, 'num_object_clones': 0, 'num_object_copies': 10064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2516, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10154, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 297}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:38.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:49:38.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10199", 'reported_seq': 12605, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:33.246296+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:49:33.246296+0000', 'last_peered': '2024-09-16T04:49:33.246296+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:49:33.246296+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20239628, 'num_objects': 2471, 'num_object_clones': 0, 'num_object_copies': 9884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2471, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10197, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1168, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1406}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:38.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:49:38.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10252", 'reported_seq': 10385, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:33.436451+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:49:33.436451+0000', 'last_peered': '2024-09-16T04:49:33.436451+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:49:33.436451+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20387084, 'num_objects': 2489, 'num_object_clones': 0, 'num_object_copies': 9956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2489, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10250, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 241}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:38.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:49:38.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9955", 'reported_seq': 10062, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:33.555532+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:49:33.555532+0000', 'last_peered': '2024-09-16T04:49:33.555532+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:49:33.555532+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9955, 'log_dups_size': 0, 'ondisk_log_size': 9955, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19308544, 'num_objects': 2357, 'num_object_clones': 0, 'num_object_copies': 9428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2357, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9955, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 179}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:38.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:49:38.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10166", 'reported_seq': 10264, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:33.112044+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:49:33.112044+0000', 'last_peered': '2024-09-16T04:49:33.112044+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:49:33.112044+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19936524, 'num_objects': 2434, 'num_object_clones': 0, 'num_object_copies': 9736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2434, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10164, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 221}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:38.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:49:38.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10198", 'reported_seq': 10303, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:32.079503+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:49:32.079503+0000', 'last_peered': '2024-09-16T04:49:32.079503+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:49:32.079503+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20968716, 'num_objects': 2560, 'num_object_clones': 0, 'num_object_copies': 10240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2560, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10196, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 371}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:38.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:49:38.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10079", 'reported_seq': 10212, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:33.576576+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:49:33.576576+0000', 'last_peered': '2024-09-16T04:49:33.576576+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:49:33.576576+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19754898, 'num_objects': 2412, 'num_object_clones': 0, 'num_object_copies': 9648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10076, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 257}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:38.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:49:38.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9912", 'reported_seq': 10002, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:33.770224+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:49:33.770224+0000', 'last_peered': '2024-09-16T04:49:33.770224+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:49:33.770224+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9912, 'log_dups_size': 0, 'ondisk_log_size': 9912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19815046, 'num_objects': 2419, 'num_object_clones': 0, 'num_object_copies': 9676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9911, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 314}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:38.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:49:38.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10080", 'reported_seq': 10169, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:32.213938+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:49:32.213938+0000', 'last_peered': '2024-09-16T04:49:32.213938+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:49:32.213938+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20093574, 'num_objects': 2453, 'num_object_clones': 0, 'num_object_copies': 9812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10079, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 232}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:38.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:49:38.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10279", 'reported_seq': 10411, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:33.646862+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:49:33.646862+0000', 'last_peered': '2024-09-16T04:49:33.646862+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:49:33.646862+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20501772, 'num_objects': 2503, 'num_object_clones': 0, 'num_object_copies': 10012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10277, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 305}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:38.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:49:38.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9943", 'reported_seq': 10046, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:32.185202+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:49:32.185202+0000', 'last_peered': '2024-09-16T04:49:32.185202+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:49:32.185202+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9943, 'log_dups_size': 0, 'ondisk_log_size': 9943, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20526348, 'num_objects': 2506, 'num_object_clones': 0, 'num_object_copies': 10024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2506, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9941, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 383}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:38.676 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:49:38.676 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9953", 'reported_seq': 10047, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:33.068751+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:49:33.068751+0000', 'last_peered': '2024-09-16T04:49:33.068751+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:49:33.068751+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9953, 'log_dups_size': 0, 'ondisk_log_size': 9953, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19281164, 'num_objects': 2354, 'num_object_clones': 0, 'num_object_copies': 9416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2354, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9951, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 126}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:38.676 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:49:39.421 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:39.422 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29493/155784 objects degraded (18.932%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":591,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":38946,"data_bytes":318998208,"bytes_used":2930180096,"bytes_avail":383616876544,"bytes_total":386547056640,"degraded_objects":29493,"degraded_total":155784,"degraded_ratio":0.18931982745339704,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":206},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:49:39.453 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:49:39.453 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:40.155 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:40.155 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:40.196 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-09-16T04:49:38.724592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":318998208,"num_objects":38946,"num_object_clones":0,"num_object_copies":155784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38946,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162541,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6089,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160468,"ondisk_log_size":160468,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2861504,"kb_used_data":625040,"kb_used_omap":14,"kb_used_meta":2236337,"kb_avail":374625856,"statfs":{"total":386547056640,"available":383616876544,"internally_reserved":0,"allocated":640040960,"data_stored":639284152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2290009649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,2,13,15,13,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20332544,"num_objects":-2482,"num_object_clones":0,"num_object_copies":-9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2482,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003054"},"pg_stats":[{"pgid":"1.f","version":"35'10134","reported_seq":10250,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.482962+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:33.482962+0000","last_peered":"2024-09-16T04:49:33.482962+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:33.482962+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20643840,"num_objects":2520,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10134,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10258","reported_seq":10344,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.085051+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:38.085051+0000","last_peered":"2024-09-16T04:49:38.085051+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:38.085051+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19789068,"num_objects":2416,"num_object_clones":0,"num_object_copies":9664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10256,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10205","reported_seq":10304,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:37.296038+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:37.296038+0000","last_peered":"2024-09-16T04:49:37.296038+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:37.296038+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20191902,"num_objects":2466,"num_object_clones":0,"num_object_copies":9864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10204,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":247}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10266","reported_seq":12787,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.302840+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:38.302840+0000","last_peered":"2024-09-16T04:49:38.302840+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:38.302840+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19690764,"num_objects":2404,"num_object_clones":0,"num_object_copies":9616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10264,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10252","reported_seq":10385,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.436451+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:33.436451+0000","last_peered":"2024-09-16T04:49:33.436451+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:33.436451+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20387084,"num_objects":2489,"num_object_clones":0,"num_object_copies":9956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10250,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":241}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9955","reported_seq":10062,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.555532+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:33.555532+0000","last_peered":"2024-09-16T04:49:33.555532+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:33.555532+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9955,"log_dups_size":0,"ondisk_log_size":9955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19308544,"num_objects":2357,"num_object_clones":0,"num_object_copies":9428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9955,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":179}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10221","reported_seq":10319,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.089207+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:38.089207+0000","last_peered":"2024-09-16T04:49:38.089207+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:38.089207+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19485964,"num_objects":2379,"num_object_clones":0,"num_object_copies":9516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10219,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":166}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10273","reported_seq":14832,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:35.713257+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:35.713257+0000","last_peered":"2024-09-16T04:49:35.713257+0000","last_clean":"2024-09-16T04:49:35.713257+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:35.713257+0000","last_undegraded":"2024-09-16T04:49:35.713257+0000","last_fullsized":"2024-09-16T04:49:35.713257+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:40.009134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20575500,"num_objects":2512,"num_object_clones":0,"num_object_copies":10048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10271,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10306","reported_seq":15062,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:33.628518+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:33.628518+0000","last_peered":"2024-09-16T04:49:33.628518+0000","last_clean":"2024-09-16T04:49:33.628518+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:33.628518+0000","last_undegraded":"2024-09-16T04:49:33.628518+0000","last_fullsized":"2024-09-16T04:49:33.628518+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:38.084090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20064792,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10303,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10275","reported_seq":10380,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:37.564838+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:37.564838+0000","last_peered":"2024-09-16T04:49:37.564838+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:37.564838+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337932,"num_objects":2483,"num_object_clones":0,"num_object_copies":9932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10273,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":294}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10079","reported_seq":10212,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.576576+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:33.576576+0000","last_peered":"2024-09-16T04:49:33.576576+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:33.576576+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19754898,"num_objects":2412,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10076,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9912","reported_seq":10002,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.770224+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:33.770224+0000","last_peered":"2024-09-16T04:49:33.770224+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:33.770224+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9912,"log_dups_size":0,"ondisk_log_size":9912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19815046,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9911,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":314}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10143","reported_seq":10232,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:37.458111+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:37.458111+0000","last_peered":"2024-09-16T04:49:37.458111+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:37.458111+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19577478,"num_objects":2390,"num_object_clones":0,"num_object_copies":9560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10142,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":169}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10279","reported_seq":10411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:33.646862+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:33.646862+0000","last_peered":"2024-09-16T04:49:33.646862+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:33.646862+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20501772,"num_objects":2503,"num_object_clones":0,"num_object_copies":10012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10277,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":305}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9995","reported_seq":10099,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:37.442333+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:37.442333+0000","last_peered":"2024-09-16T04:49:37.442333+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:37.442333+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9995,"log_dups_size":0,"ondisk_log_size":9995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20100364,"num_objects":2454,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9993,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":331}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10015","reported_seq":10110,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.258149+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:38.258149+0000","last_peered":"2024-09-16T04:49:38.258149+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:38.258149+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18773260,"num_objects":2292,"num_object_clones":0,"num_object_copies":9168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":64}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":318998208,"num_objects":38946,"num_object_clones":0,"num_object_copies":155784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38946,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162541,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6089,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":637108224,"data_stored":637108224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160468,"ondisk_log_size":160468,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745280,"kb_used_data":158252,"kb_used_omap":4,"kb_used_meta":587003,"kb_avail":93626560,"statfs":{"total":96636764160,"available":95873597440,"internally_reserved":0,"allocated":162050048,"data_stored":161860846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601091791},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753812,"kb_used_data":154788,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93618028,"statfs":{"total":96636764160,"available":95864860672,"internally_reserved":0,"allocated":158502912,"data_stored":158313710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616672,"kb_used_data":157392,"kb_used_omap":4,"kb_used_meta":459259,"kb_avail":93755168,"statfs":{"total":96636764160,"available":96005292032,"internally_reserved":0,"allocated":161169408,"data_stored":160980206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":470281935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247921,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745740,"kb_used_data":154608,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93626100,"statfs":{"total":96636764160,"available":95873126400,"internally_reserved":0,"allocated":158318592,"data_stored":158129390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":605286094},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":157589504,"data_stored":157589504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":160145408,"data_stored":160145408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":158142464,"data_stored":158142464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":161230848,"data_stored":161230848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:43.198 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:43.853 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:43.859 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:43.897 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-09-16T04:49:42.725748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":312903360,"num_objects":38202,"num_object_clones":0,"num_object_copies":152808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38202,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163285,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6089,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2846392,"kb_used_data":612688,"kb_used_omap":14,"kb_used_meta":2233585,"kb_avail":374640968,"statfs":{"total":386547056640,"available":383632351232,"internally_reserved":0,"allocated":627392512,"data_stored":626635704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2287191601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,6,12,12,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18366464,"num_objects":-2242,"num_object_clones":0,"num_object_copies":-8968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2242,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003133"},"pg_stats":[{"pgid":"1.f","version":"35'10194","reported_seq":10310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.593113+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:38.593113+0000","last_peered":"2024-09-16T04:49:38.593113+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:38.593113+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20152320,"num_objects":2460,"num_object_clones":0,"num_object_copies":9840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10194,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10258","reported_seq":10344,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.085051+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:38.085051+0000","last_peered":"2024-09-16T04:49:38.085051+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:38.085051+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19789068,"num_objects":2416,"num_object_clones":0,"num_object_copies":9664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10256,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10287","reported_seq":10385,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:42.516576+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:42.516576+0000","last_peered":"2024-09-16T04:49:42.516576+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:42.516576+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19520158,"num_objects":2384,"num_object_clones":0,"num_object_copies":9536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2384,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10286,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":165}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10266","reported_seq":12787,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.302840+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:38.302840+0000","last_peered":"2024-09-16T04:49:38.302840+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:38.302840+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19690764,"num_objects":2404,"num_object_clones":0,"num_object_copies":9616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10264,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10310","reported_seq":10443,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.706969+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:38.706969+0000","last_peered":"2024-09-16T04:49:38.706969+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:38.706969+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19911948,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10308,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10027","reported_seq":10137,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.803586+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:38.803586+0000","last_peered":"2024-09-16T04:49:38.803586+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:38.803586+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18718720,"num_objects":2285,"num_object_clones":0,"num_object_copies":9140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10027,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10221","reported_seq":10319,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.089207+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:38.089207+0000","last_peered":"2024-09-16T04:49:38.089207+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:38.089207+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19485964,"num_objects":2379,"num_object_clones":0,"num_object_copies":9516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10219,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":166}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10335","reported_seq":14897,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:40.893635+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:40.893635+0000","last_peered":"2024-09-16T04:49:40.893635+0000","last_clean":"2024-09-16T04:49:40.893635+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:40.893635+0000","last_undegraded":"2024-09-16T04:49:40.893635+0000","last_fullsized":"2024-09-16T04:49:40.893635+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:45.894130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20067596,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10333,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10361","reported_seq":15119,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:38.784988+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:38.784988+0000","last_peered":"2024-09-16T04:49:38.784988+0000","last_clean":"2024-09-16T04:49:38.784988+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:38.784988+0000","last_undegraded":"2024-09-16T04:49:38.784988+0000","last_fullsized":"2024-09-16T04:49:38.784988+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:43.085467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19614232,"num_objects":2395,"num_object_clones":0,"num_object_copies":9580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10358,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10341","reported_seq":10447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:42.432295+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:42.432295+0000","last_peered":"2024-09-16T04:49:42.432295+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:42.432295+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19797260,"num_objects":2417,"num_object_clones":0,"num_object_copies":9668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10339,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":228}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10148","reported_seq":10281,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.778628+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:38.778628+0000","last_peered":"2024-09-16T04:49:38.778628+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:38.778628+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19189650,"num_objects":2343,"num_object_clones":0,"num_object_copies":9372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10145,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":188}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9970","reported_seq":10063,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.713746+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:38.713746+0000","last_peered":"2024-09-16T04:49:38.713746+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:38.713746+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9970,"log_dups_size":0,"ondisk_log_size":9970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19339910,"num_objects":2361,"num_object_clones":0,"num_object_copies":9444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9969,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":256}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10184","reported_seq":10273,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:41.771589+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:41.771589+0000","last_peered":"2024-09-16T04:49:41.771589+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:41.771589+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19241606,"num_objects":2349,"num_object_clones":0,"num_object_copies":9396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":128}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10341","reported_seq":10474,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.601765+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:38.601765+0000","last_peered":"2024-09-16T04:49:38.601765+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:38.601765+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19993868,"num_objects":2441,"num_object_clones":0,"num_object_copies":9764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10339,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":243}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10054","reported_seq":10158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:42.523703+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:42.523703+0000","last_peered":"2024-09-16T04:49:42.523703+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:42.523703+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19617036,"num_objects":2395,"num_object_clones":0,"num_object_copies":9580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10052,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":272}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10015","reported_seq":10110,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.258149+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:38.258149+0000","last_peered":"2024-09-16T04:49:38.258149+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:38.258149+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18773260,"num_objects":2292,"num_object_clones":0,"num_object_copies":9168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":64}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":312903360,"num_objects":38202,"num_object_clones":0,"num_object_copies":152808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38202,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163285,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6089,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":625008640,"data_stored":625008640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745540,"kb_used_data":154336,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93626300,"statfs":{"total":96636764160,"available":95873331200,"internally_reserved":0,"allocated":158040064,"data_stored":157850862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605351631},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755508,"kb_used_data":150940,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93616332,"statfs":{"total":96636764160,"available":95863123968,"internally_reserved":0,"allocated":154562560,"data_stored":154373358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,4,0,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599604,"kb_used_data":152804,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93772236,"statfs":{"total":96636764160,"available":96022769664,"internally_reserved":0,"allocated":156471296,"data_stored":156282094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":457502415},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247921,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745740,"kb_used_data":154608,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93626100,"statfs":{"total":96636764160,"available":95873126400,"internally_reserved":0,"allocated":158318592,"data_stored":158129390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":605286094},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":157589504,"data_stored":157589504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":156090368,"data_stored":156090368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":154157056,"data_stored":154157056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":157171712,"data_stored":157171712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:43.898 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:44.487 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:44.489 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:44.518 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-09-16T04:49:42.725748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":312903360,"num_objects":38202,"num_object_clones":0,"num_object_copies":152808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38202,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163285,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6089,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2846392,"kb_used_data":612688,"kb_used_omap":14,"kb_used_meta":2233585,"kb_avail":374640968,"statfs":{"total":386547056640,"available":383632351232,"internally_reserved":0,"allocated":627392512,"data_stored":626635704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2287191601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,6,12,12,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18366464,"num_objects":-2242,"num_object_clones":0,"num_object_copies":-8968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2242,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003133"},"pg_stats":[{"pgid":"1.f","version":"35'10194","reported_seq":10310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.593113+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:38.593113+0000","last_peered":"2024-09-16T04:49:38.593113+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:38.593113+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20152320,"num_objects":2460,"num_object_clones":0,"num_object_copies":9840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10194,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10258","reported_seq":10344,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.085051+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:38.085051+0000","last_peered":"2024-09-16T04:49:38.085051+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:38.085051+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19789068,"num_objects":2416,"num_object_clones":0,"num_object_copies":9664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10256,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10287","reported_seq":10385,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:42.516576+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:42.516576+0000","last_peered":"2024-09-16T04:49:42.516576+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:42.516576+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19520158,"num_objects":2384,"num_object_clones":0,"num_object_copies":9536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2384,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10286,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":165}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10266","reported_seq":12787,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.302840+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:38.302840+0000","last_peered":"2024-09-16T04:49:38.302840+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:38.302840+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19690764,"num_objects":2404,"num_object_clones":0,"num_object_copies":9616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10264,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10310","reported_seq":10443,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.706969+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:38.706969+0000","last_peered":"2024-09-16T04:49:38.706969+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:38.706969+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19911948,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10308,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10027","reported_seq":10137,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.803586+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:38.803586+0000","last_peered":"2024-09-16T04:49:38.803586+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:38.803586+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18718720,"num_objects":2285,"num_object_clones":0,"num_object_copies":9140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10027,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10221","reported_seq":10319,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.089207+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:38.089207+0000","last_peered":"2024-09-16T04:49:38.089207+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:38.089207+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19485964,"num_objects":2379,"num_object_clones":0,"num_object_copies":9516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10219,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":166}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10335","reported_seq":14897,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:40.893635+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:40.893635+0000","last_peered":"2024-09-16T04:49:40.893635+0000","last_clean":"2024-09-16T04:49:40.893635+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:40.893635+0000","last_undegraded":"2024-09-16T04:49:40.893635+0000","last_fullsized":"2024-09-16T04:49:40.893635+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:45.894130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20067596,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10333,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10361","reported_seq":15119,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:38.784988+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:38.784988+0000","last_peered":"2024-09-16T04:49:38.784988+0000","last_clean":"2024-09-16T04:49:38.784988+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:38.784988+0000","last_undegraded":"2024-09-16T04:49:38.784988+0000","last_fullsized":"2024-09-16T04:49:38.784988+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:43.085467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19614232,"num_objects":2395,"num_object_clones":0,"num_object_copies":9580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10358,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10341","reported_seq":10447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:42.432295+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:42.432295+0000","last_peered":"2024-09-16T04:49:42.432295+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:42.432295+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19797260,"num_objects":2417,"num_object_clones":0,"num_object_copies":9668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10339,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":228}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10148","reported_seq":10281,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.778628+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:38.778628+0000","last_peered":"2024-09-16T04:49:38.778628+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:38.778628+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19189650,"num_objects":2343,"num_object_clones":0,"num_object_copies":9372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10145,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":188}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9970","reported_seq":10063,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.713746+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:38.713746+0000","last_peered":"2024-09-16T04:49:38.713746+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:38.713746+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":9970,"log_dups_size":0,"ondisk_log_size":9970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19339910,"num_objects":2361,"num_object_clones":0,"num_object_copies":9444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9969,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":256}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10184","reported_seq":10273,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:41.771589+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:41.771589+0000","last_peered":"2024-09-16T04:49:41.771589+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:41.771589+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19241606,"num_objects":2349,"num_object_clones":0,"num_object_copies":9396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":128}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10341","reported_seq":10474,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.601765+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:38.601765+0000","last_peered":"2024-09-16T04:49:38.601765+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:38.601765+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19993868,"num_objects":2441,"num_object_clones":0,"num_object_copies":9764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10339,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":243}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10054","reported_seq":10158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:42.523703+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:42.523703+0000","last_peered":"2024-09-16T04:49:42.523703+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:42.523703+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19617036,"num_objects":2395,"num_object_clones":0,"num_object_copies":9580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10052,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":272}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10015","reported_seq":10110,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:38.258149+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:38.258149+0000","last_peered":"2024-09-16T04:49:38.258149+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:38.258149+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18773260,"num_objects":2292,"num_object_clones":0,"num_object_copies":9168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":64}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":312903360,"num_objects":38202,"num_object_clones":0,"num_object_copies":152808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38202,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163285,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6089,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":625008640,"data_stored":625008640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745540,"kb_used_data":154336,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93626300,"statfs":{"total":96636764160,"available":95873331200,"internally_reserved":0,"allocated":158040064,"data_stored":157850862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605351631},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755508,"kb_used_data":150940,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93616332,"statfs":{"total":96636764160,"available":95863123968,"internally_reserved":0,"allocated":154562560,"data_stored":154373358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,4,0,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599604,"kb_used_data":152804,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93772236,"statfs":{"total":96636764160,"available":96022769664,"internally_reserved":0,"allocated":156471296,"data_stored":156282094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":457502415},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247921,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745740,"kb_used_data":154608,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93626100,"statfs":{"total":96636764160,"available":95873126400,"internally_reserved":0,"allocated":158318592,"data_stored":158129390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":605286094},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":157589504,"data_stored":157589504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":156090368,"data_stored":156090368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":154157056,"data_stored":154157056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":157171712,"data_stored":157171712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:44.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:49:44.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10194", 'reported_seq': 10310, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:38.593113+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:49:38.593113+0000', 'last_peered': '2024-09-16T04:49:38.593113+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:49:38.593113+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20152320, 'num_objects': 2460, 'num_object_clones': 0, 'num_object_copies': 9840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10194, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 274}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:44.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:49:44.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10258", 'reported_seq': 10344, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:38.085051+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:49:38.085051+0000', 'last_peered': '2024-09-16T04:49:38.085051+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:49:38.085051+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19789068, 'num_objects': 2416, 'num_object_clones': 0, 'num_object_copies': 9664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10256, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 213}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:44.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:49:44.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10287", 'reported_seq': 10385, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:42.516576+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:49:42.516576+0000', 'last_peered': '2024-09-16T04:49:42.516576+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:49:42.516576+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19520158, 'num_objects': 2384, 'num_object_clones': 0, 'num_object_copies': 9536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2384, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10286, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 165}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:44.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:49:44.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10266", 'reported_seq': 12787, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:38.302840+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:49:38.302840+0000', 'last_peered': '2024-09-16T04:49:38.302840+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:49:38.302840+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19690764, 'num_objects': 2404, 'num_object_clones': 0, 'num_object_copies': 9616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1007, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10264, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1226, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1397}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:44.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:49:44.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10310", 'reported_seq': 10443, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:38.706969+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:49:38.706969+0000', 'last_peered': '2024-09-16T04:49:38.706969+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:49:38.706969+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19911948, 'num_objects': 2431, 'num_object_clones': 0, 'num_object_copies': 9724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10308, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:44.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:49:44.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10027", 'reported_seq': 10137, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:38.803586+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:49:38.803586+0000', 'last_peered': '2024-09-16T04:49:38.803586+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:49:38.803586+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18718720, 'num_objects': 2285, 'num_object_clones': 0, 'num_object_copies': 9140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2285, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10027, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 107}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:44.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:49:44.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10221", 'reported_seq': 10319, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:38.089207+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:49:38.089207+0000', 'last_peered': '2024-09-16T04:49:38.089207+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:49:38.089207+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19485964, 'num_objects': 2379, 'num_object_clones': 0, 'num_object_copies': 9516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2379, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10219, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 166}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:44.520 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:49:44.520 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10341", 'reported_seq': 10447, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:42.432295+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:49:42.432295+0000', 'last_peered': '2024-09-16T04:49:42.432295+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:49:42.432295+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19797260, 'num_objects': 2417, 'num_object_clones': 0, 'num_object_copies': 9668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2417, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10339, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 228}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:44.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:49:44.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10148", 'reported_seq': 10281, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:38.778628+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:49:38.778628+0000', 'last_peered': '2024-09-16T04:49:38.778628+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:49:38.778628+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19189650, 'num_objects': 2343, 'num_object_clones': 0, 'num_object_copies': 9372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2343, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10145, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 188}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:44.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:49:44.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9970", 'reported_seq': 10063, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:38.713746+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:49:38.713746+0000', 'last_peered': '2024-09-16T04:49:38.713746+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:49:38.713746+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 9970, 'log_dups_size': 0, 'ondisk_log_size': 9970, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19339910, 'num_objects': 2361, 'num_object_clones': 0, 'num_object_copies': 9444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2361, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9969, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 256}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:44.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:49:44.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10184", 'reported_seq': 10273, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:41.771589+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:49:41.771589+0000', 'last_peered': '2024-09-16T04:49:41.771589+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:49:41.771589+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19241606, 'num_objects': 2349, 'num_object_clones': 0, 'num_object_copies': 9396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10183, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 128}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:44.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:49:44.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10341", 'reported_seq': 10474, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:38.601765+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:49:38.601765+0000', 'last_peered': '2024-09-16T04:49:38.601765+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:49:38.601765+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19993868, 'num_objects': 2441, 'num_object_clones': 0, 'num_object_copies': 9764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10339, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 243}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:44.521 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:49:44.521 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10054", 'reported_seq': 10158, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:42.523703+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:49:42.523703+0000', 'last_peered': '2024-09-16T04:49:42.523703+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:49:42.523703+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19617036, 'num_objects': 2395, 'num_object_clones': 0, 'num_object_copies': 9580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10052, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 272}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:44.522 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:49:44.522 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10015", 'reported_seq': 10110, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:38.258149+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:49:38.258149+0000', 'last_peered': '2024-09-16T04:49:38.258149+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:49:38.258149+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18773260, 'num_objects': 2292, 'num_object_clones': 0, 'num_object_copies': 9168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10013, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 64}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:44.522 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:49:45.253 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:45.253 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29432/150028 objects degraded (19.618%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":597,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":37507,"data_bytes":307209920,"bytes_used":2915196928,"bytes_avail":383631859712,"bytes_total":386547056640,"degraded_objects":29432,"degraded_total":150028,"degraded_ratio":0.19617671368011305,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":176,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":223},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:49:45.280 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:49:45.281 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:45.971 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:45.972 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:46.002 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-09-16T04:49:44.726421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":307209920,"num_objects":37507,"num_object_clones":0,"num_object_copies":150028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37507,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163980,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6150,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160807,"ondisk_log_size":160807,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2846872,"kb_used_data":604856,"kb_used_omap":14,"kb_used_meta":2241905,"kb_avail":374640488,"statfs":{"total":386547056640,"available":383631859712,"internally_reserved":0,"allocated":619372544,"data_stored":618615736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2295711281},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,6,7,9,16,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21995520,"num_objects":-2685,"num_object_clones":0,"num_object_copies":-10740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2685,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003325"},"pg_stats":[{"pgid":"1.f","version":"35'10267","reported_seq":10383,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.814222+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:43.814222+0000","last_peered":"2024-09-16T04:49:43.814222+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:43.814222+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19554304,"num_objects":2387,"num_object_clones":0,"num_object_copies":9548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10267,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10309","reported_seq":10398,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.371041+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:43.371041+0000","last_peered":"2024-09-16T04:49:43.371041+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:43.371041+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19371276,"num_objects":2365,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10307,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10287","reported_seq":10385,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:42.516576+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:42.516576+0000","last_peered":"2024-09-16T04:49:42.516576+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:42.516576+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19520158,"num_objects":2384,"num_object_clones":0,"num_object_copies":9536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2384,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10286,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":165}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10324","reported_seq":12967,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.387880+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:43.387880+0000","last_peered":"2024-09-16T04:49:43.387880+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:43.387880+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19215628,"num_objects":2346,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":946,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10322,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1287,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1400}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10360","reported_seq":10493,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.622205+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:43.622205+0000","last_peered":"2024-09-16T04:49:43.622205+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:43.622205+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19502348,"num_objects":2381,"num_object_clones":0,"num_object_copies":9524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10358,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":133}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10094","reported_seq":10203,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.745863+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:43.745863+0000","last_peered":"2024-09-16T04:49:43.745863+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:43.745863+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18169856,"num_objects":2218,"num_object_clones":0,"num_object_copies":8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10094,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":40}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10285","reported_seq":10383,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.259333+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:43.259333+0000","last_peered":"2024-09-16T04:49:43.259333+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:43.259333+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18961676,"num_objects":2315,"num_object_clones":0,"num_object_copies":9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10283,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":102}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10335","reported_seq":14897,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:40.893635+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:40.893635+0000","last_peered":"2024-09-16T04:49:40.893635+0000","last_clean":"2024-09-16T04:49:40.893635+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:40.893635+0000","last_undegraded":"2024-09-16T04:49:40.893635+0000","last_fullsized":"2024-09-16T04:49:40.893635+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:45.894130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20067596,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10333,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10424","reported_seq":15182,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:43.726742+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:43.726742+0000","last_peered":"2024-09-16T04:49:43.726742+0000","last_clean":"2024-09-16T04:49:43.726742+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:43.726742+0000","last_undegraded":"2024-09-16T04:49:43.726742+0000","last_fullsized":"2024-09-16T04:49:43.726742+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:48.134884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19098136,"num_objects":2332,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10421,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10341","reported_seq":10447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:42.432295+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:42.432295+0000","last_peered":"2024-09-16T04:49:42.432295+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:42.432295+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19797260,"num_objects":2417,"num_object_clones":0,"num_object_copies":9668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10339,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":228}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10203","reported_seq":10334,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.680842+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:43.680842+0000","last_peered":"2024-09-16T04:49:43.680842+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:43.680842+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18739090,"num_objects":2288,"num_object_clones":0,"num_object_copies":9152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10200,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":133}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10038","reported_seq":10130,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.851187+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:43.851187+0000","last_peered":"2024-09-16T04:49:43.851187+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:43.851187+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18782854,"num_objects":2293,"num_object_clones":0,"num_object_copies":9172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10184","reported_seq":10273,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:41.771589+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:41.771589+0000","last_peered":"2024-09-16T04:49:41.771589+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:41.771589+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19241606,"num_objects":2349,"num_object_clones":0,"num_object_copies":9396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":128}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10422","reported_seq":10555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.771774+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:43.771774+0000","last_peered":"2024-09-16T04:49:43.771774+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:43.771774+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330316,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10420,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":162}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10054","reported_seq":10158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:42.523703+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:42.523703+0000","last_peered":"2024-09-16T04:49:42.523703+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:42.523703+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19617036,"num_objects":2395,"num_object_clones":0,"num_object_copies":9580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10052,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":272}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10080","reported_seq":10174,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.392953+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:43.392953+0000","last_peered":"2024-09-16T04:49:43.392953+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:43.392953+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18240780,"num_objects":2227,"num_object_clones":0,"num_object_copies":8908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10078,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":307209920,"num_objects":37507,"num_object_clones":0,"num_object_copies":150028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37507,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163980,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6150,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":616853504,"data_stored":616853504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160807,"ondisk_log_size":160807,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745744,"kb_used_data":150392,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93626096,"statfs":{"total":96636764160,"available":95873122304,"internally_reserved":0,"allocated":154001408,"data_stored":153812206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609611471},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755508,"kb_used_data":150940,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93616332,"statfs":{"total":96636764160,"available":95863123968,"internally_reserved":0,"allocated":154562560,"data_stored":154373358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,4,0,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599604,"kb_used_data":152804,"kb_used_omap":4,"kb_used_meta":446779,"kb_avail":93772236,"statfs":{"total":96636764160,"available":96022769664,"internally_reserved":0,"allocated":156471296,"data_stored":156282094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":457502415},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746016,"kb_used_data":150720,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93625824,"statfs":{"total":96636764160,"available":95872843776,"internally_reserved":0,"allocated":154337280,"data_stored":154148078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":609545934},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,5,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153477120,"data_stored":153477120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":156090368,"data_stored":156090368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":154157056,"data_stored":154157056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":153128960,"data_stored":153128960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:49.004 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:49.644 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:49.645 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:49.669 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-09-16T04:49:48.727646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":302974656,"num_objects":36990,"num_object_clones":0,"num_object_copies":147960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36990,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164497,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6203,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160824,"ondisk_log_size":160824,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2866520,"kb_used_data":593956,"kb_used_omap":14,"kb_used_meta":2272433,"kb_avail":374620840,"statfs":{"total":386547056640,"available":383611740160,"internally_reserved":0,"allocated":608210944,"data_stored":607454136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2326971953},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,9,4,0,4,13,16,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20160512,"num_objects":-2461,"num_object_clones":0,"num_object_copies":-9844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2461,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003515"},"pg_stats":[{"pgid":"1.f","version":"35'10267","reported_seq":10383,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.814222+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:43.814222+0000","last_peered":"2024-09-16T04:49:43.814222+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:43.814222+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19554304,"num_objects":2387,"num_object_clones":0,"num_object_copies":9548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10267,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10366","reported_seq":10455,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.413206+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:48.413206+0000","last_peered":"2024-09-16T04:49:48.413206+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:48.413206+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18904332,"num_objects":2308,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10364,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10343","reported_seq":10440,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:47.538971+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:47.538971+0000","last_peered":"2024-09-16T04:49:47.538971+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:47.538971+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19061406,"num_objects":2328,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10342,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10373","reported_seq":13123,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.494835+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:48.494835+0000","last_peered":"2024-09-16T04:49:48.494835+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:48.494835+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18814220,"num_objects":2297,"num_object_clones":0,"num_object_copies":9188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10371,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1340,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10360","reported_seq":10493,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.622205+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:43.622205+0000","last_peered":"2024-09-16T04:49:43.622205+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:43.622205+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19502348,"num_objects":2381,"num_object_clones":0,"num_object_copies":9524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10358,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":133}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10094","reported_seq":10203,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.745863+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:43.745863+0000","last_peered":"2024-09-16T04:49:43.745863+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:43.745863+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18169856,"num_objects":2218,"num_object_clones":0,"num_object_copies":8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10094,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":40}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10343","reported_seq":10441,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.283956+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:48.283956+0000","last_peered":"2024-09-16T04:49:48.283956+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:48.283956+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18486540,"num_objects":2257,"num_object_clones":0,"num_object_copies":9028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10341,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":44}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10394","reported_seq":14956,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:45.837127+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:45.837127+0000","last_peered":"2024-09-16T04:49:45.837127+0000","last_clean":"2024-09-16T04:49:45.837127+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:45.837127+0000","last_undegraded":"2024-09-16T04:49:45.837127+0000","last_fullsized":"2024-09-16T04:49:45.837127+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19584268,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10392,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10424","reported_seq":15182,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:43.726742+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:43.726742+0000","last_peered":"2024-09-16T04:49:43.726742+0000","last_clean":"2024-09-16T04:49:43.726742+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:43.726742+0000","last_undegraded":"2024-09-16T04:49:43.726742+0000","last_fullsized":"2024-09-16T04:49:43.726742+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:48.134884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19098136,"num_objects":2332,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10421,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10392","reported_seq":10499,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:47.396860+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:47.396860+0000","last_peered":"2024-09-16T04:49:47.396860+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:47.396860+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379468,"num_objects":2366,"num_object_clones":0,"num_object_copies":9464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10390,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":177}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10203","reported_seq":10334,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.680842+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:43.680842+0000","last_peered":"2024-09-16T04:49:43.680842+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:43.680842+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18739090,"num_objects":2288,"num_object_clones":0,"num_object_copies":9152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10200,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":133}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10038","reported_seq":10130,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.851187+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:43.851187+0000","last_peered":"2024-09-16T04:49:43.851187+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:43.851187+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18782854,"num_objects":2293,"num_object_clones":0,"num_object_copies":9172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10247","reported_seq":10336,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:47.539989+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:47.539989+0000","last_peered":"2024-09-16T04:49:47.539989+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:47.539989+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18725510,"num_objects":2286,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10246,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10422","reported_seq":10555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.771774+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:43.771774+0000","last_peered":"2024-09-16T04:49:43.771774+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:43.771774+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330316,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10420,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":162}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10119","reported_seq":10223,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:47.503662+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:47.503662+0000","last_peered":"2024-09-16T04:49:47.503662+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:47.503662+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19084556,"num_objects":2330,"num_object_clones":0,"num_object_copies":9320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":207}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10139","reported_seq":10233,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.421194+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:48.421194+0000","last_peered":"2024-09-16T04:49:48.421194+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:48.421194+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17757452,"num_objects":2168,"num_object_clones":0,"num_object_copies":8672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10137,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-60}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":302974656,"num_objects":36990,"num_object_clones":0,"num_object_copies":147960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36990,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164497,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6203,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":604909568,"data_stored":604909568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160824,"ondisk_log_size":160824,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745744,"kb_used_data":150392,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93626096,"statfs":{"total":96636764160,"available":95873122304,"internally_reserved":0,"allocated":154001408,"data_stored":153812206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609611471},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755856,"kb_used_data":147108,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93615984,"statfs":{"total":96636764160,"available":95862767616,"internally_reserved":0,"allocated":150638592,"data_stored":150449390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,0,5,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618588,"kb_used_data":149564,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93753252,"statfs":{"total":96636764160,"available":96003330048,"internally_reserved":0,"allocated":153153536,"data_stored":152964334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,0,0,2,2,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247923,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746332,"kb_used_data":146892,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93625508,"statfs":{"total":96636764160,"available":95872520192,"internally_reserved":0,"allocated":150417408,"data_stored":150228206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":613805774},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,3,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":149475328,"data_stored":149475328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152104960,"data_stored":152104960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":150200320,"data_stored":150200320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":153128960,"data_stored":153128960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:49.671 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:50.297 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:50.298 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:50.339 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-09-16T04:49:48.727646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":302974656,"num_objects":36990,"num_object_clones":0,"num_object_copies":147960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36990,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164497,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6203,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160824,"ondisk_log_size":160824,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2866520,"kb_used_data":593956,"kb_used_omap":14,"kb_used_meta":2272433,"kb_avail":374620840,"statfs":{"total":386547056640,"available":383611740160,"internally_reserved":0,"allocated":608210944,"data_stored":607454136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2326971953},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,9,4,0,4,13,16,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20160512,"num_objects":-2461,"num_object_clones":0,"num_object_copies":-9844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2461,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003515"},"pg_stats":[{"pgid":"1.f","version":"35'10267","reported_seq":10383,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.814222+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:43.814222+0000","last_peered":"2024-09-16T04:49:43.814222+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:43.814222+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19554304,"num_objects":2387,"num_object_clones":0,"num_object_copies":9548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10267,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10366","reported_seq":10455,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.413206+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:48.413206+0000","last_peered":"2024-09-16T04:49:48.413206+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:48.413206+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18904332,"num_objects":2308,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10364,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10343","reported_seq":10440,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:47.538971+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:47.538971+0000","last_peered":"2024-09-16T04:49:47.538971+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:47.538971+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19061406,"num_objects":2328,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10342,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10373","reported_seq":13123,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.494835+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:48.494835+0000","last_peered":"2024-09-16T04:49:48.494835+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:48.494835+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18814220,"num_objects":2297,"num_object_clones":0,"num_object_copies":9188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10371,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1340,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10360","reported_seq":10493,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.622205+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:43.622205+0000","last_peered":"2024-09-16T04:49:43.622205+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:43.622205+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19502348,"num_objects":2381,"num_object_clones":0,"num_object_copies":9524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10358,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":133}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10094","reported_seq":10203,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.745863+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:43.745863+0000","last_peered":"2024-09-16T04:49:43.745863+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:43.745863+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18169856,"num_objects":2218,"num_object_clones":0,"num_object_copies":8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10094,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":40}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10343","reported_seq":10441,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.283956+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:48.283956+0000","last_peered":"2024-09-16T04:49:48.283956+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:48.283956+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18486540,"num_objects":2257,"num_object_clones":0,"num_object_copies":9028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10341,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":44}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10394","reported_seq":14956,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:45.837127+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:45.837127+0000","last_peered":"2024-09-16T04:49:45.837127+0000","last_clean":"2024-09-16T04:49:45.837127+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:45.837127+0000","last_undegraded":"2024-09-16T04:49:45.837127+0000","last_fullsized":"2024-09-16T04:49:45.837127+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19584268,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10392,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10424","reported_seq":15182,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:43.726742+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:43.726742+0000","last_peered":"2024-09-16T04:49:43.726742+0000","last_clean":"2024-09-16T04:49:43.726742+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:43.726742+0000","last_undegraded":"2024-09-16T04:49:43.726742+0000","last_fullsized":"2024-09-16T04:49:43.726742+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:48.134884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19098136,"num_objects":2332,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10421,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10392","reported_seq":10499,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:47.396860+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:47.396860+0000","last_peered":"2024-09-16T04:49:47.396860+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:47.396860+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379468,"num_objects":2366,"num_object_clones":0,"num_object_copies":9464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10390,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":177}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10203","reported_seq":10334,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.680842+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:43.680842+0000","last_peered":"2024-09-16T04:49:43.680842+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:43.680842+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18739090,"num_objects":2288,"num_object_clones":0,"num_object_copies":9152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10200,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":133}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10038","reported_seq":10130,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.851187+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:43.851187+0000","last_peered":"2024-09-16T04:49:43.851187+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:43.851187+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18782854,"num_objects":2293,"num_object_clones":0,"num_object_copies":9172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10247","reported_seq":10336,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:47.539989+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:47.539989+0000","last_peered":"2024-09-16T04:49:47.539989+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:47.539989+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18725510,"num_objects":2286,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10246,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10422","reported_seq":10555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:43.771774+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:43.771774+0000","last_peered":"2024-09-16T04:49:43.771774+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:43.771774+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330316,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10420,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":162}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10119","reported_seq":10223,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:47.503662+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:47.503662+0000","last_peered":"2024-09-16T04:49:47.503662+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:47.503662+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19084556,"num_objects":2330,"num_object_clones":0,"num_object_copies":9320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":207}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10139","reported_seq":10233,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.421194+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:48.421194+0000","last_peered":"2024-09-16T04:49:48.421194+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:48.421194+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17757452,"num_objects":2168,"num_object_clones":0,"num_object_copies":8672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10137,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-60}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":302974656,"num_objects":36990,"num_object_clones":0,"num_object_copies":147960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36990,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164497,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6203,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":604909568,"data_stored":604909568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160824,"ondisk_log_size":160824,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745744,"kb_used_data":150392,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93626096,"statfs":{"total":96636764160,"available":95873122304,"internally_reserved":0,"allocated":154001408,"data_stored":153812206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609611471},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755856,"kb_used_data":147108,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93615984,"statfs":{"total":96636764160,"available":95862767616,"internally_reserved":0,"allocated":150638592,"data_stored":150449390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,0,5,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618588,"kb_used_data":149564,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93753252,"statfs":{"total":96636764160,"available":96003330048,"internally_reserved":0,"allocated":153153536,"data_stored":152964334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,0,0,2,2,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247923,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746332,"kb_used_data":146892,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93625508,"statfs":{"total":96636764160,"available":95872520192,"internally_reserved":0,"allocated":150417408,"data_stored":150228206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":613805774},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,3,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":149475328,"data_stored":149475328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152104960,"data_stored":152104960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":150200320,"data_stored":150200320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":153128960,"data_stored":153128960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:50.340 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:49:50.340 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10267", 'reported_seq': 10383, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:43.814222+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:49:43.814222+0000', 'last_peered': '2024-09-16T04:49:43.814222+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:49:43.814222+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19554304, 'num_objects': 2387, 'num_object_clones': 0, 'num_object_copies': 9548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2387, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10267, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 201}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:50.340 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:49:50.340 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10366", 'reported_seq': 10455, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:48.413206+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:49:48.413206+0000', 'last_peered': '2024-09-16T04:49:48.413206+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:49:48.413206+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18904332, 'num_objects': 2308, 'num_object_clones': 0, 'num_object_copies': 9232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10364, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 105}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:50.340 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:49:50.340 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10343", 'reported_seq': 10440, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:47.538971+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:49:47.538971+0000', 'last_peered': '2024-09-16T04:49:47.538971+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:49:47.538971+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19061406, 'num_objects': 2328, 'num_object_clones': 0, 'num_object_copies': 9312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2328, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10342, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 109}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:50.340 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:49:50.340 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10373", 'reported_seq': 13123, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:48.494835+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:49:48.494835+0000', 'last_peered': '2024-09-16T04:49:48.494835+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:49:48.494835+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18814220, 'num_objects': 2297, 'num_object_clones': 0, 'num_object_copies': 9188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 893, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2297, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10371, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1340, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1404}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:50.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:49:50.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10360", 'reported_seq': 10493, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:43.622205+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:49:43.622205+0000', 'last_peered': '2024-09-16T04:49:43.622205+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:49:43.622205+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19502348, 'num_objects': 2381, 'num_object_clones': 0, 'num_object_copies': 9524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10358, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 133}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:50.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:49:50.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10094", 'reported_seq': 10203, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:43.745863+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:49:43.745863+0000', 'last_peered': '2024-09-16T04:49:43.745863+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:49:43.745863+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18169856, 'num_objects': 2218, 'num_object_clones': 0, 'num_object_copies': 8872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2218, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10094, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 40}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:50.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:49:50.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10343", 'reported_seq': 10441, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:48.283956+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:49:48.283956+0000', 'last_peered': '2024-09-16T04:49:48.283956+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:49:48.283956+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18486540, 'num_objects': 2257, 'num_object_clones': 0, 'num_object_copies': 9028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2257, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10341, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 44}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:50.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:49:50.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10392", 'reported_seq': 10499, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:47.396860+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:49:47.396860+0000', 'last_peered': '2024-09-16T04:49:47.396860+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:49:47.396860+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19379468, 'num_objects': 2366, 'num_object_clones': 0, 'num_object_copies': 9464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10390, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 177}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:50.341 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:49:50.341 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10203", 'reported_seq': 10334, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:43.680842+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:49:43.680842+0000', 'last_peered': '2024-09-16T04:49:43.680842+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:49:43.680842+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18739090, 'num_objects': 2288, 'num_object_clones': 0, 'num_object_copies': 9152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10200, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 133}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:50.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:49:50.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10038", 'reported_seq': 10130, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:43.851187+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:49:43.851187+0000', 'last_peered': '2024-09-16T04:49:43.851187+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:49:43.851187+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18782854, 'num_objects': 2293, 'num_object_clones': 0, 'num_object_copies': 9172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10037, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 188}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:50.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:49:50.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10247", 'reported_seq': 10336, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:47.539989+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:49:47.539989+0000', 'last_peered': '2024-09-16T04:49:47.539989+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:49:47.539989+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18725510, 'num_objects': 2286, 'num_object_clones': 0, 'num_object_copies': 9144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2286, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10246, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 65}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:50.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:49:50.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10422", 'reported_seq': 10555, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:43.771774+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:49:43.771774+0000', 'last_peered': '2024-09-16T04:49:43.771774+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:49:43.771774+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19330316, 'num_objects': 2360, 'num_object_clones': 0, 'num_object_copies': 9440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10420, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 162}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:50.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:49:50.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10119", 'reported_seq': 10223, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:47.503662+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:49:47.503662+0000', 'last_peered': '2024-09-16T04:49:47.503662+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:49:47.503662+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19084556, 'num_objects': 2330, 'num_object_clones': 0, 'num_object_copies': 9320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10117, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 207}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:50.342 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:49:50.342 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10139", 'reported_seq': 10233, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:48.421194+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:49:48.421194+0000', 'last_peered': '2024-09-16T04:49:48.421194+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:49:48.421194+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17757452, 'num_objects': 2168, 'num_object_clones': 0, 'num_object_copies': 8672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10137, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -60}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:50.343 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:49:51.060 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:51.060 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29379/146208 objects degraded (20.094%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":603,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":36552,"data_bytes":299386560,"bytes_used":2935586816,"bytes_avail":383611469824,"bytes_total":386547056640,"degraded_objects":29379,"degraded_total":146208,"degraded_ratio":0.20093975705843733,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":199},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:49:51.078 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:49:51.078 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:51.696 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:51.697 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:51.720 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-16T04:49:50.728197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":299386560,"num_objects":36552,"num_object_clones":0,"num_object_copies":146208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36552,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164935,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6203,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160962,"ondisk_log_size":160962,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2866784,"kb_used_data":590068,"kb_used_omap":14,"kb_used_meta":2276593,"kb_avail":374620576,"statfs":{"total":386547056640,"available":383611469824,"internally_reserved":0,"allocated":604229632,"data_stored":603472824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2331231793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,9,4,1,4,11,11,16,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19611648,"num_objects":-2394,"num_object_clones":0,"num_object_copies":-9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2394,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003605"},"pg_stats":[{"pgid":"1.f","version":"35'10345","reported_seq":10459,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.556194+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:48.556194+0000","last_peered":"2024-09-16T04:49:48.556194+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:48.556194+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18915328,"num_objects":2309,"num_object_clones":0,"num_object_copies":9236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10345,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10366","reported_seq":10455,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.413206+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:48.413206+0000","last_peered":"2024-09-16T04:49:48.413206+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:48.413206+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18904332,"num_objects":2308,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10364,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10343","reported_seq":10440,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:47.538971+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:47.538971+0000","last_peered":"2024-09-16T04:49:47.538971+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:47.538971+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19061406,"num_objects":2328,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10342,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10373","reported_seq":13123,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.494835+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:48.494835+0000","last_peered":"2024-09-16T04:49:48.494835+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:48.494835+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18814220,"num_objects":2297,"num_object_clones":0,"num_object_copies":9188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10371,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1340,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10424","reported_seq":10556,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.882365+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:48.882365+0000","last_peered":"2024-09-16T04:49:48.882365+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:48.882365+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18978060,"num_objects":2317,"num_object_clones":0,"num_object_copies":9268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10422,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":69}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10146","reported_seq":10257,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.517943+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:48.517943+0000","last_peered":"2024-09-16T04:49:48.517943+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:48.517943+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17743872,"num_objects":2166,"num_object_clones":0,"num_object_copies":8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10146,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-12}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10343","reported_seq":10441,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.283956+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:48.283956+0000","last_peered":"2024-09-16T04:49:48.283956+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:48.283956+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18486540,"num_objects":2257,"num_object_clones":0,"num_object_copies":9028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10341,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":44}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10394","reported_seq":14956,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:45.837127+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:45.837127+0000","last_peered":"2024-09-16T04:49:45.837127+0000","last_clean":"2024-09-16T04:49:45.837127+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:45.837127+0000","last_undegraded":"2024-09-16T04:49:45.837127+0000","last_fullsized":"2024-09-16T04:49:45.837127+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19584268,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10392,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10494","reported_seq":15252,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:48.869255+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:48.869255+0000","last_peered":"2024-09-16T04:49:48.869255+0000","last_clean":"2024-09-16T04:49:48.869255+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:48.869255+0000","last_undegraded":"2024-09-16T04:49:48.869255+0000","last_fullsized":"2024-09-16T04:49:48.869255+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:53.226377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18524696,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10491,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10392","reported_seq":10499,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:47.396860+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:47.396860+0000","last_peered":"2024-09-16T04:49:47.396860+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:47.396860+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379468,"num_objects":2366,"num_object_clones":0,"num_object_copies":9464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10390,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":177}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10263","reported_seq":10396,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.865151+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:48.865151+0000","last_peered":"2024-09-16T04:49:48.865151+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:48.865151+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18247570,"num_objects":2228,"num_object_clones":0,"num_object_copies":8912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10260,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":73}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10099","reported_seq":10192,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.764542+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:48.764542+0000","last_peered":"2024-09-16T04:49:48.764542+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:48.764542+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18283142,"num_objects":2232,"num_object_clones":0,"num_object_copies":8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10098,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":127}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10247","reported_seq":10336,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:47.539989+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:47.539989+0000","last_peered":"2024-09-16T04:49:47.539989+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:47.539989+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18725510,"num_objects":2286,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10246,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10475","reported_seq":10608,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.456833+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:48.456833+0000","last_peered":"2024-09-16T04:49:48.456833+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:48.456833+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18896140,"num_objects":2307,"num_object_clones":0,"num_object_copies":9228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10473,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":109}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10119","reported_seq":10223,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:47.503662+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:47.503662+0000","last_peered":"2024-09-16T04:49:47.503662+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:47.503662+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19084556,"num_objects":2330,"num_object_clones":0,"num_object_copies":9320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":207}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10139","reported_seq":10233,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:48.421194+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:48.421194+0000","last_peered":"2024-09-16T04:49:48.421194+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:48.421194+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17757452,"num_objects":2168,"num_object_clones":0,"num_object_copies":8672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10137,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-60}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":299386560,"num_objects":36552,"num_object_clones":0,"num_object_copies":146208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36552,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164935,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6203,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":600911872,"data_stored":600911872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160962,"ondisk_log_size":160962,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746008,"kb_used_data":146504,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93625832,"statfs":{"total":96636764160,"available":95872851968,"internally_reserved":0,"allocated":150020096,"data_stored":149830894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613871311},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,1,0,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755856,"kb_used_data":147108,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93615984,"statfs":{"total":96636764160,"available":95862767616,"internally_reserved":0,"allocated":150638592,"data_stored":150449390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,0,5,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618588,"kb_used_data":149564,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93753252,"statfs":{"total":96636764160,"available":96003330048,"internally_reserved":0,"allocated":153153536,"data_stored":152964334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,0,0,2,2,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247923,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746332,"kb_used_data":146892,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93625508,"statfs":{"total":96636764160,"available":95872520192,"internally_reserved":0,"allocated":150417408,"data_stored":150228206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":613805774},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,3,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":149475328,"data_stored":149475328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152104960,"data_stored":152104960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":150200320,"data_stored":150200320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":149131264,"data_stored":149131264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:54.722 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:55.348 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:55.349 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:55.367 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-09-16T04:49:54.729338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":291366592,"num_objects":35573,"num_object_clones":0,"num_object_copies":142292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35573,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165914,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6266,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2846492,"kb_used_data":573976,"kb_used_omap":14,"kb_used_meta":2272433,"kb_avail":374640868,"statfs":{"total":386547056640,"available":383632248832,"internally_reserved":0,"allocated":587751424,"data_stored":586994616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2326971953},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,10,1,7,9,17,12,13,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21536768,"num_objects":-2629,"num_object_clones":0,"num_object_copies":-10516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2629,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003589"},"pg_stats":[{"pgid":"1.f","version":"35'10388","reported_seq":10505,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.800367+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:53.800367+0000","last_peered":"2024-09-16T04:49:53.800367+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:53.800367+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18563072,"num_objects":2266,"num_object_clones":0,"num_object_copies":9064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10388,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":80}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10424","reported_seq":10512,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.613516+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:53.613516+0000","last_peered":"2024-09-16T04:49:53.613516+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:53.613516+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18429196,"num_objects":2250,"num_object_clones":0,"num_object_copies":9000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10422,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":47}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10399","reported_seq":10498,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:52.431578+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:52.431578+0000","last_peered":"2024-09-16T04:49:52.431578+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:52.431578+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18602654,"num_objects":2272,"num_object_clones":0,"num_object_copies":9088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2272,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10398,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":53}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10419","reported_seq":13294,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.609506+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:53.609506+0000","last_peered":"2024-09-16T04:49:53.609506+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:53.609506+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18437388,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":830,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10417,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1403,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10489","reported_seq":10622,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.866506+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:53.866506+0000","last_peered":"2024-09-16T04:49:53.866506+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:53.866506+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18445580,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10487,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10210","reported_seq":10321,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.832105+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:53.832105+0000","last_peered":"2024-09-16T04:49:53.832105+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:53.832105+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17219584,"num_objects":2102,"num_object_clones":0,"num_object_copies":8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10210,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-76}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10415","reported_seq":10513,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.429364+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:53.429364+0000","last_peered":"2024-09-16T04:49:53.429364+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:53.429364+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17896716,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-28}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10470","reported_seq":15032,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:50.920391+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:50.920391+0000","last_peered":"2024-09-16T04:49:50.920391+0000","last_clean":"2024-09-16T04:49:50.920391+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:50.920391+0000","last_undegraded":"2024-09-16T04:49:50.920391+0000","last_fullsized":"2024-09-16T04:49:50.920391+0000","mapping_epoch":32,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:51.085214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18961676,"num_objects":2315,"num_object_clones":0,"num_object_copies":9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10468,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10549","reported_seq":15306,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:53.876973+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:53.876973+0000","last_peered":"2024-09-16T04:49:53.876973+0000","last_clean":"2024-09-16T04:49:53.876973+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:53.876973+0000","last_undegraded":"2024-09-16T04:49:53.876973+0000","last_fullsized":"2024-09-16T04:49:53.876973+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:58.226664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18074136,"num_objects":2207,"num_object_clones":0,"num_object_copies":8828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10546,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10461","reported_seq":10568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:52.660256+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:52.660256+0000","last_peered":"2024-09-16T04:49:52.660256+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:52.660256+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18814220,"num_objects":2297,"num_object_clones":0,"num_object_copies":9188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10459,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10314","reported_seq":10447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.783202+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:53.783202+0000","last_peered":"2024-09-16T04:49:53.783202+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:53.783202+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17829778,"num_objects":2177,"num_object_clones":0,"num_object_copies":8708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10311,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":22}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10190","reported_seq":10282,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:54.012233+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:54.012233+0000","last_peered":"2024-09-16T04:49:54.012233+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:54.012233+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17537670,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10189,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":36}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10309","reported_seq":10398,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:52.496592+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:52.496592+0000","last_peered":"2024-09-16T04:49:52.496592+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:52.496592+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18217606,"num_objects":2224,"num_object_clones":0,"num_object_copies":8896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10308,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10527","reported_seq":10660,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.888185+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:53.888185+0000","last_peered":"2024-09-16T04:49:53.888185+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:53.888185+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18470156,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10525,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":57}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10178","reported_seq":10282,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:52.519685+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:52.519685+0000","last_peered":"2024-09-16T04:49:52.519685+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:52.519685+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18601228,"num_objects":2271,"num_object_clones":0,"num_object_copies":9084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10176,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":148}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10199","reported_seq":10294,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.141794+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:53.141794+0000","last_peered":"2024-09-16T04:49:53.141794+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:53.141794+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17265932,"num_objects":2108,"num_object_clones":0,"num_object_copies":8432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10197,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-120}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":291366592,"num_objects":35573,"num_object_clones":0,"num_object_copies":142292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35573,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165914,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6266,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":584933376,"data_stored":584933376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742180,"kb_used_data":142672,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93629660,"statfs":{"total":96636764160,"available":95876771840,"internally_reserved":0,"allocated":146096128,"data_stored":145906926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613871311},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2,1,7,4,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756156,"kb_used_data":143276,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93615684,"statfs":{"total":96636764160,"available":95862460416,"internally_reserved":0,"allocated":146714624,"data_stored":146525422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618124,"kb_used_data":144956,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93753716,"statfs":{"total":96636764160,"available":96003805184,"internally_reserved":0,"allocated":148434944,"data_stored":148245742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,0,0,2,2,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730032,"kb_used_data":143072,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93641808,"statfs":{"total":96636764160,"available":95889211392,"internally_reserved":0,"allocated":146505728,"data_stored":146316526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":601026254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":145473536,"data_stored":145473536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148111360,"data_stored":148111360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":146210816,"data_stored":146210816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":145137664,"data_stored":145137664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:55.368 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:55.970 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:55.971 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:55.994 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-09-16T04:49:54.729338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":291366592,"num_objects":35573,"num_object_clones":0,"num_object_copies":142292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35573,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165914,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6266,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2846492,"kb_used_data":573976,"kb_used_omap":14,"kb_used_meta":2272433,"kb_avail":374640868,"statfs":{"total":386547056640,"available":383632248832,"internally_reserved":0,"allocated":587751424,"data_stored":586994616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2326971953},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,10,1,7,9,17,12,13,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21536768,"num_objects":-2629,"num_object_clones":0,"num_object_copies":-10516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2629,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003589"},"pg_stats":[{"pgid":"1.f","version":"35'10388","reported_seq":10505,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.800367+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:53.800367+0000","last_peered":"2024-09-16T04:49:53.800367+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:53.800367+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18563072,"num_objects":2266,"num_object_clones":0,"num_object_copies":9064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10388,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":80}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10424","reported_seq":10512,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.613516+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:53.613516+0000","last_peered":"2024-09-16T04:49:53.613516+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:53.613516+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18429196,"num_objects":2250,"num_object_clones":0,"num_object_copies":9000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10422,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":47}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10399","reported_seq":10498,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:52.431578+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:52.431578+0000","last_peered":"2024-09-16T04:49:52.431578+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:52.431578+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18602654,"num_objects":2272,"num_object_clones":0,"num_object_copies":9088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2272,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10398,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":53}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10419","reported_seq":13294,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.609506+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:53.609506+0000","last_peered":"2024-09-16T04:49:53.609506+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:53.609506+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18437388,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":830,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10417,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1403,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10489","reported_seq":10622,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.866506+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:53.866506+0000","last_peered":"2024-09-16T04:49:53.866506+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:53.866506+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18445580,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10487,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10210","reported_seq":10321,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.832105+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:53.832105+0000","last_peered":"2024-09-16T04:49:53.832105+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:53.832105+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17219584,"num_objects":2102,"num_object_clones":0,"num_object_copies":8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10210,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-76}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10415","reported_seq":10513,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.429364+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:53.429364+0000","last_peered":"2024-09-16T04:49:53.429364+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:53.429364+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17896716,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-28}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10470","reported_seq":15032,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:50.920391+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:50.920391+0000","last_peered":"2024-09-16T04:49:50.920391+0000","last_clean":"2024-09-16T04:49:50.920391+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:50.920391+0000","last_undegraded":"2024-09-16T04:49:50.920391+0000","last_fullsized":"2024-09-16T04:49:50.920391+0000","mapping_epoch":32,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:51.085214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18961676,"num_objects":2315,"num_object_clones":0,"num_object_copies":9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10468,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10549","reported_seq":15306,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:53.876973+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:53.876973+0000","last_peered":"2024-09-16T04:49:53.876973+0000","last_clean":"2024-09-16T04:49:53.876973+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:53.876973+0000","last_undegraded":"2024-09-16T04:49:53.876973+0000","last_fullsized":"2024-09-16T04:49:53.876973+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:58.226664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18074136,"num_objects":2207,"num_object_clones":0,"num_object_copies":8828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10546,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10461","reported_seq":10568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:52.660256+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:52.660256+0000","last_peered":"2024-09-16T04:49:52.660256+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:52.660256+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18814220,"num_objects":2297,"num_object_clones":0,"num_object_copies":9188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10459,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10314","reported_seq":10447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.783202+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:53.783202+0000","last_peered":"2024-09-16T04:49:53.783202+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:53.783202+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17829778,"num_objects":2177,"num_object_clones":0,"num_object_copies":8708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10311,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":22}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10190","reported_seq":10282,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:54.012233+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:54.012233+0000","last_peered":"2024-09-16T04:49:54.012233+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:54.012233+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17537670,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10189,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":36}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10309","reported_seq":10398,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:52.496592+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:52.496592+0000","last_peered":"2024-09-16T04:49:52.496592+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:52.496592+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18217606,"num_objects":2224,"num_object_clones":0,"num_object_copies":8896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10308,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10527","reported_seq":10660,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.888185+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:53.888185+0000","last_peered":"2024-09-16T04:49:53.888185+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:53.888185+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18470156,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10525,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":57}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10178","reported_seq":10282,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:52.519685+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:52.519685+0000","last_peered":"2024-09-16T04:49:52.519685+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:52.519685+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18601228,"num_objects":2271,"num_object_clones":0,"num_object_copies":9084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10176,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":148}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10199","reported_seq":10294,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.141794+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:53.141794+0000","last_peered":"2024-09-16T04:49:53.141794+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:53.141794+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17265932,"num_objects":2108,"num_object_clones":0,"num_object_copies":8432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10197,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-120}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":291366592,"num_objects":35573,"num_object_clones":0,"num_object_copies":142292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35573,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165914,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6266,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":584933376,"data_stored":584933376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742180,"kb_used_data":142672,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93629660,"statfs":{"total":96636764160,"available":95876771840,"internally_reserved":0,"allocated":146096128,"data_stored":145906926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613871311},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2,1,7,4,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756156,"kb_used_data":143276,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93615684,"statfs":{"total":96636764160,"available":95862460416,"internally_reserved":0,"allocated":146714624,"data_stored":146525422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618124,"kb_used_data":144956,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93753716,"statfs":{"total":96636764160,"available":96003805184,"internally_reserved":0,"allocated":148434944,"data_stored":148245742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,0,0,2,2,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730032,"kb_used_data":143072,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93641808,"statfs":{"total":96636764160,"available":95889211392,"internally_reserved":0,"allocated":146505728,"data_stored":146316526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":601026254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":145473536,"data_stored":145473536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148111360,"data_stored":148111360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":146210816,"data_stored":146210816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":145137664,"data_stored":145137664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:49:55.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:49:55.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10388", 'reported_seq': 10505, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:53.800367+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:49:53.800367+0000', 'last_peered': '2024-09-16T04:49:53.800367+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:49:53.800367+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18563072, 'num_objects': 2266, 'num_object_clones': 0, 'num_object_copies': 9064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2266, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10388, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 80}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:55.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:49:55.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10424", 'reported_seq': 10512, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:53.613516+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:49:53.613516+0000', 'last_peered': '2024-09-16T04:49:53.613516+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:49:53.613516+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18429196, 'num_objects': 2250, 'num_object_clones': 0, 'num_object_copies': 9000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10422, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 47}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:55.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:49:55.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10399", 'reported_seq': 10498, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:52.431578+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:49:52.431578+0000', 'last_peered': '2024-09-16T04:49:52.431578+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:49:52.431578+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18602654, 'num_objects': 2272, 'num_object_clones': 0, 'num_object_copies': 9088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2272, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10398, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 53}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:55.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:49:55.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10419", 'reported_seq': 13294, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:53.609506+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:49:53.609506+0000', 'last_peered': '2024-09-16T04:49:53.609506+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:49:53.609506+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18437388, 'num_objects': 2251, 'num_object_clones': 0, 'num_object_copies': 9004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 830, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10417, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1403, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1421}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:55.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:49:55.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10489", 'reported_seq': 10622, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:53.866506+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:49:53.866506+0000', 'last_peered': '2024-09-16T04:49:53.866506+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:49:53.866506+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18445580, 'num_objects': 2252, 'num_object_clones': 0, 'num_object_copies': 9008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10487, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:55.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:49:55.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10210", 'reported_seq': 10321, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:53.832105+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:49:53.832105+0000', 'last_peered': '2024-09-16T04:49:53.832105+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:49:53.832105+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17219584, 'num_objects': 2102, 'num_object_clones': 0, 'num_object_copies': 8408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2102, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10210, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -76}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:55.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:49:55.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10415", 'reported_seq': 10513, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:53.429364+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:49:53.429364+0000', 'last_peered': '2024-09-16T04:49:53.429364+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:49:53.429364+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17896716, 'num_objects': 2185, 'num_object_clones': 0, 'num_object_copies': 8740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10413, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -28}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:55.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:49:55.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10461", 'reported_seq': 10568, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:52.660256+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:49:52.660256+0000', 'last_peered': '2024-09-16T04:49:52.660256+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:49:52.660256+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18814220, 'num_objects': 2297, 'num_object_clones': 0, 'num_object_copies': 9188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2297, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10459, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 108}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:55.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:49:55.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10314", 'reported_seq': 10447, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:53.783202+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:49:53.783202+0000', 'last_peered': '2024-09-16T04:49:53.783202+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:49:53.783202+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17829778, 'num_objects': 2177, 'num_object_clones': 0, 'num_object_copies': 8708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10311, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 22}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:55.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:49:55.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10190", 'reported_seq': 10282, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:54.012233+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:49:54.012233+0000', 'last_peered': '2024-09-16T04:49:54.012233+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:49:54.012233+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17537670, 'num_objects': 2141, 'num_object_clones': 0, 'num_object_copies': 8564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10189, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 36}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:55.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:49:55.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10309", 'reported_seq': 10398, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:52.496592+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:49:52.496592+0000', 'last_peered': '2024-09-16T04:49:52.496592+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:49:52.496592+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18217606, 'num_objects': 2224, 'num_object_clones': 0, 'num_object_copies': 8896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2224, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10308, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:55.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:49:55.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10527", 'reported_seq': 10660, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:53.888185+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:49:53.888185+0000', 'last_peered': '2024-09-16T04:49:53.888185+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:49:53.888185+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18470156, 'num_objects': 2255, 'num_object_clones': 0, 'num_object_copies': 9020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10525, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 57}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:49:55.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:49:55.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10178", 'reported_seq': 10282, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:52.519685+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:49:52.519685+0000', 'last_peered': '2024-09-16T04:49:52.519685+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:49:52.519685+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18601228, 'num_objects': 2271, 'num_object_clones': 0, 'num_object_copies': 9084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2271, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10176, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 148}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:49:55.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:49:55.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10199", 'reported_seq': 10294, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:53.141794+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:49:53.141794+0000', 'last_peered': '2024-09-16T04:49:53.141794+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:49:53.141794+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17265932, 'num_objects': 2108, 'num_object_clones': 0, 'num_object_copies': 8432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10197, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -120}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:49:55.998 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:49:56.682 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:56.682 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29316/142292 objects degraded (20.603%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":609,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":35573,"data_bytes":291366592,"bytes_used":2914807808,"bytes_avail":383632248832,"bytes_total":386547056640,"degraded_objects":29316,"degraded_total":142292,"degraded_ratio":0.20602704298203695,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":219},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:49:56.715 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:49:56.716 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:49:57.305 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:49:57.306 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:49:57.328 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-09-16T04:49:56.729998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":290875072,"num_objects":35513,"num_object_clones":0,"num_object_copies":142052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165974,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6266,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160801,"ondisk_log_size":160801,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2846652,"kb_used_data":569976,"kb_used_omap":14,"kb_used_meta":2276593,"kb_avail":374640708,"statfs":{"total":386547056640,"available":383632084992,"internally_reserved":0,"allocated":583655424,"data_stored":582898616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2331231793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,7,9,18,18,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16334848,"num_objects":-1994,"num_object_clones":0,"num_object_copies":-7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1994,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003577"},"pg_stats":[{"pgid":"1.f","version":"35'10388","reported_seq":10505,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.800367+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:53.800367+0000","last_peered":"2024-09-16T04:49:53.800367+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:53.800367+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18563072,"num_objects":2266,"num_object_clones":0,"num_object_copies":9064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10388,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":80}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10424","reported_seq":10512,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.613516+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:53.613516+0000","last_peered":"2024-09-16T04:49:53.613516+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:53.613516+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18429196,"num_objects":2250,"num_object_clones":0,"num_object_copies":9000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10422,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":47}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10399","reported_seq":10498,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:52.431578+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:52.431578+0000","last_peered":"2024-09-16T04:49:52.431578+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:52.431578+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18602654,"num_objects":2272,"num_object_clones":0,"num_object_copies":9088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2272,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10398,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":53}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10419","reported_seq":13294,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.609506+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:53.609506+0000","last_peered":"2024-09-16T04:49:53.609506+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:53.609506+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18437388,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":830,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10417,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1403,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10489","reported_seq":10622,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.866506+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:53.866506+0000","last_peered":"2024-09-16T04:49:53.866506+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:53.866506+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18445580,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10487,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10210","reported_seq":10321,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.832105+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:53.832105+0000","last_peered":"2024-09-16T04:49:53.832105+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:53.832105+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17219584,"num_objects":2102,"num_object_clones":0,"num_object_copies":8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10210,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-76}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10415","reported_seq":10513,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.429364+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:53.429364+0000","last_peered":"2024-09-16T04:49:53.429364+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:53.429364+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17896716,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-28}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10530","reported_seq":15092,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:56.065223+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:56.065223+0000","last_peered":"2024-09-16T04:49:56.065223+0000","last_clean":"2024-09-16T04:49:56.065223+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:56.065223+0000","last_undegraded":"2024-09-16T04:49:56.065223+0000","last_fullsized":"2024-09-16T04:49:56.065223+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:56.996738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18470156,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10528,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10549","reported_seq":15306,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:53.876973+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:53.876973+0000","last_peered":"2024-09-16T04:49:53.876973+0000","last_clean":"2024-09-16T04:49:53.876973+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:53.876973+0000","last_undegraded":"2024-09-16T04:49:53.876973+0000","last_fullsized":"2024-09-16T04:49:53.876973+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:58.226664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18074136,"num_objects":2207,"num_object_clones":0,"num_object_copies":8828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10546,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10461","reported_seq":10568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:52.660256+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:52.660256+0000","last_peered":"2024-09-16T04:49:52.660256+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:52.660256+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18814220,"num_objects":2297,"num_object_clones":0,"num_object_copies":9188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10459,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10314","reported_seq":10447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.783202+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:53.783202+0000","last_peered":"2024-09-16T04:49:53.783202+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:53.783202+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17829778,"num_objects":2177,"num_object_clones":0,"num_object_copies":8708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10311,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":22}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10190","reported_seq":10282,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:54.012233+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:54.012233+0000","last_peered":"2024-09-16T04:49:54.012233+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:54.012233+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17537670,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10189,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":36}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10309","reported_seq":10398,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:52.496592+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:52.496592+0000","last_peered":"2024-09-16T04:49:52.496592+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:52.496592+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18217606,"num_objects":2224,"num_object_clones":0,"num_object_copies":8896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10308,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10527","reported_seq":10660,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.888185+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:53.888185+0000","last_peered":"2024-09-16T04:49:53.888185+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:53.888185+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18470156,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10525,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":57}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10178","reported_seq":10282,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:52.519685+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:52.519685+0000","last_peered":"2024-09-16T04:49:52.519685+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:52.519685+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18601228,"num_objects":2271,"num_object_clones":0,"num_object_copies":9084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10176,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":148}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10199","reported_seq":10294,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:53.141794+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:53.141794+0000","last_peered":"2024-09-16T04:49:53.141794+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:53.141794+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17265932,"num_objects":2108,"num_object_clones":0,"num_object_copies":8432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10197,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-120}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":290875072,"num_objects":35513,"num_object_clones":0,"num_object_copies":142052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165974,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6266,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":580820992,"data_stored":580820992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160801,"ondisk_log_size":160801,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742180,"kb_used_data":142672,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93629660,"statfs":{"total":96636764160,"available":95876771840,"internally_reserved":0,"allocated":146096128,"data_stored":145906926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613871311},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2,1,7,4,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756156,"kb_used_data":143276,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93615684,"statfs":{"total":96636764160,"available":95862460416,"internally_reserved":0,"allocated":146714624,"data_stored":146525422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618284,"kb_used_data":140956,"kb_used_omap":4,"kb_used_meta":477307,"kb_avail":93753556,"statfs":{"total":96636764160,"available":96003641344,"internally_reserved":0,"allocated":144338944,"data_stored":144149742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":488763087},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730032,"kb_used_data":143072,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93641808,"statfs":{"total":96636764160,"available":95889211392,"internally_reserved":0,"allocated":146505728,"data_stored":146316526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":601026254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":145473536,"data_stored":145473536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143998976,"data_stored":143998976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":146210816,"data_stored":146210816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":145137664,"data_stored":145137664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:00.330 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:00.976 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:00.976 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:00.999 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-16T04:50:00.731348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":283248320,"num_objects":34582,"num_object_clones":0,"num_object_copies":138328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166905,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6321,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160932,"ondisk_log_size":160932,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2884972,"kb_used_data":558312,"kb_used_omap":14,"kb_used_meta":2326577,"kb_avail":374602388,"statfs":{"total":386547056640,"available":383592845312,"internally_reserved":0,"allocated":571711488,"data_stored":570954680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2382415409},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,3,2,12,19,21,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19726336,"num_objects":-2408,"num_object_clones":0,"num_object_copies":-9632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003702"},"pg_stats":[{"pgid":"1.f","version":"35'10469","reported_seq":10586,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:59.073225+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:59.073225+0000","last_peered":"2024-09-16T04:49:59.073225+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:59.073225+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17899520,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10469,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10496","reported_seq":10585,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.735834+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:58.735834+0000","last_peered":"2024-09-16T04:49:58.735834+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:58.735834+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17839372,"num_objects":2178,"num_object_clones":0,"num_object_copies":8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10494,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-25}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10458","reported_seq":10555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:57.690662+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:57.690662+0000","last_peered":"2024-09-16T04:49:57.690662+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:57.690662+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18119326,"num_objects":2213,"num_object_clones":0,"num_object_copies":8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10457,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10475","reported_seq":13461,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.666318+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:58.666318+0000","last_peered":"2024-09-16T04:49:58.666318+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:58.666318+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17978636,"num_objects":2195,"num_object_clones":0,"num_object_copies":8780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10473,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1458,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10543","reported_seq":10674,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.771393+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:58.771393+0000","last_peered":"2024-09-16T04:49:58.771393+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:58.771393+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18003212,"num_objects":2198,"num_object_clones":0,"num_object_copies":8792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10541,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-50}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10264","reported_seq":10373,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.942701+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:58.942701+0000","last_peered":"2024-09-16T04:49:58.942701+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:58.942701+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":2048,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10264,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-130}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10477","reported_seq":10575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.633896+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:58.633896+0000","last_peered":"2024-09-16T04:49:58.633896+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:58.633896+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17388812,"num_objects":2123,"num_object_clones":0,"num_object_copies":8492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10475,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-90}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10530","reported_seq":15092,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:56.065223+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:56.065223+0000","last_peered":"2024-09-16T04:49:56.065223+0000","last_clean":"2024-09-16T04:49:56.065223+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:56.065223+0000","last_undegraded":"2024-09-16T04:49:56.065223+0000","last_fullsized":"2024-09-16T04:49:56.065223+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:56.996738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18470156,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10528,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10616","reported_seq":15374,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:58.913651+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:58.913651+0000","last_peered":"2024-09-16T04:49:58.913651+0000","last_clean":"2024-09-16T04:49:58.913651+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:58.913651+0000","last_undegraded":"2024-09-16T04:49:58.913651+0000","last_fullsized":"2024-09-16T04:49:58.913651+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17525272,"num_objects":2140,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10517","reported_seq":10623,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:57.740428+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:57.740428+0000","last_peered":"2024-09-16T04:49:57.740428+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:57.740428+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355468,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10515,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":52}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10368","reported_seq":10501,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.979951+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:58.979951+0000","last_peered":"2024-09-16T04:49:58.979951+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:58.979951+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17387410,"num_objects":2123,"num_object_clones":0,"num_object_copies":8492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10365,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-32}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10250","reported_seq":10343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:59.109646+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:59.109646+0000","last_peered":"2024-09-16T04:49:59.109646+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:59.109646+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17046150,"num_objects":2081,"num_object_clones":0,"num_object_copies":8324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10249,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-24}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10370","reported_seq":10459,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:57.599276+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:57.599276+0000","last_peered":"2024-09-16T04:49:57.599276+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:57.599276+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17717894,"num_objects":2163,"num_object_clones":0,"num_object_copies":8652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10369,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-58}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10589","reported_seq":10722,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:59.037051+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:59.037051+0000","last_peered":"2024-09-16T04:49:59.037051+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:59.037051+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17962252,"num_objects":2193,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10587,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-5}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10252","reported_seq":10351,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:57.597535+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:57.597535+0000","last_peered":"2024-09-16T04:49:57.597535+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:57.597535+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17995020,"num_objects":2197,"num_object_clones":0,"num_object_copies":8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10250,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":74}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10258","reported_seq":10353,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.656864+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:58.656864+0000","last_peered":"2024-09-16T04:49:58.656864+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:58.656864+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16782604,"num_objects":2049,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10256,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":283248320,"num_objects":34582,"num_object_clones":0,"num_object_copies":138328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166905,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6321,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":568623104,"data_stored":568623104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160932,"ondisk_log_size":160932,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743888,"kb_used_data":138816,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93627952,"statfs":{"total":96636764160,"available":95875022848,"internally_reserved":0,"allocated":142147584,"data_stored":141958382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619572943},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,8,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774448,"kb_used_data":139360,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93597392,"statfs":{"total":96636764160,"available":95843729408,"internally_reserved":0,"allocated":142704640,"data_stored":142515438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,9,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618284,"kb_used_data":140956,"kb_used_omap":4,"kb_used_meta":477307,"kb_avail":93753556,"statfs":{"total":96636764160,"available":96003641344,"internally_reserved":0,"allocated":144338944,"data_stored":144149742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":488763087},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748352,"kb_used_data":139180,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93623488,"statfs":{"total":96636764160,"available":95870451712,"internally_reserved":0,"allocated":142520320,"data_stored":142331118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":623767246},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,2,0,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":141385728,"data_stored":141385728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143998976,"data_stored":143998976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":142217216,"data_stored":142217216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":141021184,"data_stored":141021184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:01.000 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:01.584 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:01.584 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:01.614 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-16T04:50:00.731348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":283248320,"num_objects":34582,"num_object_clones":0,"num_object_copies":138328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166905,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6321,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160932,"ondisk_log_size":160932,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2884972,"kb_used_data":558312,"kb_used_omap":14,"kb_used_meta":2326577,"kb_avail":374602388,"statfs":{"total":386547056640,"available":383592845312,"internally_reserved":0,"allocated":571711488,"data_stored":570954680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2382415409},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,3,2,12,19,21,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19726336,"num_objects":-2408,"num_object_clones":0,"num_object_copies":-9632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003702"},"pg_stats":[{"pgid":"1.f","version":"35'10469","reported_seq":10586,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:59.073225+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:59.073225+0000","last_peered":"2024-09-16T04:49:59.073225+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:59.073225+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17899520,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10469,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10496","reported_seq":10585,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.735834+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:58.735834+0000","last_peered":"2024-09-16T04:49:58.735834+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:58.735834+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17839372,"num_objects":2178,"num_object_clones":0,"num_object_copies":8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10494,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-25}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10458","reported_seq":10555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:57.690662+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:57.690662+0000","last_peered":"2024-09-16T04:49:57.690662+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:57.690662+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18119326,"num_objects":2213,"num_object_clones":0,"num_object_copies":8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10457,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10475","reported_seq":13461,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.666318+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:58.666318+0000","last_peered":"2024-09-16T04:49:58.666318+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:58.666318+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17978636,"num_objects":2195,"num_object_clones":0,"num_object_copies":8780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10473,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1458,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10543","reported_seq":10674,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.771393+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:58.771393+0000","last_peered":"2024-09-16T04:49:58.771393+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:58.771393+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18003212,"num_objects":2198,"num_object_clones":0,"num_object_copies":8792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10541,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-50}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10264","reported_seq":10373,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.942701+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:58.942701+0000","last_peered":"2024-09-16T04:49:58.942701+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:58.942701+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":2048,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10264,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-130}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10477","reported_seq":10575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.633896+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:58.633896+0000","last_peered":"2024-09-16T04:49:58.633896+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:58.633896+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17388812,"num_objects":2123,"num_object_clones":0,"num_object_copies":8492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10475,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-90}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10530","reported_seq":15092,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:56.065223+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:49:56.065223+0000","last_peered":"2024-09-16T04:49:56.065223+0000","last_clean":"2024-09-16T04:49:56.065223+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:49:56.065223+0000","last_undegraded":"2024-09-16T04:49:56.065223+0000","last_fullsized":"2024-09-16T04:49:56.065223+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:49:56.996738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18470156,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10528,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10616","reported_seq":15374,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:58.913651+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:58.913651+0000","last_peered":"2024-09-16T04:49:58.913651+0000","last_clean":"2024-09-16T04:49:58.913651+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:58.913651+0000","last_undegraded":"2024-09-16T04:49:58.913651+0000","last_fullsized":"2024-09-16T04:49:58.913651+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17525272,"num_objects":2140,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10517","reported_seq":10623,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:57.740428+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:57.740428+0000","last_peered":"2024-09-16T04:49:57.740428+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:57.740428+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355468,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10515,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":52}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10368","reported_seq":10501,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.979951+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:58.979951+0000","last_peered":"2024-09-16T04:49:58.979951+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:58.979951+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17387410,"num_objects":2123,"num_object_clones":0,"num_object_copies":8492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10365,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-32}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10250","reported_seq":10343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:59.109646+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:59.109646+0000","last_peered":"2024-09-16T04:49:59.109646+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:59.109646+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17046150,"num_objects":2081,"num_object_clones":0,"num_object_copies":8324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10249,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-24}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10370","reported_seq":10459,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:57.599276+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:57.599276+0000","last_peered":"2024-09-16T04:49:57.599276+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:57.599276+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17717894,"num_objects":2163,"num_object_clones":0,"num_object_copies":8652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10369,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-58}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10589","reported_seq":10722,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:59.037051+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:59.037051+0000","last_peered":"2024-09-16T04:49:59.037051+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:59.037051+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17962252,"num_objects":2193,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10587,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-5}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10252","reported_seq":10351,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:57.597535+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:57.597535+0000","last_peered":"2024-09-16T04:49:57.597535+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:57.597535+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17995020,"num_objects":2197,"num_object_clones":0,"num_object_copies":8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10250,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":74}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10258","reported_seq":10353,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.656864+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:58.656864+0000","last_peered":"2024-09-16T04:49:58.656864+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:58.656864+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16782604,"num_objects":2049,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10256,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":283248320,"num_objects":34582,"num_object_clones":0,"num_object_copies":138328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166905,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6321,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":568623104,"data_stored":568623104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160932,"ondisk_log_size":160932,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743888,"kb_used_data":138816,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93627952,"statfs":{"total":96636764160,"available":95875022848,"internally_reserved":0,"allocated":142147584,"data_stored":141958382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619572943},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,8,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774448,"kb_used_data":139360,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93597392,"statfs":{"total":96636764160,"available":95843729408,"internally_reserved":0,"allocated":142704640,"data_stored":142515438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,9,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618284,"kb_used_data":140956,"kb_used_omap":4,"kb_used_meta":477307,"kb_avail":93753556,"statfs":{"total":96636764160,"available":96003641344,"internally_reserved":0,"allocated":144338944,"data_stored":144149742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":488763087},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748352,"kb_used_data":139180,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93623488,"statfs":{"total":96636764160,"available":95870451712,"internally_reserved":0,"allocated":142520320,"data_stored":142331118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":623767246},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,2,0,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":141385728,"data_stored":141385728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143998976,"data_stored":143998976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":142217216,"data_stored":142217216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":141021184,"data_stored":141021184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:01.615 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:50:01.615 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10469", 'reported_seq': 10586, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:59.073225+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:49:59.073225+0000', 'last_peered': '2024-09-16T04:49:59.073225+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:49:59.073225+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17899520, 'num_objects': 2185, 'num_object_clones': 0, 'num_object_copies': 8740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10469, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:01.616 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:50:01.616 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10496", 'reported_seq': 10585, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:58.735834+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:49:58.735834+0000', 'last_peered': '2024-09-16T04:49:58.735834+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:49:58.735834+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17839372, 'num_objects': 2178, 'num_object_clones': 0, 'num_object_copies': 8712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10494, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -25}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:01.616 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:50:01.616 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10458", 'reported_seq': 10555, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:57.690662+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:49:57.690662+0000', 'last_peered': '2024-09-16T04:49:57.690662+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:49:57.690662+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18119326, 'num_objects': 2213, 'num_object_clones': 0, 'num_object_copies': 8852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2213, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10457, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -6}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:01.616 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:50:01.616 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10475", 'reported_seq': 13461, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:58.666318+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:49:58.666318+0000', 'last_peered': '2024-09-16T04:49:58.666318+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:49:58.666318+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17978636, 'num_objects': 2195, 'num_object_clones': 0, 'num_object_copies': 8780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 775, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10473, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1458, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1420}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:01.616 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:50:01.616 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10543", 'reported_seq': 10674, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:58.771393+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:49:58.771393+0000', 'last_peered': '2024-09-16T04:49:58.771393+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:49:58.771393+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18003212, 'num_objects': 2198, 'num_object_clones': 0, 'num_object_copies': 8792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2198, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10541, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -50}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:01.616 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:50:01.616 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10264", 'reported_seq': 10373, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:58.942701+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:49:58.942701+0000', 'last_peered': '2024-09-16T04:49:58.942701+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:49:58.942701+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16777216, 'num_objects': 2048, 'num_object_clones': 0, 'num_object_copies': 8192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2048, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10264, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -130}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:01.617 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:50:01.617 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10477", 'reported_seq': 10575, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:58.633896+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:49:58.633896+0000', 'last_peered': '2024-09-16T04:49:58.633896+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:49:58.633896+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17388812, 'num_objects': 2123, 'num_object_clones': 0, 'num_object_copies': 8492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10475, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -90}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:01.617 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:50:01.617 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10517", 'reported_seq': 10623, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:57.740428+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:49:57.740428+0000', 'last_peered': '2024-09-16T04:49:57.740428+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:49:57.740428+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18355468, 'num_objects': 2241, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10515, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 52}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:01.617 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:50:01.617 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10368", 'reported_seq': 10501, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:58.979951+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:49:58.979951+0000', 'last_peered': '2024-09-16T04:49:58.979951+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:49:58.979951+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17387410, 'num_objects': 2123, 'num_object_clones': 0, 'num_object_copies': 8492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10365, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -32}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:01.617 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:50:01.617 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10250", 'reported_seq': 10343, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:59.109646+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:49:59.109646+0000', 'last_peered': '2024-09-16T04:49:59.109646+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:49:59.109646+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17046150, 'num_objects': 2081, 'num_object_clones': 0, 'num_object_copies': 8324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2081, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10249, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -24}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:01.617 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:50:01.617 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10370", 'reported_seq': 10459, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:57.599276+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:49:57.599276+0000', 'last_peered': '2024-09-16T04:49:57.599276+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:49:57.599276+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17717894, 'num_objects': 2163, 'num_object_clones': 0, 'num_object_copies': 8652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10369, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -58}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:01.617 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:50:01.618 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10589", 'reported_seq': 10722, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:59.037051+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:49:59.037051+0000', 'last_peered': '2024-09-16T04:49:59.037051+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:49:59.037051+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17962252, 'num_objects': 2193, 'num_object_clones': 0, 'num_object_copies': 8772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10587, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -5}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:01.618 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:50:01.618 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10252", 'reported_seq': 10351, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:57.597535+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:49:57.597535+0000', 'last_peered': '2024-09-16T04:49:57.597535+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:49:57.597535+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17995020, 'num_objects': 2197, 'num_object_clones': 0, 'num_object_copies': 8788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2197, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10250, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 74}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:01.618 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:50:01.618 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10258", 'reported_seq': 10353, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:49:58.656864+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:49:58.656864+0000', 'last_peered': '2024-09-16T04:49:58.656864+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:49:58.656864+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16782604, 'num_objects': 2049, 'num_object_clones': 0, 'num_object_copies': 8196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10256, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -179}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:01.618 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:50:02.348 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:02.348 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29261/138328 objects degraded (21.153%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":614,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":34582,"data_bytes":283248320,"bytes_used":2954211328,"bytes_avail":383592845312,"bytes_total":386547056640,"degraded_objects":29261,"degraded_total":138328,"degraded_ratio":0.21153345671158405,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":200},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:50:02.385 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:50:02.385 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:02.959 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:02.960 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:02.989 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-16T04:50:02.732180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":282912448,"num_objects":34541,"num_object_clones":0,"num_object_copies":138164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34541,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166946,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6321,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160973,"ondisk_log_size":160973,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2868604,"kb_used_data":554424,"kb_used_omap":14,"kb_used_meta":2314097,"kb_avail":374618756,"statfs":{"total":386547056640,"available":383609606144,"internally_reserved":0,"allocated":567730176,"data_stored":566973368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2369635889},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,4,5,15,16,18,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16474112,"num_objects":-2011,"num_object_clones":0,"num_object_copies":-8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2011,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003982"},"pg_stats":[{"pgid":"1.f","version":"35'10469","reported_seq":10586,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:59.073225+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:49:59.073225+0000","last_peered":"2024-09-16T04:49:59.073225+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:49:59.073225+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17899520,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10469,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10496","reported_seq":10585,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.735834+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:49:58.735834+0000","last_peered":"2024-09-16T04:49:58.735834+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:49:58.735834+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17839372,"num_objects":2178,"num_object_clones":0,"num_object_copies":8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10494,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-25}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10458","reported_seq":10555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:57.690662+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:49:57.690662+0000","last_peered":"2024-09-16T04:49:57.690662+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:49:57.690662+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18119326,"num_objects":2213,"num_object_clones":0,"num_object_copies":8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10457,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10475","reported_seq":13461,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.666318+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:49:58.666318+0000","last_peered":"2024-09-16T04:49:58.666318+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:49:58.666318+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17978636,"num_objects":2195,"num_object_clones":0,"num_object_copies":8780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10473,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1458,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10543","reported_seq":10674,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.771393+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:49:58.771393+0000","last_peered":"2024-09-16T04:49:58.771393+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:49:58.771393+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18003212,"num_objects":2198,"num_object_clones":0,"num_object_copies":8792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10541,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-50}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10264","reported_seq":10373,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.942701+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:49:58.942701+0000","last_peered":"2024-09-16T04:49:58.942701+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:49:58.942701+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":2048,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10264,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-130}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10477","reported_seq":10575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.633896+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:49:58.633896+0000","last_peered":"2024-09-16T04:49:58.633896+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:49:58.633896+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17388812,"num_objects":2123,"num_object_clones":0,"num_object_copies":8492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10475,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-90}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10571","reported_seq":15133,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:00.715575+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:00.715575+0000","last_peered":"2024-09-16T04:50:00.715575+0000","last_clean":"2024-09-16T04:50:00.715575+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:00.715575+0000","last_undegraded":"2024-09-16T04:50:00.715575+0000","last_fullsized":"2024-09-16T04:50:00.715575+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:02.199843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18134284,"num_objects":2214,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10569,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10616","reported_seq":15374,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:49:58.913651+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:49:58.913651+0000","last_peered":"2024-09-16T04:49:58.913651+0000","last_clean":"2024-09-16T04:49:58.913651+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:49:58.913651+0000","last_undegraded":"2024-09-16T04:49:58.913651+0000","last_fullsized":"2024-09-16T04:49:58.913651+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17525272,"num_objects":2140,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10517","reported_seq":10623,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:57.740428+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:49:57.740428+0000","last_peered":"2024-09-16T04:49:57.740428+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:49:57.740428+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18355468,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10515,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":52}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10368","reported_seq":10501,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.979951+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:49:58.979951+0000","last_peered":"2024-09-16T04:49:58.979951+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:49:58.979951+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17387410,"num_objects":2123,"num_object_clones":0,"num_object_copies":8492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10365,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-32}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10250","reported_seq":10343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:59.109646+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:49:59.109646+0000","last_peered":"2024-09-16T04:49:59.109646+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:49:59.109646+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17046150,"num_objects":2081,"num_object_clones":0,"num_object_copies":8324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10249,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-24}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10370","reported_seq":10459,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:57.599276+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:49:57.599276+0000","last_peered":"2024-09-16T04:49:57.599276+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:49:57.599276+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17717894,"num_objects":2163,"num_object_clones":0,"num_object_copies":8652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10369,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-58}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10589","reported_seq":10722,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:59.037051+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:49:59.037051+0000","last_peered":"2024-09-16T04:49:59.037051+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:49:59.037051+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17962252,"num_objects":2193,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10587,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-5}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10252","reported_seq":10351,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:57.597535+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:49:57.597535+0000","last_peered":"2024-09-16T04:49:57.597535+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:49:57.597535+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17995020,"num_objects":2197,"num_object_clones":0,"num_object_copies":8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10250,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":74}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10258","reported_seq":10353,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:49:58.656864+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:49:58.656864+0000","last_peered":"2024-09-16T04:49:58.656864+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:49:58.656864+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16782604,"num_objects":2049,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10256,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":282912448,"num_objects":34541,"num_object_clones":0,"num_object_copies":138164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34541,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166946,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6321,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":564572160,"data_stored":564572160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160973,"ondisk_log_size":160973,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743888,"kb_used_data":138816,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93627952,"statfs":{"total":96636764160,"available":95875022848,"internally_reserved":0,"allocated":142147584,"data_stored":141958382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619572943},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,8,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774448,"kb_used_data":139360,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93597392,"statfs":{"total":96636764160,"available":95843729408,"internally_reserved":0,"allocated":142704640,"data_stored":142515438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,9,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601916,"kb_used_data":137068,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93769924,"statfs":{"total":96636764160,"available":96020402176,"internally_reserved":0,"allocated":140357632,"data_stored":140168430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475983567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,0,6,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748352,"kb_used_data":139180,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93623488,"statfs":{"total":96636764160,"available":95870451712,"internally_reserved":0,"allocated":142520320,"data_stored":142331118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":623767246},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,2,0,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":141385728,"data_stored":141385728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139948032,"data_stored":139948032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":142217216,"data_stored":142217216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":141021184,"data_stored":141021184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:05.991 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:06.623 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:06.623 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:06.644 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-09-16T04:50:04.732780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":275375808,"num_objects":33621,"num_object_clones":0,"num_object_copies":134484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167866,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6381,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160693,"ondisk_log_size":160693,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2868796,"kb_used_data":542108,"kb_used_omap":14,"kb_used_meta":2326577,"kb_avail":374618564,"statfs":{"total":386547056640,"available":383609409536,"internally_reserved":0,"allocated":555118592,"data_stored":554361784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2382415409},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,8,10,19,20,17,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21372928,"num_objects":-2609,"num_object_clones":0,"num_object_copies":-10436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2609,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004064"},"pg_stats":[{"pgid":"1.f","version":"35'10530","reported_seq":10647,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.832064+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:03.832064+0000","last_peered":"2024-09-16T04:50:03.832064+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:03.832064+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17399808,"num_objects":2124,"num_object_clones":0,"num_object_copies":8496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10530,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-62}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10560","reported_seq":10649,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.469760+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:03.469760+0000","last_peered":"2024-09-16T04:50:03.469760+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:03.469760+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17315084,"num_objects":2114,"num_object_clones":0,"num_object_copies":8456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10558,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-89}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10527","reported_seq":10626,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:02.680813+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:02.680813+0000","last_peered":"2024-09-16T04:50:02.680813+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:02.680813+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17554078,"num_objects":2144,"num_object_clones":0,"num_object_copies":8576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2144,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10526,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-75}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10537","reported_seq":13640,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.806315+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:03.806315+0000","last_peered":"2024-09-16T04:50:03.806315+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:03.806315+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17470732,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1518,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1418}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10592","reported_seq":10725,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.899139+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:03.899139+0000","last_peered":"2024-09-16T04:50:03.899139+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:03.899139+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17601804,"num_objects":2149,"num_object_clones":0,"num_object_copies":8596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10590,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-99}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10320","reported_seq":10431,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:04.134456+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:04.134456+0000","last_peered":"2024-09-16T04:50:04.134456+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:04.134456+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16318464,"num_objects":1992,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10320,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10534","reported_seq":10632,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.610004+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:03.610004+0000","last_peered":"2024-09-16T04:50:03.610004+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:03.610004+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16921868,"num_objects":2066,"num_object_clones":0,"num_object_copies":8264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10532,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-147}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10571","reported_seq":15133,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:00.715575+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:00.715575+0000","last_peered":"2024-09-16T04:50:00.715575+0000","last_clean":"2024-09-16T04:50:00.715575+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:00.715575+0000","last_undegraded":"2024-09-16T04:50:00.715575+0000","last_fullsized":"2024-09-16T04:50:00.715575+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:02.199843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18134284,"num_objects":2214,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10569,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10672","reported_seq":15430,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:04.024715+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:04.024715+0000","last_peered":"2024-09-16T04:50:04.024715+0000","last_clean":"2024-09-16T04:50:04.024715+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:04.024715+0000","last_undegraded":"2024-09-16T04:50:04.024715+0000","last_fullsized":"2024-09-16T04:50:04.024715+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:04.228762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17066520,"num_objects":2084,"num_object_clones":0,"num_object_copies":8336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10669,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10578","reported_seq":10685,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:02.681875+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:02.681875+0000","last_peered":"2024-09-16T04:50:02.681875+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:02.681875+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17855756,"num_objects":2180,"num_object_clones":0,"num_object_copies":8720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10576,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-9}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10417","reported_seq":10550,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:04.157315+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:04.157315+0000","last_peered":"2024-09-16T04:50:04.157315+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:04.157315+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16986002,"num_objects":2074,"num_object_clones":0,"num_object_copies":8296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10414,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-81}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10332","reported_seq":10425,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:04.056202+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:04.056202+0000","last_peered":"2024-09-16T04:50:04.056202+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:04.056202+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16374406,"num_objects":1999,"num_object_clones":0,"num_object_copies":7996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10331,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-106}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10436","reported_seq":10525,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:02.747914+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:02.747914+0000","last_peered":"2024-09-16T04:50:02.747914+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:02.747914+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17177222,"num_objects":2097,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10435,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-124}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10654","reported_seq":10787,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:04.108754+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:04.108754+0000","last_peered":"2024-09-16T04:50:04.108754+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:04.108754+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17429772,"num_objects":2128,"num_object_clones":0,"num_object_copies":8512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10652,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-70}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10314","reported_seq":10417,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:02.605343+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:02.605343+0000","last_peered":"2024-09-16T04:50:02.605343+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:02.605343+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17487116,"num_objects":2135,"num_object_clones":0,"num_object_copies":8540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10312,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":12}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10319","reported_seq":10411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.523389+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:03.523389+0000","last_peered":"2024-09-16T04:50:03.523389+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:03.523389+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16282892,"num_objects":1988,"num_object_clones":0,"num_object_copies":7952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10317,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-240}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":275375808,"num_objects":33621,"num_object_clones":0,"num_object_copies":134484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167866,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6381,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552726528,"data_stored":552726528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160693,"ondisk_log_size":160693,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744204,"kb_used_data":134972,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93627636,"statfs":{"total":96636764160,"available":95874699264,"internally_reserved":0,"allocated":138211328,"data_stored":138022126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623832783},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774696,"kb_used_data":135448,"kb_used_omap":1,"kb_used_meta":639230,"kb_avail":93597144,"statfs":{"total":96636764160,"available":95843475456,"internally_reserved":0,"allocated":138698752,"data_stored":138509550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":654571973},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,3,10,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601916,"kb_used_data":137068,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93769924,"statfs":{"total":96636764160,"available":96020402176,"internally_reserved":0,"allocated":140357632,"data_stored":140168430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475983567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,0,6,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747980,"kb_used_data":134620,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93623860,"statfs":{"total":96636764160,"available":95870832640,"internally_reserved":0,"allocated":137850880,"data_stored":137661678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":628027086},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,8,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":137453568,"data_stored":137453568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139948032,"data_stored":139948032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":138240000,"data_stored":138240000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":137084928,"data_stored":137084928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:06.646 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:07.236 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:07.238 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:07.270 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-16T04:50:06.733580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":274859712,"num_objects":33558,"num_object_clones":0,"num_object_copies":134232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167929,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6381,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160656,"ondisk_log_size":160656,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2869044,"kb_used_data":538196,"kb_used_omap":14,"kb_used_meta":2330737,"kb_avail":374618316,"statfs":{"total":386547056640,"available":383609155584,"internally_reserved":0,"allocated":551112704,"data_stored":550355896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2386675249},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,5,8,22,16,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16506880,"num_objects":-2015,"num_object_clones":0,"num_object_copies":-8060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2015,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004241"},"pg_stats":[{"pgid":"1.f","version":"35'10530","reported_seq":10647,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.832064+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:03.832064+0000","last_peered":"2024-09-16T04:50:03.832064+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:03.832064+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17399808,"num_objects":2124,"num_object_clones":0,"num_object_copies":8496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10530,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-62}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10560","reported_seq":10649,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.469760+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:03.469760+0000","last_peered":"2024-09-16T04:50:03.469760+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:03.469760+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17315084,"num_objects":2114,"num_object_clones":0,"num_object_copies":8456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10558,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-89}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10527","reported_seq":10626,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:02.680813+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:02.680813+0000","last_peered":"2024-09-16T04:50:02.680813+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:02.680813+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17554078,"num_objects":2144,"num_object_clones":0,"num_object_copies":8576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2144,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10526,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-75}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10537","reported_seq":13640,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.806315+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:03.806315+0000","last_peered":"2024-09-16T04:50:03.806315+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:03.806315+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17470732,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1518,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1418}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10592","reported_seq":10725,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.899139+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:03.899139+0000","last_peered":"2024-09-16T04:50:03.899139+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:03.899139+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17601804,"num_objects":2149,"num_object_clones":0,"num_object_copies":8596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10590,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-99}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10320","reported_seq":10431,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:04.134456+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:04.134456+0000","last_peered":"2024-09-16T04:50:04.134456+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:04.134456+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16318464,"num_objects":1992,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10320,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10534","reported_seq":10632,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.610004+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:03.610004+0000","last_peered":"2024-09-16T04:50:03.610004+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:03.610004+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16921868,"num_objects":2066,"num_object_clones":0,"num_object_copies":8264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10532,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-147}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10634","reported_seq":15196,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:06.187464+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:06.187464+0000","last_peered":"2024-09-16T04:50:06.187464+0000","last_clean":"2024-09-16T04:50:06.187464+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:06.187464+0000","last_undegraded":"2024-09-16T04:50:06.187464+0000","last_fullsized":"2024-09-16T04:50:06.187464+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:08.173758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17618188,"num_objects":2151,"num_object_clones":0,"num_object_copies":8604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10632,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10672","reported_seq":15430,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:04.024715+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:04.024715+0000","last_peered":"2024-09-16T04:50:04.024715+0000","last_clean":"2024-09-16T04:50:04.024715+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:04.024715+0000","last_undegraded":"2024-09-16T04:50:04.024715+0000","last_fullsized":"2024-09-16T04:50:04.024715+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:04.228762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17066520,"num_objects":2084,"num_object_clones":0,"num_object_copies":8336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10669,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10578","reported_seq":10685,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:02.681875+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:02.681875+0000","last_peered":"2024-09-16T04:50:02.681875+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:02.681875+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17855756,"num_objects":2180,"num_object_clones":0,"num_object_copies":8720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10576,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-9}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10417","reported_seq":10550,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:04.157315+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:04.157315+0000","last_peered":"2024-09-16T04:50:04.157315+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:04.157315+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16986002,"num_objects":2074,"num_object_clones":0,"num_object_copies":8296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10414,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-81}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10332","reported_seq":10425,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:04.056202+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:04.056202+0000","last_peered":"2024-09-16T04:50:04.056202+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:04.056202+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16374406,"num_objects":1999,"num_object_clones":0,"num_object_copies":7996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10331,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-106}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10436","reported_seq":10525,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:02.747914+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:02.747914+0000","last_peered":"2024-09-16T04:50:02.747914+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:02.747914+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17177222,"num_objects":2097,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10435,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-124}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10654","reported_seq":10787,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:04.108754+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:04.108754+0000","last_peered":"2024-09-16T04:50:04.108754+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:04.108754+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17429772,"num_objects":2128,"num_object_clones":0,"num_object_copies":8512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10652,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-70}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10314","reported_seq":10417,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:02.605343+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:02.605343+0000","last_peered":"2024-09-16T04:50:02.605343+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:02.605343+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17487116,"num_objects":2135,"num_object_clones":0,"num_object_copies":8540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10312,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":12}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10319","reported_seq":10411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.523389+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:03.523389+0000","last_peered":"2024-09-16T04:50:03.523389+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:03.523389+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16282892,"num_objects":1988,"num_object_clones":0,"num_object_copies":7952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10317,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-240}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":274859712,"num_objects":33558,"num_object_clones":0,"num_object_copies":134232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167929,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6381,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548700160,"data_stored":548700160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160656,"ondisk_log_size":160656,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744204,"kb_used_data":134972,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93627636,"statfs":{"total":96636764160,"available":95874699264,"internally_reserved":0,"allocated":138211328,"data_stored":138022126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623832783},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774696,"kb_used_data":135448,"kb_used_omap":1,"kb_used_meta":639230,"kb_avail":93597144,"statfs":{"total":96636764160,"available":95843475456,"internally_reserved":0,"allocated":138698752,"data_stored":138509550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":654571973},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,3,10,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602164,"kb_used_data":133156,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93769676,"statfs":{"total":96636764160,"available":96020148224,"internally_reserved":0,"allocated":136351744,"data_stored":136162542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747980,"kb_used_data":134620,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93623860,"statfs":{"total":96636764160,"available":95870832640,"internally_reserved":0,"allocated":137850880,"data_stored":137661678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":628027086},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,8,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":137453568,"data_stored":137453568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135921664,"data_stored":135921664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":138240000,"data_stored":138240000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":137084928,"data_stored":137084928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:07.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:50:07.273 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10530", 'reported_seq': 10647, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:03.832064+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:50:03.832064+0000', 'last_peered': '2024-09-16T04:50:03.832064+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:50:03.832064+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17399808, 'num_objects': 2124, 'num_object_clones': 0, 'num_object_copies': 8496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10530, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -62}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:07.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:50:07.273 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10560", 'reported_seq': 10649, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:03.469760+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:50:03.469760+0000', 'last_peered': '2024-09-16T04:50:03.469760+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:50:03.469760+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17315084, 'num_objects': 2114, 'num_object_clones': 0, 'num_object_copies': 8456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2114, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10558, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -89}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:07.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:50:07.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10527", 'reported_seq': 10626, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:02.680813+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:50:02.680813+0000', 'last_peered': '2024-09-16T04:50:02.680813+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:50:02.680813+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17554078, 'num_objects': 2144, 'num_object_clones': 0, 'num_object_copies': 8576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2144, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10526, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -75}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:07.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:50:07.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10537", 'reported_seq': 13640, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:03.806315+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:50:03.806315+0000', 'last_peered': '2024-09-16T04:50:03.806315+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:50:03.806315+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17470732, 'num_objects': 2133, 'num_object_clones': 0, 'num_object_copies': 8532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 715, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10535, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1518, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1418}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:07.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:50:07.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10592", 'reported_seq': 10725, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:03.899139+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:50:03.899139+0000', 'last_peered': '2024-09-16T04:50:03.899139+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:50:03.899139+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17601804, 'num_objects': 2149, 'num_object_clones': 0, 'num_object_copies': 8596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2149, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10590, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -99}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:07.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:50:07.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10320", 'reported_seq': 10431, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:04.134456+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:50:04.134456+0000', 'last_peered': '2024-09-16T04:50:04.134456+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:50:04.134456+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16318464, 'num_objects': 1992, 'num_object_clones': 0, 'num_object_copies': 7968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10320, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:07.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:50:07.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10534", 'reported_seq': 10632, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:03.610004+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:50:03.610004+0000', 'last_peered': '2024-09-16T04:50:03.610004+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:50:03.610004+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16921868, 'num_objects': 2066, 'num_object_clones': 0, 'num_object_copies': 8264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2066, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10532, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -147}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:07.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:50:07.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10578", 'reported_seq': 10685, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:02.681875+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:50:02.681875+0000', 'last_peered': '2024-09-16T04:50:02.681875+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:50:02.681875+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17855756, 'num_objects': 2180, 'num_object_clones': 0, 'num_object_copies': 8720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10576, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -9}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:07.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:50:07.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10417", 'reported_seq': 10550, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:04.157315+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:50:04.157315+0000', 'last_peered': '2024-09-16T04:50:04.157315+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:50:04.157315+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16986002, 'num_objects': 2074, 'num_object_clones': 0, 'num_object_copies': 8296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2074, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10414, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -81}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:07.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:50:07.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10332", 'reported_seq': 10425, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:04.056202+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:50:04.056202+0000', 'last_peered': '2024-09-16T04:50:04.056202+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:50:04.056202+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16374406, 'num_objects': 1999, 'num_object_clones': 0, 'num_object_copies': 7996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10331, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -106}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:07.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:50:07.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10436", 'reported_seq': 10525, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:02.747914+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:50:02.747914+0000', 'last_peered': '2024-09-16T04:50:02.747914+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:50:02.747914+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17177222, 'num_objects': 2097, 'num_object_clones': 0, 'num_object_copies': 8388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2097, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10435, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -124}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:07.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:50:07.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10654", 'reported_seq': 10787, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:04.108754+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:50:04.108754+0000', 'last_peered': '2024-09-16T04:50:04.108754+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:50:04.108754+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17429772, 'num_objects': 2128, 'num_object_clones': 0, 'num_object_copies': 8512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10652, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -70}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:07.277 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:50:07.277 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10314", 'reported_seq': 10417, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:02.605343+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:50:02.605343+0000', 'last_peered': '2024-09-16T04:50:02.605343+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:50:02.605343+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17487116, 'num_objects': 2135, 'num_object_clones': 0, 'num_object_copies': 8540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2135, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10312, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 12}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:07.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:50:07.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10319", 'reported_seq': 10411, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:03.523389+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:50:03.523389+0000', 'last_peered': '2024-09-16T04:50:03.523389+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:50:03.523389+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16282892, 'num_objects': 1988, 'num_object_clones': 0, 'num_object_copies': 7952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1988, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10317, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -240}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:07.278 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:50:07.979 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:07.979 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29201/134232 objects degraded (21.754%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":620,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":33558,"data_bytes":274859712,"bytes_used":2937901056,"bytes_avail":383609155584,"bytes_total":386547056640,"degraded_objects":29201,"degraded_total":134232,"degraded_ratio":0.21754127182788008,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":167},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:50:08.007 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:50:08.007 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:08.607 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:08.608 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:08.626 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-16T04:50:06.733580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":274859712,"num_objects":33558,"num_object_clones":0,"num_object_copies":134232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167929,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6381,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160656,"ondisk_log_size":160656,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2869044,"kb_used_data":538196,"kb_used_omap":14,"kb_used_meta":2330737,"kb_avail":374618316,"statfs":{"total":386547056640,"available":383609155584,"internally_reserved":0,"allocated":551112704,"data_stored":550355896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2386675249},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,5,8,22,16,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16506880,"num_objects":-2015,"num_object_clones":0,"num_object_copies":-8060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2015,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004241"},"pg_stats":[{"pgid":"1.f","version":"35'10530","reported_seq":10647,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.832064+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:03.832064+0000","last_peered":"2024-09-16T04:50:03.832064+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:03.832064+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17399808,"num_objects":2124,"num_object_clones":0,"num_object_copies":8496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10530,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-62}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10560","reported_seq":10649,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.469760+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:03.469760+0000","last_peered":"2024-09-16T04:50:03.469760+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:03.469760+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17315084,"num_objects":2114,"num_object_clones":0,"num_object_copies":8456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10558,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-89}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10527","reported_seq":10626,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:02.680813+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:02.680813+0000","last_peered":"2024-09-16T04:50:02.680813+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:02.680813+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17554078,"num_objects":2144,"num_object_clones":0,"num_object_copies":8576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2144,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10526,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-75}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10537","reported_seq":13640,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.806315+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:03.806315+0000","last_peered":"2024-09-16T04:50:03.806315+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:03.806315+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17470732,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1518,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1418}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10592","reported_seq":10725,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.899139+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:03.899139+0000","last_peered":"2024-09-16T04:50:03.899139+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:03.899139+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17601804,"num_objects":2149,"num_object_clones":0,"num_object_copies":8596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10590,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-99}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10320","reported_seq":10431,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:04.134456+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:04.134456+0000","last_peered":"2024-09-16T04:50:04.134456+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:04.134456+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16318464,"num_objects":1992,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10320,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10534","reported_seq":10632,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.610004+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:03.610004+0000","last_peered":"2024-09-16T04:50:03.610004+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:03.610004+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16921868,"num_objects":2066,"num_object_clones":0,"num_object_copies":8264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10532,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-147}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10634","reported_seq":15196,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:06.187464+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:06.187464+0000","last_peered":"2024-09-16T04:50:06.187464+0000","last_clean":"2024-09-16T04:50:06.187464+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:06.187464+0000","last_undegraded":"2024-09-16T04:50:06.187464+0000","last_fullsized":"2024-09-16T04:50:06.187464+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:08.173758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17618188,"num_objects":2151,"num_object_clones":0,"num_object_copies":8604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10632,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10672","reported_seq":15430,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:04.024715+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:04.024715+0000","last_peered":"2024-09-16T04:50:04.024715+0000","last_clean":"2024-09-16T04:50:04.024715+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:04.024715+0000","last_undegraded":"2024-09-16T04:50:04.024715+0000","last_fullsized":"2024-09-16T04:50:04.024715+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:04.228762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17066520,"num_objects":2084,"num_object_clones":0,"num_object_copies":8336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10669,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10578","reported_seq":10685,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:02.681875+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:02.681875+0000","last_peered":"2024-09-16T04:50:02.681875+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:02.681875+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17855756,"num_objects":2180,"num_object_clones":0,"num_object_copies":8720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10576,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-9}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10417","reported_seq":10550,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:04.157315+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:04.157315+0000","last_peered":"2024-09-16T04:50:04.157315+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:04.157315+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16986002,"num_objects":2074,"num_object_clones":0,"num_object_copies":8296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10414,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-81}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10332","reported_seq":10425,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:04.056202+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:04.056202+0000","last_peered":"2024-09-16T04:50:04.056202+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:04.056202+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16374406,"num_objects":1999,"num_object_clones":0,"num_object_copies":7996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10331,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-106}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10436","reported_seq":10525,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:02.747914+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:02.747914+0000","last_peered":"2024-09-16T04:50:02.747914+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:02.747914+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17177222,"num_objects":2097,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10435,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-124}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10654","reported_seq":10787,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:04.108754+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:04.108754+0000","last_peered":"2024-09-16T04:50:04.108754+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:04.108754+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17429772,"num_objects":2128,"num_object_clones":0,"num_object_copies":8512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10652,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-70}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10314","reported_seq":10417,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:02.605343+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:02.605343+0000","last_peered":"2024-09-16T04:50:02.605343+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:02.605343+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17487116,"num_objects":2135,"num_object_clones":0,"num_object_copies":8540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10312,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":12}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10319","reported_seq":10411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:03.523389+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:03.523389+0000","last_peered":"2024-09-16T04:50:03.523389+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:03.523389+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16282892,"num_objects":1988,"num_object_clones":0,"num_object_copies":7952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10317,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-240}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":274859712,"num_objects":33558,"num_object_clones":0,"num_object_copies":134232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167929,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6381,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548700160,"data_stored":548700160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160656,"ondisk_log_size":160656,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744204,"kb_used_data":134972,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93627636,"statfs":{"total":96636764160,"available":95874699264,"internally_reserved":0,"allocated":138211328,"data_stored":138022126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623832783},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":774696,"kb_used_data":135448,"kb_used_omap":1,"kb_used_meta":639230,"kb_avail":93597144,"statfs":{"total":96636764160,"available":95843475456,"internally_reserved":0,"allocated":138698752,"data_stored":138509550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":654571973},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,3,10,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602164,"kb_used_data":133156,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93769676,"statfs":{"total":96636764160,"available":96020148224,"internally_reserved":0,"allocated":136351744,"data_stored":136162542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747980,"kb_used_data":134620,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93623860,"statfs":{"total":96636764160,"available":95870832640,"internally_reserved":0,"allocated":137850880,"data_stored":137661678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":628027086},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,8,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":137453568,"data_stored":137453568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135921664,"data_stored":135921664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":138240000,"data_stored":138240000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":137084928,"data_stored":137084928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:11.628 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:12.278 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:12.278 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:12.305 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-09-16T04:50:10.734983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":267175616,"num_objects":32620,"num_object_clones":0,"num_object_copies":130480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32620,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168867,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6425,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160894,"ondisk_log_size":160894,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2853072,"kb_used_data":526408,"kb_used_omap":14,"kb_used_meta":2326577,"kb_avail":374634288,"statfs":{"total":386547056640,"available":383625510912,"internally_reserved":0,"allocated":539041792,"data_stored":538284984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2382415409},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,10,12,13,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21651456,"num_objects":-2643,"num_object_clones":0,"num_object_copies":-10572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2643,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004345"},"pg_stats":[{"pgid":"1.f","version":"35'10597","reported_seq":10711,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:09.210121+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:09.210121+0000","last_peered":"2024-09-16T04:50:09.210121+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:09.210121+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16850944,"num_objects":2057,"num_object_clones":0,"num_object_copies":8228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10597,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10632","reported_seq":10720,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:08.950747+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:08.950747+0000","last_peered":"2024-09-16T04:50:08.950747+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:08.950747+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16725260,"num_objects":2042,"num_object_clones":0,"num_object_copies":8168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10630,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-161}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10595","reported_seq":10694,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:07.823069+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:07.823069+0000","last_peered":"2024-09-16T04:50:07.823069+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:07.823069+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16997022,"num_objects":2076,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10594,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-143}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10611","reported_seq":13805,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:08.862037+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:08.862037+0000","last_peered":"2024-09-16T04:50:08.862037+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:08.862037+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16864524,"num_objects":2059,"num_object_clones":0,"num_object_copies":8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":671,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10609,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1562,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1388}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10652","reported_seq":10785,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:08.981344+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:08.981344+0000","last_peered":"2024-09-16T04:50:08.981344+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:08.981344+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17110284,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10650,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-159}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10367","reported_seq":10478,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:09.202930+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:09.202930+0000","last_peered":"2024-09-16T04:50:09.202930+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:09.202930+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15933440,"num_objects":1945,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10367,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10613","reported_seq":10709,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:08.779747+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:08.779747+0000","last_peered":"2024-09-16T04:50:08.779747+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:08.779747+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16274700,"num_objects":1987,"num_object_clones":0,"num_object_copies":7948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10611,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-226}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10634","reported_seq":15196,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:06.187464+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:06.187464+0000","last_peered":"2024-09-16T04:50:06.187464+0000","last_clean":"2024-09-16T04:50:06.187464+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:06.187464+0000","last_undegraded":"2024-09-16T04:50:06.187464+0000","last_fullsized":"2024-09-16T04:50:06.187464+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:08.173758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17618188,"num_objects":2151,"num_object_clones":0,"num_object_copies":8604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10632,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10737","reported_seq":15495,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:09.092845+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:09.092845+0000","last_peered":"2024-09-16T04:50:09.092845+0000","last_clean":"2024-09-16T04:50:09.092845+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:09.092845+0000","last_undegraded":"2024-09-16T04:50:09.092845+0000","last_fullsized":"2024-09-16T04:50:09.092845+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:09.302829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16534040,"num_objects":2019,"num_object_clones":0,"num_object_copies":8076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10734,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10633","reported_seq":10740,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:07.544988+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:07.544988+0000","last_peered":"2024-09-16T04:50:07.544988+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:07.544988+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17405196,"num_objects":2125,"num_object_clones":0,"num_object_copies":8500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10631,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-64}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10474","reported_seq":10607,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:09.077550+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:09.077550+0000","last_peered":"2024-09-16T04:50:09.077550+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:09.077550+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16519058,"num_objects":2017,"num_object_clones":0,"num_object_copies":8068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10471,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10387","reported_seq":10480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:09.039126+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:09.039126+0000","last_peered":"2024-09-16T04:50:09.039126+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:09.039126+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15923846,"num_objects":1944,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10386,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-161}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10498","reported_seq":10586,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:07.802283+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:07.802283+0000","last_peered":"2024-09-16T04:50:07.802283+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:07.802283+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16669318,"num_objects":2035,"num_object_clones":0,"num_object_copies":8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10497,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-186}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10713","reported_seq":10846,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:09.161428+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:09.161428+0000","last_peered":"2024-09-16T04:50:09.161428+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:09.161428+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16946444,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10711,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-129}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10376","reported_seq":10479,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:07.763234+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:07.763234+0000","last_peered":"2024-09-16T04:50:07.763234+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:07.763234+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16979212,"num_objects":2073,"num_object_clones":0,"num_object_copies":8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10374,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-50}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10375","reported_seq":10470,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:08.768133+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:08.768133+0000","last_peered":"2024-09-16T04:50:08.768133+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:08.768133+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15824140,"num_objects":1932,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10373,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-296}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":267175616,"num_objects":32620,"num_object_clones":0,"num_object_copies":130480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32620,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168867,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6425,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":536506368,"data_stored":536506368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160894,"ondisk_log_size":160894,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744420,"kb_used_data":131028,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93627420,"statfs":{"total":96636764160,"available":95874478080,"internally_reserved":0,"allocated":134172672,"data_stored":133983470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628092623},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,0,5,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758312,"kb_used_data":131544,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93613528,"statfs":{"total":96636764160,"available":95860252672,"internally_reserved":0,"allocated":134701056,"data_stored":134511854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602164,"kb_used_data":133156,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93769676,"statfs":{"total":96636764160,"available":96020148224,"internally_reserved":0,"allocated":136351744,"data_stored":136162542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748176,"kb_used_data":130680,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93623664,"statfs":{"total":96636764160,"available":95870631936,"internally_reserved":0,"allocated":133816320,"data_stored":133627118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":632286926},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,2,9,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":133345280,"data_stored":133345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135921664,"data_stored":135921664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":134156288,"data_stored":134156288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":133083136,"data_stored":133083136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:12.306 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:12.932 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:12.932 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:12.965 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-16T04:50:12.735494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":266618560,"num_objects":32552,"num_object_clones":0,"num_object_copies":130208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32552,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168935,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6425,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160862,"ondisk_log_size":160862,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2849112,"kb_used_data":522432,"kb_used_omap":14,"kb_used_meta":2326577,"kb_avail":374638248,"statfs":{"total":386547056640,"available":383629565952,"internally_reserved":0,"allocated":534970368,"data_stored":534213560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2382415409},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,4,7,10,14,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16629760,"num_objects":-2030,"num_object_clones":0,"num_object_copies":-8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2030,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004145"},"pg_stats":[{"pgid":"1.f","version":"35'10597","reported_seq":10711,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:09.210121+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:09.210121+0000","last_peered":"2024-09-16T04:50:09.210121+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:09.210121+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16850944,"num_objects":2057,"num_object_clones":0,"num_object_copies":8228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10597,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10632","reported_seq":10720,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:08.950747+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:08.950747+0000","last_peered":"2024-09-16T04:50:08.950747+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:08.950747+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16725260,"num_objects":2042,"num_object_clones":0,"num_object_copies":8168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10630,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-161}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10595","reported_seq":10694,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:07.823069+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:07.823069+0000","last_peered":"2024-09-16T04:50:07.823069+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:07.823069+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16997022,"num_objects":2076,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10594,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-143}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10611","reported_seq":13805,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:08.862037+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:08.862037+0000","last_peered":"2024-09-16T04:50:08.862037+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:08.862037+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16864524,"num_objects":2059,"num_object_clones":0,"num_object_copies":8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":671,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10609,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1562,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1388}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10652","reported_seq":10785,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:08.981344+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:08.981344+0000","last_peered":"2024-09-16T04:50:08.981344+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:08.981344+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17110284,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10650,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-159}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10367","reported_seq":10478,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:09.202930+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:09.202930+0000","last_peered":"2024-09-16T04:50:09.202930+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:09.202930+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15933440,"num_objects":1945,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10367,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10613","reported_seq":10709,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:08.779747+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:08.779747+0000","last_peered":"2024-09-16T04:50:08.779747+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:08.779747+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16274700,"num_objects":1987,"num_object_clones":0,"num_object_copies":7948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10611,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-226}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10702","reported_seq":15264,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:11.358593+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:11.358593+0000","last_peered":"2024-09-16T04:50:11.358593+0000","last_clean":"2024-09-16T04:50:11.358593+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:11.358593+0000","last_undegraded":"2024-09-16T04:50:11.358593+0000","last_fullsized":"2024-09-16T04:50:11.358593+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:14.165380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17061132,"num_objects":2083,"num_object_clones":0,"num_object_copies":8332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10737","reported_seq":15495,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:09.092845+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:09.092845+0000","last_peered":"2024-09-16T04:50:09.092845+0000","last_clean":"2024-09-16T04:50:09.092845+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:09.092845+0000","last_undegraded":"2024-09-16T04:50:09.092845+0000","last_fullsized":"2024-09-16T04:50:09.092845+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:09.302829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16534040,"num_objects":2019,"num_object_clones":0,"num_object_copies":8076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10734,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10633","reported_seq":10740,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:07.544988+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:07.544988+0000","last_peered":"2024-09-16T04:50:07.544988+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:07.544988+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17405196,"num_objects":2125,"num_object_clones":0,"num_object_copies":8500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10631,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-64}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10474","reported_seq":10607,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:09.077550+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:09.077550+0000","last_peered":"2024-09-16T04:50:09.077550+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:09.077550+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16519058,"num_objects":2017,"num_object_clones":0,"num_object_copies":8068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10471,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10387","reported_seq":10480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:09.039126+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:09.039126+0000","last_peered":"2024-09-16T04:50:09.039126+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:09.039126+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15923846,"num_objects":1944,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10386,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-161}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10498","reported_seq":10586,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:07.802283+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:07.802283+0000","last_peered":"2024-09-16T04:50:07.802283+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:07.802283+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16669318,"num_objects":2035,"num_object_clones":0,"num_object_copies":8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10497,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-186}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10713","reported_seq":10846,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:09.161428+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:09.161428+0000","last_peered":"2024-09-16T04:50:09.161428+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:09.161428+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16946444,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10711,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-129}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10376","reported_seq":10479,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:07.763234+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:07.763234+0000","last_peered":"2024-09-16T04:50:07.763234+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:07.763234+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16979212,"num_objects":2073,"num_object_clones":0,"num_object_copies":8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10374,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-50}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10375","reported_seq":10470,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:08.768133+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:08.768133+0000","last_peered":"2024-09-16T04:50:08.768133+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:08.768133+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15824140,"num_objects":1932,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10373,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-296}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":266618560,"num_objects":32552,"num_object_clones":0,"num_object_copies":130208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32552,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168935,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6425,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532332544,"data_stored":532332544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160862,"ondisk_log_size":160862,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744420,"kb_used_data":131028,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93627420,"statfs":{"total":96636764160,"available":95874478080,"internally_reserved":0,"allocated":134172672,"data_stored":133983470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628092623},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,0,5,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758312,"kb_used_data":131544,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93613528,"statfs":{"total":96636764160,"available":95860252672,"internally_reserved":0,"allocated":134701056,"data_stored":134511854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598204,"kb_used_data":129180,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93773636,"statfs":{"total":96636764160,"available":96024203264,"internally_reserved":0,"allocated":132280320,"data_stored":132091118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748176,"kb_used_data":130680,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93623664,"statfs":{"total":96636764160,"available":95870631936,"internally_reserved":0,"allocated":133816320,"data_stored":133627118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":632286926},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,2,9,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":133345280,"data_stored":133345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":131747840,"data_stored":131747840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":134156288,"data_stored":134156288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":133083136,"data_stored":133083136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:12.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:50:12.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10597", 'reported_seq': 10711, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:09.210121+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:50:09.210121+0000', 'last_peered': '2024-09-16T04:50:09.210121+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:50:09.210121+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16850944, 'num_objects': 2057, 'num_object_clones': 0, 'num_object_copies': 8228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2057, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10597, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -129}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:12.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:50:12.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10632", 'reported_seq': 10720, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:08.950747+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:50:08.950747+0000', 'last_peered': '2024-09-16T04:50:08.950747+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:50:08.950747+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16725260, 'num_objects': 2042, 'num_object_clones': 0, 'num_object_copies': 8168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2042, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10630, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -161}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:12.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:50:12.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10595", 'reported_seq': 10694, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:07.823069+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:50:07.823069+0000', 'last_peered': '2024-09-16T04:50:07.823069+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:50:07.823069+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16997022, 'num_objects': 2076, 'num_object_clones': 0, 'num_object_copies': 8304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2076, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10594, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -143}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:12.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:50:12.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10611", 'reported_seq': 13805, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:08.862037+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:50:08.862037+0000', 'last_peered': '2024-09-16T04:50:08.862037+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:50:08.862037+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16864524, 'num_objects': 2059, 'num_object_clones': 0, 'num_object_copies': 8236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 671, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10609, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1562, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1388}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:12.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:50:12.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10652", 'reported_seq': 10785, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:08.981344+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:50:08.981344+0000', 'last_peered': '2024-09-16T04:50:08.981344+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:50:08.981344+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17110284, 'num_objects': 2089, 'num_object_clones': 0, 'num_object_copies': 8356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2089, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10650, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -159}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:12.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:50:12.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10367", 'reported_seq': 10478, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:09.202930+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:50:09.202930+0000', 'last_peered': '2024-09-16T04:50:09.202930+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:50:09.202930+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15933440, 'num_objects': 1945, 'num_object_clones': 0, 'num_object_copies': 7780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1945, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10367, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -233}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:12.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:50:12.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10613", 'reported_seq': 10709, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:08.779747+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:50:08.779747+0000', 'last_peered': '2024-09-16T04:50:08.779747+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:50:08.779747+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16274700, 'num_objects': 1987, 'num_object_clones': 0, 'num_object_copies': 7948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1987, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10611, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -226}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:12.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:50:12.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10633", 'reported_seq': 10740, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:07.544988+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:50:07.544988+0000', 'last_peered': '2024-09-16T04:50:07.544988+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:50:07.544988+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17405196, 'num_objects': 2125, 'num_object_clones': 0, 'num_object_copies': 8500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2125, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10631, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -64}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:12.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:50:12.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10474", 'reported_seq': 10607, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:09.077550+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:50:09.077550+0000', 'last_peered': '2024-09-16T04:50:09.077550+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:50:09.077550+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16519058, 'num_objects': 2017, 'num_object_clones': 0, 'num_object_copies': 8068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2017, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10471, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -138}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:12.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:50:12.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10387", 'reported_seq': 10480, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:09.039126+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:50:09.039126+0000', 'last_peered': '2024-09-16T04:50:09.039126+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:50:09.039126+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15923846, 'num_objects': 1944, 'num_object_clones': 0, 'num_object_copies': 7776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1944, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10386, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -161}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:12.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:50:12.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10498", 'reported_seq': 10586, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:07.802283+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:50:07.802283+0000', 'last_peered': '2024-09-16T04:50:07.802283+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:50:07.802283+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16669318, 'num_objects': 2035, 'num_object_clones': 0, 'num_object_copies': 8140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2035, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10497, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -186}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:12.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:50:12.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10713", 'reported_seq': 10846, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:09.161428+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:50:09.161428+0000', 'last_peered': '2024-09-16T04:50:09.161428+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:50:09.161428+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16946444, 'num_objects': 2069, 'num_object_clones': 0, 'num_object_copies': 8276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2069, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10711, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -129}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:12.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:50:12.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10376", 'reported_seq': 10479, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:07.763234+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:50:07.763234+0000', 'last_peered': '2024-09-16T04:50:07.763234+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:50:07.763234+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16979212, 'num_objects': 2073, 'num_object_clones': 0, 'num_object_copies': 8292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2073, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10374, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -50}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:12.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:50:12.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10375", 'reported_seq': 10470, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:08.768133+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:50:08.768133+0000', 'last_peered': '2024-09-16T04:50:08.768133+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:50:08.768133+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15824140, 'num_objects': 1932, 'num_object_clones': 0, 'num_object_copies': 7728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10373, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -296}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:12.969 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:50:13.702 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:13.702 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29157/130208 objects degraded (22.393%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":626,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":32552,"data_bytes":266618560,"bytes_used":2917490688,"bytes_avail":383629565952,"bytes_total":386547056640,"degraded_objects":29157,"degraded_total":130208,"degraded_ratio":0.22392633325141312,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":169},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:50:13.738 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:50:13.738 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:14.364 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:14.365 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:14.385 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-16T04:50:12.735494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":266618560,"num_objects":32552,"num_object_clones":0,"num_object_copies":130208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32552,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168935,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6425,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160862,"ondisk_log_size":160862,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2849112,"kb_used_data":522432,"kb_used_omap":14,"kb_used_meta":2326577,"kb_avail":374638248,"statfs":{"total":386547056640,"available":383629565952,"internally_reserved":0,"allocated":534970368,"data_stored":534213560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2382415409},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,4,7,10,14,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16629760,"num_objects":-2030,"num_object_clones":0,"num_object_copies":-8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2030,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004145"},"pg_stats":[{"pgid":"1.f","version":"35'10597","reported_seq":10711,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:09.210121+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:09.210121+0000","last_peered":"2024-09-16T04:50:09.210121+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:09.210121+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16850944,"num_objects":2057,"num_object_clones":0,"num_object_copies":8228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10597,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10632","reported_seq":10720,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:08.950747+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:08.950747+0000","last_peered":"2024-09-16T04:50:08.950747+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:08.950747+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16725260,"num_objects":2042,"num_object_clones":0,"num_object_copies":8168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10630,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-161}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10595","reported_seq":10694,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:07.823069+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:07.823069+0000","last_peered":"2024-09-16T04:50:07.823069+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:07.823069+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16997022,"num_objects":2076,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10594,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-143}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10611","reported_seq":13805,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:08.862037+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:08.862037+0000","last_peered":"2024-09-16T04:50:08.862037+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:08.862037+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16864524,"num_objects":2059,"num_object_clones":0,"num_object_copies":8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":671,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10609,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1562,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1388}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10652","reported_seq":10785,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:08.981344+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:08.981344+0000","last_peered":"2024-09-16T04:50:08.981344+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:08.981344+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17110284,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10650,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-159}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10367","reported_seq":10478,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:09.202930+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:09.202930+0000","last_peered":"2024-09-16T04:50:09.202930+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:09.202930+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15933440,"num_objects":1945,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10367,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10613","reported_seq":10709,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:08.779747+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:08.779747+0000","last_peered":"2024-09-16T04:50:08.779747+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:08.779747+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16274700,"num_objects":1987,"num_object_clones":0,"num_object_copies":7948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10611,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-226}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10702","reported_seq":15264,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:11.358593+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:11.358593+0000","last_peered":"2024-09-16T04:50:11.358593+0000","last_clean":"2024-09-16T04:50:11.358593+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:11.358593+0000","last_undegraded":"2024-09-16T04:50:11.358593+0000","last_fullsized":"2024-09-16T04:50:11.358593+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:14.165380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17061132,"num_objects":2083,"num_object_clones":0,"num_object_copies":8332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10737","reported_seq":15495,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:09.092845+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:09.092845+0000","last_peered":"2024-09-16T04:50:09.092845+0000","last_clean":"2024-09-16T04:50:09.092845+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:09.092845+0000","last_undegraded":"2024-09-16T04:50:09.092845+0000","last_fullsized":"2024-09-16T04:50:09.092845+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:09.302829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16534040,"num_objects":2019,"num_object_clones":0,"num_object_copies":8076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10734,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10633","reported_seq":10740,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:07.544988+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:07.544988+0000","last_peered":"2024-09-16T04:50:07.544988+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:07.544988+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17405196,"num_objects":2125,"num_object_clones":0,"num_object_copies":8500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10631,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-64}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10474","reported_seq":10607,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:09.077550+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:09.077550+0000","last_peered":"2024-09-16T04:50:09.077550+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:09.077550+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16519058,"num_objects":2017,"num_object_clones":0,"num_object_copies":8068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10471,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10387","reported_seq":10480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:09.039126+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:09.039126+0000","last_peered":"2024-09-16T04:50:09.039126+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:09.039126+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15923846,"num_objects":1944,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10386,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-161}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10498","reported_seq":10586,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:07.802283+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:07.802283+0000","last_peered":"2024-09-16T04:50:07.802283+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:07.802283+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16669318,"num_objects":2035,"num_object_clones":0,"num_object_copies":8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10497,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-186}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10713","reported_seq":10846,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:09.161428+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:09.161428+0000","last_peered":"2024-09-16T04:50:09.161428+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:09.161428+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16946444,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10711,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-129}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10376","reported_seq":10479,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:07.763234+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:07.763234+0000","last_peered":"2024-09-16T04:50:07.763234+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:07.763234+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16979212,"num_objects":2073,"num_object_clones":0,"num_object_copies":8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10374,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-50}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10375","reported_seq":10470,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:08.768133+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:08.768133+0000","last_peered":"2024-09-16T04:50:08.768133+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:08.768133+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15824140,"num_objects":1932,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10373,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-296}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":266618560,"num_objects":32552,"num_object_clones":0,"num_object_copies":130208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32552,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168935,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6425,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532332544,"data_stored":532332544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160862,"ondisk_log_size":160862,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744420,"kb_used_data":131028,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93627420,"statfs":{"total":96636764160,"available":95874478080,"internally_reserved":0,"allocated":134172672,"data_stored":133983470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628092623},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,0,5,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758312,"kb_used_data":131544,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93613528,"statfs":{"total":96636764160,"available":95860252672,"internally_reserved":0,"allocated":134701056,"data_stored":134511854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598204,"kb_used_data":129180,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93773636,"statfs":{"total":96636764160,"available":96024203264,"internally_reserved":0,"allocated":132280320,"data_stored":132091118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480243407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748176,"kb_used_data":130680,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93623664,"statfs":{"total":96636764160,"available":95870631936,"internally_reserved":0,"allocated":133816320,"data_stored":133627118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":632286926},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,2,9,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":133345280,"data_stored":133345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":131747840,"data_stored":131747840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":134156288,"data_stored":134156288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":133083136,"data_stored":133083136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:17.386 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:18.010 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:18.011 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:18.035 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-09-16T04:50:16.737711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":258590400,"num_objects":31572,"num_object_clones":0,"num_object_copies":126288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169915,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6480,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160942,"ondisk_log_size":160942,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2829180,"kb_used_data":506648,"kb_used_omap":14,"kb_used_meta":2322417,"kb_avail":374658180,"statfs":{"total":386547056640,"available":383649976320,"internally_reserved":0,"allocated":518807552,"data_stored":518050744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2378155569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,6,5,4,15,16,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16785408,"num_objects":-2049,"num_object_clones":0,"num_object_copies":-8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2049,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004930"},"pg_stats":[{"pgid":"1.f","version":"35'10651","reported_seq":10768,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.282094+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:14.282094+0000","last_peered":"2024-09-16T04:50:14.282094+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:14.282094+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16408576,"num_objects":2003,"num_object_clones":0,"num_object_copies":8012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10651,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10695","reported_seq":10784,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:13.884897+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:13.884897+0000","last_peered":"2024-09-16T04:50:13.884897+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:13.884897+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16209164,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10693,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-224}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10667","reported_seq":10766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:12.876979+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:12.876979+0000","last_peered":"2024-09-16T04:50:12.876979+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:12.876979+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16407198,"num_objects":2004,"num_object_clones":0,"num_object_copies":8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2004,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10666,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-215}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10678","reported_seq":13981,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.091597+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:14.091597+0000","last_peered":"2024-09-16T04:50:14.091597+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:14.091597+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16315660,"num_objects":1992,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10676,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10716","reported_seq":10849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.014261+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:14.014261+0000","last_peered":"2024-09-16T04:50:14.014261+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:14.014261+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16585996,"num_objects":2025,"num_object_clones":0,"num_object_copies":8100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10714,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-223}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10424","reported_seq":10535,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.246898+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:14.246898+0000","last_peered":"2024-09-16T04:50:14.246898+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:14.246898+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15466496,"num_objects":1888,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10424,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10674","reported_seq":10772,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.036227+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:14.036227+0000","last_peered":"2024-09-16T04:50:14.036227+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:14.036227+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15774988,"num_objects":1926,"num_object_clones":0,"num_object_copies":7704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10672,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-287}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10762","reported_seq":15324,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:16.449054+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:16.449054+0000","last_peered":"2024-09-16T04:50:16.449054+0000","last_clean":"2024-09-16T04:50:16.449054+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:16.449054+0000","last_undegraded":"2024-09-16T04:50:16.449054+0000","last_fullsized":"2024-09-16T04:50:16.449054+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:20.187355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16569612,"num_objects":2023,"num_object_clones":0,"num_object_copies":8092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10760,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10794","reported_seq":15552,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:14.049461+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:14.049461+0000","last_peered":"2024-09-16T04:50:14.049461+0000","last_clean":"2024-09-16T04:50:14.049461+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:14.049461+0000","last_undegraded":"2024-09-16T04:50:14.049461+0000","last_fullsized":"2024-09-16T04:50:14.049461+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:14.359032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16067096,"num_objects":1962,"num_object_clones":0,"num_object_copies":7848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10791,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10686","reported_seq":10793,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:12.869505+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:12.869505+0000","last_peered":"2024-09-16T04:50:12.869505+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:12.869505+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16971020,"num_objects":2072,"num_object_clones":0,"num_object_copies":8288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10684,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-117}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10543","reported_seq":10676,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.209880+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:14.209880+0000","last_peered":"2024-09-16T04:50:14.209880+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:14.209880+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15953810,"num_objects":1948,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10540,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10449","reported_seq":10541,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.283469+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:14.283469+0000","last_peered":"2024-09-16T04:50:14.283469+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:14.283469+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15415942,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10448,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-223}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10558","reported_seq":10647,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:12.832427+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:12.832427+0000","last_peered":"2024-09-16T04:50:12.832427+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:12.832427+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16177798,"num_objects":1975,"num_object_clones":0,"num_object_copies":7900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10557,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-246}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10764","reported_seq":10897,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.157188+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:14.157188+0000","last_peered":"2024-09-16T04:50:14.157188+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:14.157188+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16528652,"num_objects":2018,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10762,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-180}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10438","reported_seq":10542,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:12.670246+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:12.670246+0000","last_peered":"2024-09-16T04:50:12.670246+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:12.670246+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16471308,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10436,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-112}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10443","reported_seq":10537,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.033414+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:14.033414+0000","last_peered":"2024-09-16T04:50:14.033414+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:14.033414+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15267084,"num_objects":1864,"num_object_clones":0,"num_object_copies":7456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10441,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":258590400,"num_objects":31572,"num_object_clones":0,"num_object_copies":126288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169915,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6480,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516100096,"data_stored":516100096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160942,"ondisk_log_size":160942,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744584,"kb_used_data":127000,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93627256,"statfs":{"total":96636764160,"available":95874310144,"internally_reserved":0,"allocated":130048000,"data_stored":129858798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,0,5,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754340,"kb_used_data":127572,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93617500,"statfs":{"total":96636764160,"available":95864320000,"internally_reserved":0,"allocated":130633728,"data_stored":130444526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598544,"kb_used_data":125376,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93773296,"statfs":{"total":96636764160,"available":96023855104,"internally_reserved":0,"allocated":128385024,"data_stored":128195822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,3,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247928,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731712,"kb_used_data":126700,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93640128,"statfs":{"total":96636764160,"available":95887491072,"internally_reserved":0,"allocated":129740800,"data_stored":129551598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":619507406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,0,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":129216512,"data_stored":129216512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":127766528,"data_stored":127766528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130076672,"data_stored":130076672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129040384,"data_stored":129040384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:18.036 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:18.667 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:18.668 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:18.686 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-09-16T04:50:16.737711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":258590400,"num_objects":31572,"num_object_clones":0,"num_object_copies":126288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169915,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6480,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160942,"ondisk_log_size":160942,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2829180,"kb_used_data":506648,"kb_used_omap":14,"kb_used_meta":2322417,"kb_avail":374658180,"statfs":{"total":386547056640,"available":383649976320,"internally_reserved":0,"allocated":518807552,"data_stored":518050744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2378155569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,6,5,4,15,16,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16785408,"num_objects":-2049,"num_object_clones":0,"num_object_copies":-8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2049,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004930"},"pg_stats":[{"pgid":"1.f","version":"35'10651","reported_seq":10768,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.282094+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:14.282094+0000","last_peered":"2024-09-16T04:50:14.282094+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:14.282094+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16408576,"num_objects":2003,"num_object_clones":0,"num_object_copies":8012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10651,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10695","reported_seq":10784,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:13.884897+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:13.884897+0000","last_peered":"2024-09-16T04:50:13.884897+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:13.884897+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16209164,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10693,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-224}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10667","reported_seq":10766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:12.876979+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:12.876979+0000","last_peered":"2024-09-16T04:50:12.876979+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:12.876979+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16407198,"num_objects":2004,"num_object_clones":0,"num_object_copies":8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2004,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10666,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-215}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10678","reported_seq":13981,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.091597+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:14.091597+0000","last_peered":"2024-09-16T04:50:14.091597+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:14.091597+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16315660,"num_objects":1992,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10676,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10716","reported_seq":10849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.014261+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:14.014261+0000","last_peered":"2024-09-16T04:50:14.014261+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:14.014261+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16585996,"num_objects":2025,"num_object_clones":0,"num_object_copies":8100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10714,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-223}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10424","reported_seq":10535,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.246898+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:14.246898+0000","last_peered":"2024-09-16T04:50:14.246898+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:14.246898+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15466496,"num_objects":1888,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10424,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10674","reported_seq":10772,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.036227+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:14.036227+0000","last_peered":"2024-09-16T04:50:14.036227+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:14.036227+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15774988,"num_objects":1926,"num_object_clones":0,"num_object_copies":7704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10672,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-287}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10762","reported_seq":15324,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:16.449054+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:16.449054+0000","last_peered":"2024-09-16T04:50:16.449054+0000","last_clean":"2024-09-16T04:50:16.449054+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:16.449054+0000","last_undegraded":"2024-09-16T04:50:16.449054+0000","last_fullsized":"2024-09-16T04:50:16.449054+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:20.187355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16569612,"num_objects":2023,"num_object_clones":0,"num_object_copies":8092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10760,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10794","reported_seq":15552,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:14.049461+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:14.049461+0000","last_peered":"2024-09-16T04:50:14.049461+0000","last_clean":"2024-09-16T04:50:14.049461+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:14.049461+0000","last_undegraded":"2024-09-16T04:50:14.049461+0000","last_fullsized":"2024-09-16T04:50:14.049461+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:14.359032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16067096,"num_objects":1962,"num_object_clones":0,"num_object_copies":7848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10791,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10686","reported_seq":10793,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:12.869505+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:12.869505+0000","last_peered":"2024-09-16T04:50:12.869505+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:12.869505+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16971020,"num_objects":2072,"num_object_clones":0,"num_object_copies":8288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10684,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-117}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10543","reported_seq":10676,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.209880+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:14.209880+0000","last_peered":"2024-09-16T04:50:14.209880+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:14.209880+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15953810,"num_objects":1948,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10540,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10449","reported_seq":10541,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.283469+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:14.283469+0000","last_peered":"2024-09-16T04:50:14.283469+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:14.283469+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15415942,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10448,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-223}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10558","reported_seq":10647,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:12.832427+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:12.832427+0000","last_peered":"2024-09-16T04:50:12.832427+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:12.832427+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16177798,"num_objects":1975,"num_object_clones":0,"num_object_copies":7900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10557,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-246}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10764","reported_seq":10897,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.157188+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:14.157188+0000","last_peered":"2024-09-16T04:50:14.157188+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:14.157188+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16528652,"num_objects":2018,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10762,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-180}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10438","reported_seq":10542,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:12.670246+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:12.670246+0000","last_peered":"2024-09-16T04:50:12.670246+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:12.670246+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16471308,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10436,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-112}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10443","reported_seq":10537,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.033414+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:14.033414+0000","last_peered":"2024-09-16T04:50:14.033414+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:14.033414+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15267084,"num_objects":1864,"num_object_clones":0,"num_object_copies":7456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10441,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":258590400,"num_objects":31572,"num_object_clones":0,"num_object_copies":126288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169915,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6480,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516100096,"data_stored":516100096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160942,"ondisk_log_size":160942,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744584,"kb_used_data":127000,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93627256,"statfs":{"total":96636764160,"available":95874310144,"internally_reserved":0,"allocated":130048000,"data_stored":129858798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,0,5,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754340,"kb_used_data":127572,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93617500,"statfs":{"total":96636764160,"available":95864320000,"internally_reserved":0,"allocated":130633728,"data_stored":130444526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598544,"kb_used_data":125376,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93773296,"statfs":{"total":96636764160,"available":96023855104,"internally_reserved":0,"allocated":128385024,"data_stored":128195822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,3,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247928,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731712,"kb_used_data":126700,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93640128,"statfs":{"total":96636764160,"available":95887491072,"internally_reserved":0,"allocated":129740800,"data_stored":129551598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":619507406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,0,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":129216512,"data_stored":129216512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":127766528,"data_stored":127766528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130076672,"data_stored":130076672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129040384,"data_stored":129040384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:18.687 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:50:18.687 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10651", 'reported_seq': 10768, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:14.282094+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:50:14.282094+0000', 'last_peered': '2024-09-16T04:50:14.282094+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:50:14.282094+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16408576, 'num_objects': 2003, 'num_object_clones': 0, 'num_object_copies': 8012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2003, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10651, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:18.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:50:18.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10695", 'reported_seq': 10784, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:13.884897+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:50:13.884897+0000', 'last_peered': '2024-09-16T04:50:13.884897+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:50:13.884897+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16209164, 'num_objects': 1979, 'num_object_clones': 0, 'num_object_copies': 7916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10693, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -224}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:18.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:50:18.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10667", 'reported_seq': 10766, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:12.876979+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:50:12.876979+0000', 'last_peered': '2024-09-16T04:50:12.876979+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:50:12.876979+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16407198, 'num_objects': 2004, 'num_object_clones': 0, 'num_object_copies': 8016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2004, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10666, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -215}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:18.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:50:18.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10678", 'reported_seq': 13981, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:14.091597+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:50:14.091597+0000', 'last_peered': '2024-09-16T04:50:14.091597+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:50:14.091597+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16315660, 'num_objects': 1992, 'num_object_clones': 0, 'num_object_copies': 7968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 616, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10676, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1617, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1376}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:18.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:50:18.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10716", 'reported_seq': 10849, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:14.014261+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:50:14.014261+0000', 'last_peered': '2024-09-16T04:50:14.014261+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:50:14.014261+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16585996, 'num_objects': 2025, 'num_object_clones': 0, 'num_object_copies': 8100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10714, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -223}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:18.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:50:18.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10424", 'reported_seq': 10535, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:14.246898+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:50:14.246898+0000', 'last_peered': '2024-09-16T04:50:14.246898+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:50:14.246898+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15466496, 'num_objects': 1888, 'num_object_clones': 0, 'num_object_copies': 7552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1888, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10424, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -290}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:18.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:50:18.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10674", 'reported_seq': 10772, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:14.036227+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:50:14.036227+0000', 'last_peered': '2024-09-16T04:50:14.036227+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:50:14.036227+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15774988, 'num_objects': 1926, 'num_object_clones': 0, 'num_object_copies': 7704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10672, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -287}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:18.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:50:18.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10686", 'reported_seq': 10793, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:12.869505+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:50:12.869505+0000', 'last_peered': '2024-09-16T04:50:12.869505+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:50:12.869505+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16971020, 'num_objects': 2072, 'num_object_clones': 0, 'num_object_copies': 8288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2072, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10684, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -117}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:18.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:50:18.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10543", 'reported_seq': 10676, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:14.209880+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:50:14.209880+0000', 'last_peered': '2024-09-16T04:50:14.209880+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:50:14.209880+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15953810, 'num_objects': 1948, 'num_object_clones': 0, 'num_object_copies': 7792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10540, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -207}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:18.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:50:18.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10449", 'reported_seq': 10541, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:14.283469+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:50:14.283469+0000', 'last_peered': '2024-09-16T04:50:14.283469+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:50:14.283469+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15415942, 'num_objects': 1882, 'num_object_clones': 0, 'num_object_copies': 7528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1882, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10448, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -223}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:18.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:50:18.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10558", 'reported_seq': 10647, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:12.832427+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:50:12.832427+0000', 'last_peered': '2024-09-16T04:50:12.832427+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:50:12.832427+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16177798, 'num_objects': 1975, 'num_object_clones': 0, 'num_object_copies': 7900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10557, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -246}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:18.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:50:18.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10764", 'reported_seq': 10897, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:14.157188+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:50:14.157188+0000', 'last_peered': '2024-09-16T04:50:14.157188+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:50:14.157188+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16528652, 'num_objects': 2018, 'num_object_clones': 0, 'num_object_copies': 8072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10762, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -180}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:18.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:50:18.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10438", 'reported_seq': 10542, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:12.670246+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:50:12.670246+0000', 'last_peered': '2024-09-16T04:50:12.670246+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:50:12.670246+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16471308, 'num_objects': 2011, 'num_object_clones': 0, 'num_object_copies': 8044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10436, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -112}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:18.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:50:18.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10443", 'reported_seq': 10537, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:14.033414+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:50:14.033414+0000', 'last_peered': '2024-09-16T04:50:14.033414+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:50:14.033414+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15267084, 'num_objects': 1864, 'num_object_clones': 0, 'num_object_copies': 7456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1864, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10441, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -364}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:18.692 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:50:19.475 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:19.476 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29102/125192 objects degraded (23.246%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":631,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":31298,"data_bytes":256345792,"bytes_used":2897260544,"bytes_avail":383649796096,"bytes_total":386547056640,"degraded_objects":29102,"degraded_total":125192,"degraded_ratio":0.23245894306345452,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":188},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:50:19.513 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:50:19.513 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:20.089 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:20.089 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:20.122 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-16T04:50:18.738482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":256345792,"num_objects":31298,"num_object_clones":0,"num_object_copies":125192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31298,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170189,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6480,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160816,"ondisk_log_size":160816,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2829356,"kb_used_data":502664,"kb_used_omap":14,"kb_used_meta":2326577,"kb_avail":374658004,"statfs":{"total":386547056640,"available":383649796096,"internally_reserved":0,"allocated":514727936,"data_stored":513971128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2382415409},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,5,7,4,14,14,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18513920,"num_objects":-2260,"num_object_clones":0,"num_object_copies":-9040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2260,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004902"},"pg_stats":[{"pgid":"1.f","version":"35'10651","reported_seq":10768,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.282094+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:14.282094+0000","last_peered":"2024-09-16T04:50:14.282094+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:14.282094+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16408576,"num_objects":2003,"num_object_clones":0,"num_object_copies":8012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10651,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10695","reported_seq":10784,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:13.884897+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:13.884897+0000","last_peered":"2024-09-16T04:50:13.884897+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:13.884897+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16209164,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10693,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-224}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10739","reported_seq":10838,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:17.832115+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:17.832115+0000","last_peered":"2024-09-16T04:50:17.832115+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:17.832115+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15817374,"num_objects":1932,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10738,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10678","reported_seq":13981,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.091597+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:14.091597+0000","last_peered":"2024-09-16T04:50:14.091597+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:14.091597+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16315660,"num_objects":1992,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10676,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10716","reported_seq":10849,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.014261+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:14.014261+0000","last_peered":"2024-09-16T04:50:14.014261+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:14.014261+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16585996,"num_objects":2025,"num_object_clones":0,"num_object_copies":8100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10714,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-223}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10424","reported_seq":10535,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.246898+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:14.246898+0000","last_peered":"2024-09-16T04:50:14.246898+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:14.246898+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15466496,"num_objects":1888,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10424,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10674","reported_seq":10772,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.036227+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:14.036227+0000","last_peered":"2024-09-16T04:50:14.036227+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:14.036227+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15774988,"num_objects":1926,"num_object_clones":0,"num_object_copies":7704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10672,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-287}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10762","reported_seq":15324,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:16.449054+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:16.449054+0000","last_peered":"2024-09-16T04:50:16.449054+0000","last_clean":"2024-09-16T04:50:16.449054+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:16.449054+0000","last_undegraded":"2024-09-16T04:50:16.449054+0000","last_fullsized":"2024-09-16T04:50:16.449054+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:20.187355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16569612,"num_objects":2023,"num_object_clones":0,"num_object_copies":8092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10760,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10794","reported_seq":15552,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:14.049461+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:14.049461+0000","last_peered":"2024-09-16T04:50:14.049461+0000","last_clean":"2024-09-16T04:50:14.049461+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:14.049461+0000","last_undegraded":"2024-09-16T04:50:14.049461+0000","last_fullsized":"2024-09-16T04:50:14.049461+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:14.359032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16067096,"num_objects":1962,"num_object_clones":0,"num_object_copies":7848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10791,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10754","reported_seq":10860,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:18.019498+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:18.019498+0000","last_peered":"2024-09-16T04:50:18.019498+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:18.019498+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16413964,"num_objects":2004,"num_object_clones":0,"num_object_copies":8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10752,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-185}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10543","reported_seq":10676,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.209880+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:14.209880+0000","last_peered":"2024-09-16T04:50:14.209880+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:14.209880+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15953810,"num_objects":1948,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10540,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10449","reported_seq":10541,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.283469+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:14.283469+0000","last_peered":"2024-09-16T04:50:14.283469+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:14.283469+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15415942,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10448,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-223}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10624","reported_seq":10713,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:17.857900+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:17.857900+0000","last_peered":"2024-09-16T04:50:17.857900+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:17.857900+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15637126,"num_objects":1909,"num_object_clones":0,"num_object_copies":7636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10623,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-312}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10764","reported_seq":10897,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.157188+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:14.157188+0000","last_peered":"2024-09-16T04:50:14.157188+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:14.157188+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16528652,"num_objects":2018,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10762,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-180}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10506","reported_seq":10610,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:17.765982+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:17.765982+0000","last_peered":"2024-09-16T04:50:17.765982+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:17.765982+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15914252,"num_objects":1943,"num_object_clones":0,"num_object_copies":7772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-180}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10443","reported_seq":10537,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:14.033414+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:14.033414+0000","last_peered":"2024-09-16T04:50:14.033414+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:14.033414+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15267084,"num_objects":1864,"num_object_clones":0,"num_object_copies":7456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10441,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":256345792,"num_objects":31298,"num_object_clones":0,"num_object_copies":125192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31298,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170189,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6480,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512032768,"data_stored":512032768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160816,"ondisk_log_size":160816,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744584,"kb_used_data":127000,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93627256,"statfs":{"total":96636764160,"available":95874310144,"internally_reserved":0,"allocated":130048000,"data_stored":129858798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,0,5,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754516,"kb_used_data":123588,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93617324,"statfs":{"total":96636764160,"available":95864139776,"internally_reserved":0,"allocated":126554112,"data_stored":126364910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,0,3,1,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598544,"kb_used_data":125376,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93773296,"statfs":{"total":96636764160,"available":96023855104,"internally_reserved":0,"allocated":128385024,"data_stored":128195822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484503247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,3,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247928,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731712,"kb_used_data":126700,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93640128,"statfs":{"total":96636764160,"available":95887491072,"internally_reserved":0,"allocated":129740800,"data_stored":129551598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":619507406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,0,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":129216512,"data_stored":129216512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":127766528,"data_stored":127766528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126009344,"data_stored":126009344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129040384,"data_stored":129040384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:23.124 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:23.779 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:23.779 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:23.805 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-09-16T04:50:22.739755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":250382016,"num_objects":30570,"num_object_clones":0,"num_object_copies":122280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30570,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170917,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6545,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160744,"ondisk_log_size":160744,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2796088,"kb_used_data":490224,"kb_used_omap":14,"kb_used_meta":2305777,"kb_avail":374691272,"statfs":{"total":386547056640,"available":383683862528,"internally_reserved":0,"allocated":501989376,"data_stored":501232568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2361116209},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,4,9,8,17,9,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16793600,"num_objects":-2050,"num_object_clones":0,"num_object_copies":-8200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2050,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004772"},"pg_stats":[{"pgid":"1.f","version":"35'10719","reported_seq":10836,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.023025+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:19.023025+0000","last_peered":"2024-09-16T04:50:19.023025+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:19.023025+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15851520,"num_objects":1935,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10719,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-251}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10766","reported_seq":10855,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:18.716303+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:18.716303+0000","last_peered":"2024-09-16T04:50:18.716303+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:18.716303+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15627532,"num_objects":1908,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10764,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-295}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10739","reported_seq":10838,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:17.832115+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:17.832115+0000","last_peered":"2024-09-16T04:50:17.832115+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:17.832115+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15817374,"num_objects":1932,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10738,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10745","reported_seq":14178,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.130016+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:19.130016+0000","last_peered":"2024-09-16T04:50:19.130016+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:19.130016+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15766796,"num_objects":1925,"num_object_clones":0,"num_object_copies":7700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10743,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1682,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1374}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10776","reported_seq":10909,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.292480+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:19.292480+0000","last_peered":"2024-09-16T04:50:19.292480+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:19.292480+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16094476,"num_objects":1965,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-283}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10473","reported_seq":10584,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.467973+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:19.467973+0000","last_peered":"2024-09-16T04:50:19.467973+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:19.467973+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15065088,"num_objects":1839,"num_object_clones":0,"num_object_copies":7356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10473,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10731","reported_seq":10829,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.055015+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:19.055015+0000","last_peered":"2024-09-16T04:50:19.055015+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:19.055015+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15308044,"num_objects":1869,"num_object_clones":0,"num_object_copies":7476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10729,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-344}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10831","reported_seq":15393,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:21.661743+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:21.661743+0000","last_peered":"2024-09-16T04:50:21.661743+0000","last_clean":"2024-09-16T04:50:21.661743+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:21.661743+0000","last_undegraded":"2024-09-16T04:50:21.661743+0000","last_fullsized":"2024-09-16T04:50:21.661743+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:26.187103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16004364,"num_objects":1954,"num_object_clones":0,"num_object_copies":7816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10850","reported_seq":15608,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:19.469827+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:19.469827+0000","last_peered":"2024-09-16T04:50:19.469827+0000","last_clean":"2024-09-16T04:50:19.469827+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:19.469827+0000","last_undegraded":"2024-09-16T04:50:19.469827+0000","last_fullsized":"2024-09-16T04:50:19.469827+0000","mapping_epoch":34,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:20.382014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15608344,"num_objects":1906,"num_object_clones":0,"num_object_copies":7624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10847,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10754","reported_seq":10860,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:18.019498+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:18.019498+0000","last_peered":"2024-09-16T04:50:18.019498+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:18.019498+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16413964,"num_objects":2004,"num_object_clones":0,"num_object_copies":8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10752,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-185}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10597","reported_seq":10730,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.424486+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:19.424486+0000","last_peered":"2024-09-16T04:50:19.424486+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:19.424486+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15511442,"num_objects":1894,"num_object_clones":0,"num_object_copies":7576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10594,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10510","reported_seq":10603,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.278043+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:19.278043+0000","last_peered":"2024-09-16T04:50:19.278043+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:19.278043+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14916230,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10509,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-284}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10624","reported_seq":10713,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:17.857900+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:17.857900+0000","last_peered":"2024-09-16T04:50:17.857900+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:17.857900+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15637126,"num_objects":1909,"num_object_clones":0,"num_object_copies":7636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10623,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-312}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10830","reported_seq":10963,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.394747+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:19.394747+0000","last_peered":"2024-09-16T04:50:19.394747+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:19.394747+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15987980,"num_objects":1952,"num_object_clones":0,"num_object_copies":7808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10828,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-246}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10506","reported_seq":10610,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:17.765982+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:17.765982+0000","last_peered":"2024-09-16T04:50:17.765982+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:17.765982+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15914252,"num_objects":1943,"num_object_clones":0,"num_object_copies":7772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-180}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10493","reported_seq":10586,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.021576+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:19.021576+0000","last_peered":"2024-09-16T04:50:19.021576+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:19.021576+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14857484,"num_objects":1814,"num_object_clones":0,"num_object_copies":7256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10491,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-414}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":250382016,"num_objects":30570,"num_object_clones":0,"num_object_copies":122280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30570,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170917,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6545,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499802112,"data_stored":499802112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160744,"ondisk_log_size":160744,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727464,"kb_used_data":122384,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93644376,"statfs":{"total":96636764160,"available":95891841024,"internally_reserved":0,"allocated":125321216,"data_stored":125132014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619572943},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,2,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754516,"kb_used_data":123588,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93617324,"statfs":{"total":96636764160,"available":95864139776,"internally_reserved":0,"allocated":126554112,"data_stored":126364910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,0,3,1,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582192,"kb_used_data":121504,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93789648,"statfs":{"total":96636764160,"available":96040599552,"internally_reserved":0,"allocated":124420096,"data_stored":124230894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471723727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,1,6,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731916,"kb_used_data":122748,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93639924,"statfs":{"total":96636764160,"available":95887282176,"internally_reserved":0,"allocated":125693952,"data_stored":125504750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":623767246},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,5,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125186048,"data_stored":125186048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":123691008,"data_stored":123691008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126009344,"data_stored":126009344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":124915712,"data_stored":124915712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:23.806 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:24.345 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:24.345 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:24.371 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-09-16T04:50:22.739755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":250382016,"num_objects":30570,"num_object_clones":0,"num_object_copies":122280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30570,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170917,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6545,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160744,"ondisk_log_size":160744,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2796088,"kb_used_data":490224,"kb_used_omap":14,"kb_used_meta":2305777,"kb_avail":374691272,"statfs":{"total":386547056640,"available":383683862528,"internally_reserved":0,"allocated":501989376,"data_stored":501232568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2361116209},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,4,9,8,17,9,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16793600,"num_objects":-2050,"num_object_clones":0,"num_object_copies":-8200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2050,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004772"},"pg_stats":[{"pgid":"1.f","version":"35'10719","reported_seq":10836,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.023025+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:19.023025+0000","last_peered":"2024-09-16T04:50:19.023025+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:19.023025+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15851520,"num_objects":1935,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10719,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-251}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10766","reported_seq":10855,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:18.716303+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:18.716303+0000","last_peered":"2024-09-16T04:50:18.716303+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:18.716303+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15627532,"num_objects":1908,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10764,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-295}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10739","reported_seq":10838,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:17.832115+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:17.832115+0000","last_peered":"2024-09-16T04:50:17.832115+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:17.832115+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15817374,"num_objects":1932,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10738,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10745","reported_seq":14178,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.130016+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:19.130016+0000","last_peered":"2024-09-16T04:50:19.130016+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:19.130016+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15766796,"num_objects":1925,"num_object_clones":0,"num_object_copies":7700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10743,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1682,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1374}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10776","reported_seq":10909,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.292480+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:19.292480+0000","last_peered":"2024-09-16T04:50:19.292480+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:19.292480+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16094476,"num_objects":1965,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-283}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10473","reported_seq":10584,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.467973+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:19.467973+0000","last_peered":"2024-09-16T04:50:19.467973+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:19.467973+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15065088,"num_objects":1839,"num_object_clones":0,"num_object_copies":7356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10473,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10731","reported_seq":10829,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.055015+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:19.055015+0000","last_peered":"2024-09-16T04:50:19.055015+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:19.055015+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15308044,"num_objects":1869,"num_object_clones":0,"num_object_copies":7476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10729,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-344}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10831","reported_seq":15393,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:21.661743+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:21.661743+0000","last_peered":"2024-09-16T04:50:21.661743+0000","last_clean":"2024-09-16T04:50:21.661743+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:21.661743+0000","last_undegraded":"2024-09-16T04:50:21.661743+0000","last_fullsized":"2024-09-16T04:50:21.661743+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:26.187103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16004364,"num_objects":1954,"num_object_clones":0,"num_object_copies":7816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10850","reported_seq":15608,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:19.469827+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:19.469827+0000","last_peered":"2024-09-16T04:50:19.469827+0000","last_clean":"2024-09-16T04:50:19.469827+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:19.469827+0000","last_undegraded":"2024-09-16T04:50:19.469827+0000","last_fullsized":"2024-09-16T04:50:19.469827+0000","mapping_epoch":34,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:20.382014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15608344,"num_objects":1906,"num_object_clones":0,"num_object_copies":7624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10847,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10754","reported_seq":10860,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:18.019498+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:18.019498+0000","last_peered":"2024-09-16T04:50:18.019498+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:18.019498+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16413964,"num_objects":2004,"num_object_clones":0,"num_object_copies":8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10752,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-185}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10597","reported_seq":10730,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.424486+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:19.424486+0000","last_peered":"2024-09-16T04:50:19.424486+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:19.424486+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15511442,"num_objects":1894,"num_object_clones":0,"num_object_copies":7576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10594,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10510","reported_seq":10603,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.278043+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:19.278043+0000","last_peered":"2024-09-16T04:50:19.278043+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:19.278043+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14916230,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10509,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-284}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10624","reported_seq":10713,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:17.857900+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:17.857900+0000","last_peered":"2024-09-16T04:50:17.857900+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:17.857900+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15637126,"num_objects":1909,"num_object_clones":0,"num_object_copies":7636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10623,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-312}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10830","reported_seq":10963,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.394747+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:19.394747+0000","last_peered":"2024-09-16T04:50:19.394747+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:19.394747+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15987980,"num_objects":1952,"num_object_clones":0,"num_object_copies":7808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10828,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-246}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10506","reported_seq":10610,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:17.765982+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:17.765982+0000","last_peered":"2024-09-16T04:50:17.765982+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:17.765982+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15914252,"num_objects":1943,"num_object_clones":0,"num_object_copies":7772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-180}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10493","reported_seq":10586,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:19.021576+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:19.021576+0000","last_peered":"2024-09-16T04:50:19.021576+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:19.021576+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14857484,"num_objects":1814,"num_object_clones":0,"num_object_copies":7256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10491,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-414}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":250382016,"num_objects":30570,"num_object_clones":0,"num_object_copies":122280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30570,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170917,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6545,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499802112,"data_stored":499802112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160744,"ondisk_log_size":160744,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727464,"kb_used_data":122384,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93644376,"statfs":{"total":96636764160,"available":95891841024,"internally_reserved":0,"allocated":125321216,"data_stored":125132014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619572943},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,2,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754516,"kb_used_data":123588,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93617324,"statfs":{"total":96636764160,"available":95864139776,"internally_reserved":0,"allocated":126554112,"data_stored":126364910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,0,3,1,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582192,"kb_used_data":121504,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93789648,"statfs":{"total":96636764160,"available":96040599552,"internally_reserved":0,"allocated":124420096,"data_stored":124230894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471723727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,1,6,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731916,"kb_used_data":122748,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93639924,"statfs":{"total":96636764160,"available":95887282176,"internally_reserved":0,"allocated":125693952,"data_stored":125504750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":623767246},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,5,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125186048,"data_stored":125186048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":123691008,"data_stored":123691008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126009344,"data_stored":126009344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":124915712,"data_stored":124915712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:24.372 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:50:24.372 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10719", 'reported_seq': 10836, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:19.023025+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:50:19.023025+0000', 'last_peered': '2024-09-16T04:50:19.023025+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:50:19.023025+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15851520, 'num_objects': 1935, 'num_object_clones': 0, 'num_object_copies': 7740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1935, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10719, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -251}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:24.372 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:50:24.372 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10766", 'reported_seq': 10855, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:18.716303+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:50:18.716303+0000', 'last_peered': '2024-09-16T04:50:18.716303+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:50:18.716303+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15627532, 'num_objects': 1908, 'num_object_clones': 0, 'num_object_copies': 7632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10764, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -295}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:24.373 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:50:24.373 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10739", 'reported_seq': 10838, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:17.832115+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:50:17.832115+0000', 'last_peered': '2024-09-16T04:50:17.832115+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:50:17.832115+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15817374, 'num_objects': 1932, 'num_object_clones': 0, 'num_object_copies': 7728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1932, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10738, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -287}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:24.373 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:50:24.373 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10745", 'reported_seq': 14178, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:19.130016+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:50:19.130016+0000', 'last_peered': '2024-09-16T04:50:19.130016+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:50:19.130016+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15766796, 'num_objects': 1925, 'num_object_clones': 0, 'num_object_copies': 7700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 551, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10743, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1682, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1374}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:24.373 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:50:24.373 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10776", 'reported_seq': 10909, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:19.292480+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:50:19.292480+0000', 'last_peered': '2024-09-16T04:50:19.292480+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:50:19.292480+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16094476, 'num_objects': 1965, 'num_object_clones': 0, 'num_object_copies': 7860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1965, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10774, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -283}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:24.373 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:50:24.373 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10473", 'reported_seq': 10584, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:19.467973+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:50:19.467973+0000', 'last_peered': '2024-09-16T04:50:19.467973+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:50:19.467973+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15065088, 'num_objects': 1839, 'num_object_clones': 0, 'num_object_copies': 7356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1839, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10473, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -339}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:24.373 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:50:24.373 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10731", 'reported_seq': 10829, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:19.055015+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:50:19.055015+0000', 'last_peered': '2024-09-16T04:50:19.055015+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:50:19.055015+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15308044, 'num_objects': 1869, 'num_object_clones': 0, 'num_object_copies': 7476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1869, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10729, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -344}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:24.374 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:50:24.374 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10754", 'reported_seq': 10860, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:18.019498+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:50:18.019498+0000', 'last_peered': '2024-09-16T04:50:18.019498+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:50:18.019498+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16413964, 'num_objects': 2004, 'num_object_clones': 0, 'num_object_copies': 8016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2004, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10752, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -185}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:24.374 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:50:24.374 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10597", 'reported_seq': 10730, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:19.424486+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:50:19.424486+0000', 'last_peered': '2024-09-16T04:50:19.424486+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:50:19.424486+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15511442, 'num_objects': 1894, 'num_object_clones': 0, 'num_object_copies': 7576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1894, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10594, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -261}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:24.374 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:50:24.374 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10510", 'reported_seq': 10603, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:19.278043+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:50:19.278043+0000', 'last_peered': '2024-09-16T04:50:19.278043+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:50:19.278043+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14916230, 'num_objects': 1821, 'num_object_clones': 0, 'num_object_copies': 7284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1821, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10509, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -284}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:24.374 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:50:24.374 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10624", 'reported_seq': 10713, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:17.857900+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:50:17.857900+0000', 'last_peered': '2024-09-16T04:50:17.857900+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:50:17.857900+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15637126, 'num_objects': 1909, 'num_object_clones': 0, 'num_object_copies': 7636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10623, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -312}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:24.374 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:50:24.374 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10830", 'reported_seq': 10963, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:19.394747+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:50:19.394747+0000', 'last_peered': '2024-09-16T04:50:19.394747+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:50:19.394747+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15987980, 'num_objects': 1952, 'num_object_clones': 0, 'num_object_copies': 7808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1952, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10828, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -246}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:24.374 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:50:24.375 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10506", 'reported_seq': 10610, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:17.765982+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:50:17.765982+0000', 'last_peered': '2024-09-16T04:50:17.765982+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:50:17.765982+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15914252, 'num_objects': 1943, 'num_object_clones': 0, 'num_object_copies': 7772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10504, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -180}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:24.375 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:50:24.375 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10493", 'reported_seq': 10586, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:19.021576+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:50:19.021576+0000', 'last_peered': '2024-09-16T04:50:19.021576+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:50:19.021576+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14857484, 'num_objects': 1814, 'num_object_clones': 0, 'num_object_copies': 7256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1814, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10491, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -414}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:24.375 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:50:25.058 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:25.058 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29037/122280 objects degraded (23.746%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":637,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":30570,"data_bytes":250382016,"bytes_used":2863194112,"bytes_avail":383683862528,"bytes_total":386547056640,"degraded_objects":29037,"degraded_total":122280,"degraded_ratio":0.23746319921491663,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":170},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:50:25.080 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:50:25.080 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:25.685 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:25.686 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:25.712 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-16T04:50:24.740414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":242673344,"num_objects":29629,"num_object_clones":0,"num_object_copies":118516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29629,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171858,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6606,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160585,"ondisk_log_size":160585,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2796716,"kb_used_data":478380,"kb_used_omap":14,"kb_used_meta":2318257,"kb_avail":374690644,"statfs":{"total":386547056640,"available":383683219456,"internally_reserved":0,"allocated":489861120,"data_stored":489104312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2373895729},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,8,5,13,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23945216,"num_objects":-2923,"num_object_clones":0,"num_object_copies":-11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2923,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004920"},"pg_stats":[{"pgid":"1.f","version":"35'10776","reported_seq":10891,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.491806+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:24.491806+0000","last_peered":"2024-09-16T04:50:24.491806+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:24.491806+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15384576,"num_objects":1878,"num_object_clones":0,"num_object_copies":7512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10776,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10844","reported_seq":10933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.084464+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:24.084464+0000","last_peered":"2024-09-16T04:50:24.084464+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:24.084464+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14988556,"num_objects":1830,"num_object_clones":0,"num_object_copies":7320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10842,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-373}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10804","reported_seq":10903,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:22.880847+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:22.880847+0000","last_peered":"2024-09-16T04:50:22.880847+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:22.880847+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15284894,"num_objects":1867,"num_object_clones":0,"num_object_copies":7468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1867,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10803,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-352}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10809","reported_seq":14364,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.130503+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:24.130503+0000","last_peered":"2024-09-16T04:50:24.130503+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:24.130503+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15242508,"num_objects":1861,"num_object_clones":0,"num_object_copies":7444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1743,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1371}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10832","reported_seq":10965,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.486882+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:24.486882+0000","last_peered":"2024-09-16T04:50:24.486882+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:24.486882+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15635724,"num_objects":1909,"num_object_clones":0,"num_object_copies":7636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10830,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10523","reported_seq":10634,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.143965+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:24.143965+0000","last_peered":"2024-09-16T04:50:24.143965+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:24.143965+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14655488,"num_objects":1789,"num_object_clones":0,"num_object_copies":7156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10809","reported_seq":10907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:23.818214+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:23.818214+0000","last_peered":"2024-09-16T04:50:23.818214+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:23.818214+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14669068,"num_objects":1791,"num_object_clones":0,"num_object_copies":7164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-422}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10831","reported_seq":15393,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:21.661743+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:21.661743+0000","last_peered":"2024-09-16T04:50:21.661743+0000","last_clean":"2024-09-16T04:50:21.661743+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:21.661743+0000","last_undegraded":"2024-09-16T04:50:21.661743+0000","last_fullsized":"2024-09-16T04:50:21.661743+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:26.187103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16004364,"num_objects":1954,"num_object_clones":0,"num_object_copies":7816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10908","reported_seq":15666,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:24.334953+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:24.334953+0000","last_peered":"2024-09-16T04:50:24.334953+0000","last_clean":"2024-09-16T04:50:24.334953+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:24.334953+0000","last_undegraded":"2024-09-16T04:50:24.334953+0000","last_fullsized":"2024-09-16T04:50:24.334953+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:25.548394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15133208,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10905,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10826","reported_seq":10933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:22.783124+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:22.783124+0000","last_peered":"2024-09-16T04:50:22.783124+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:22.783124+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15824140,"num_objects":1932,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10824,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10652","reported_seq":10785,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.435596+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:24.435596+0000","last_peered":"2024-09-16T04:50:24.435596+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:24.435596+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15060882,"num_objects":1839,"num_object_clones":0,"num_object_copies":7356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10649,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10570","reported_seq":10662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.493248+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:24.493248+0000","last_peered":"2024-09-16T04:50:24.493248+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:24.493248+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14424710,"num_objects":1761,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10569,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-344}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10677","reported_seq":10766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:22.650775+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:22.650775+0000","last_peered":"2024-09-16T04:50:22.650775+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:22.650775+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15202950,"num_objects":1856,"num_object_clones":0,"num_object_copies":7424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10676,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-365}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10901","reported_seq":11034,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.457767+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:24.457767+0000","last_peered":"2024-09-16T04:50:24.457767+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:24.457767+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15406348,"num_objects":1881,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10899,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-317}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10578","reported_seq":10682,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:22.985803+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:22.985803+0000","last_peered":"2024-09-16T04:50:22.985803+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:22.985803+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15324428,"num_objects":1871,"num_object_clones":0,"num_object_copies":7484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10576,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10545","reported_seq":10639,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:23.931217+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:23.931217+0000","last_peered":"2024-09-16T04:50:23.931217+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:23.931217+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14431500,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10543,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-466}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":242673344,"num_objects":29629,"num_object_clones":0,"num_object_copies":118516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29629,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171858,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6606,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487702528,"data_stored":487702528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160585,"ondisk_log_size":160585,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727604,"kb_used_data":118372,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93644236,"statfs":{"total":96636764160,"available":95891697664,"internally_reserved":0,"allocated":121212928,"data_stored":121023726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623832783},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754812,"kb_used_data":119724,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93617028,"statfs":{"total":96636764160,"available":95863836672,"internally_reserved":0,"allocated":122597376,"data_stored":122408174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582192,"kb_used_data":121504,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93789648,"statfs":{"total":96636764160,"available":96040599552,"internally_reserved":0,"allocated":124420096,"data_stored":124230894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471723727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,1,6,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732108,"kb_used_data":118780,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93639732,"statfs":{"total":96636764160,"available":95887085568,"internally_reserved":0,"allocated":121630720,"data_stored":121441518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":628027086},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":121151488,"data_stored":121151488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":123691008,"data_stored":123691008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":121982976,"data_stored":121982976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":120877056,"data_stored":120877056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:28.714 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:29.316 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:29.316 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:29.338 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-16T04:50:28.742158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":240289472,"num_objects":29338,"num_object_clones":0,"num_object_copies":117352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29338,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172149,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6606,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160676,"ondisk_log_size":160676,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2780828,"kb_used_data":470812,"kb_used_omap":14,"kb_used_meta":2309937,"kb_avail":374706532,"statfs":{"total":386547056640,"available":383699488768,"internally_reserved":0,"allocated":482111488,"data_stored":481354680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2365376049},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,9,7,20,9,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18300928,"num_objects":-2234,"num_object_clones":0,"num_object_copies":-8936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2234,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004447"},"pg_stats":[{"pgid":"1.f","version":"35'10776","reported_seq":10891,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.491806+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:24.491806+0000","last_peered":"2024-09-16T04:50:24.491806+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:24.491806+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15384576,"num_objects":1878,"num_object_clones":0,"num_object_copies":7512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10776,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10844","reported_seq":10933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.084464+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:24.084464+0000","last_peered":"2024-09-16T04:50:24.084464+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:24.084464+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14988556,"num_objects":1830,"num_object_clones":0,"num_object_copies":7320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10842,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-373}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10879","reported_seq":10978,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:27.845128+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:27.845128+0000","last_peered":"2024-09-16T04:50:27.845128+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:27.845128+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14670494,"num_objects":1792,"num_object_clones":0,"num_object_copies":7168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1792,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10878,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10809","reported_seq":14364,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.130503+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:24.130503+0000","last_peered":"2024-09-16T04:50:24.130503+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:24.130503+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15242508,"num_objects":1861,"num_object_clones":0,"num_object_copies":7444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1743,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1371}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10832","reported_seq":10965,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.486882+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:24.486882+0000","last_peered":"2024-09-16T04:50:24.486882+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:24.486882+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15635724,"num_objects":1909,"num_object_clones":0,"num_object_copies":7636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10830,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10523","reported_seq":10634,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.143965+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:24.143965+0000","last_peered":"2024-09-16T04:50:24.143965+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:24.143965+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14655488,"num_objects":1789,"num_object_clones":0,"num_object_copies":7156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10809","reported_seq":10907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:23.818214+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:23.818214+0000","last_peered":"2024-09-16T04:50:23.818214+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:23.818214+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14669068,"num_objects":1791,"num_object_clones":0,"num_object_copies":7164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-422}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10872","reported_seq":15434,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:26.340236+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:26.340236+0000","last_peered":"2024-09-16T04:50:26.340236+0000","last_clean":"2024-09-16T04:50:26.340236+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:26.340236+0000","last_undegraded":"2024-09-16T04:50:26.340236+0000","last_fullsized":"2024-09-16T04:50:26.340236+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15668492,"num_objects":1913,"num_object_clones":0,"num_object_copies":7652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10870,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10908","reported_seq":15666,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:24.334953+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:24.334953+0000","last_peered":"2024-09-16T04:50:24.334953+0000","last_clean":"2024-09-16T04:50:24.334953+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:24.334953+0000","last_undegraded":"2024-09-16T04:50:24.334953+0000","last_fullsized":"2024-09-16T04:50:24.334953+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:25.548394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15133208,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10905,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10881","reported_seq":10988,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:28.007937+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:28.007937+0000","last_peered":"2024-09-16T04:50:28.007937+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:28.007937+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15373580,"num_objects":1877,"num_object_clones":0,"num_object_copies":7508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10879,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-312}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10652","reported_seq":10785,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.435596+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:24.435596+0000","last_peered":"2024-09-16T04:50:24.435596+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:24.435596+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15060882,"num_objects":1839,"num_object_clones":0,"num_object_copies":7356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10649,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10570","reported_seq":10662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.493248+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:24.493248+0000","last_peered":"2024-09-16T04:50:24.493248+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:24.493248+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14424710,"num_objects":1761,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10569,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-344}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10735","reported_seq":10824,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:27.949992+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:27.949992+0000","last_peered":"2024-09-16T04:50:27.949992+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:27.949992+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14727814,"num_objects":1798,"num_object_clones":0,"num_object_copies":7192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10734,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10901","reported_seq":11034,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.457767+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:24.457767+0000","last_peered":"2024-09-16T04:50:24.457767+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:24.457767+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15406348,"num_objects":1881,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10899,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-317}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10640","reported_seq":10744,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:28.004868+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:28.004868+0000","last_peered":"2024-09-16T04:50:28.004868+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:28.004868+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14816524,"num_objects":1809,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10638,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-314}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10545","reported_seq":10639,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:23.931217+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:23.931217+0000","last_peered":"2024-09-16T04:50:23.931217+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:23.931217+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14431500,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10543,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-466}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":240289472,"num_objects":29338,"num_object_clones":0,"num_object_copies":117352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29338,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172149,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6606,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":479830016,"data_stored":479830016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160676,"ondisk_log_size":160676,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727604,"kb_used_data":118372,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93644236,"statfs":{"total":96636764160,"available":95891697664,"internally_reserved":0,"allocated":121212928,"data_stored":121023726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623832783},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738588,"kb_used_data":115980,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93633252,"statfs":{"total":96636764160,"available":95880450048,"internally_reserved":0,"allocated":118763520,"data_stored":118574318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,9,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582528,"kb_used_data":117680,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93789312,"statfs":{"total":96636764160,"available":96040255488,"internally_reserved":0,"allocated":120504320,"data_stored":120315118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475983567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,2,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732108,"kb_used_data":118780,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93639732,"statfs":{"total":96636764160,"available":95887085568,"internally_reserved":0,"allocated":121630720,"data_stored":121441518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":628027086},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":121151488,"data_stored":121151488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":119779328,"data_stored":119779328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":118022144,"data_stored":118022144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":120877056,"data_stored":120877056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:29.339 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:29.934 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:29.935 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:29.968 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-16T04:50:28.742158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":240289472,"num_objects":29338,"num_object_clones":0,"num_object_copies":117352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29338,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172149,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6606,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160676,"ondisk_log_size":160676,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2780828,"kb_used_data":470812,"kb_used_omap":14,"kb_used_meta":2309937,"kb_avail":374706532,"statfs":{"total":386547056640,"available":383699488768,"internally_reserved":0,"allocated":482111488,"data_stored":481354680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2365376049},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,9,7,20,9,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18300928,"num_objects":-2234,"num_object_clones":0,"num_object_copies":-8936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2234,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004447"},"pg_stats":[{"pgid":"1.f","version":"35'10776","reported_seq":10891,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.491806+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:24.491806+0000","last_peered":"2024-09-16T04:50:24.491806+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:24.491806+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15384576,"num_objects":1878,"num_object_clones":0,"num_object_copies":7512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10776,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10844","reported_seq":10933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.084464+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:24.084464+0000","last_peered":"2024-09-16T04:50:24.084464+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:24.084464+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14988556,"num_objects":1830,"num_object_clones":0,"num_object_copies":7320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10842,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-373}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10879","reported_seq":10978,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:27.845128+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:27.845128+0000","last_peered":"2024-09-16T04:50:27.845128+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:27.845128+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14670494,"num_objects":1792,"num_object_clones":0,"num_object_copies":7168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1792,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10878,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10809","reported_seq":14364,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.130503+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:24.130503+0000","last_peered":"2024-09-16T04:50:24.130503+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:24.130503+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15242508,"num_objects":1861,"num_object_clones":0,"num_object_copies":7444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1743,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1371}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10832","reported_seq":10965,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.486882+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:24.486882+0000","last_peered":"2024-09-16T04:50:24.486882+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:24.486882+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15635724,"num_objects":1909,"num_object_clones":0,"num_object_copies":7636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10830,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10523","reported_seq":10634,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.143965+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:24.143965+0000","last_peered":"2024-09-16T04:50:24.143965+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:24.143965+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14655488,"num_objects":1789,"num_object_clones":0,"num_object_copies":7156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10809","reported_seq":10907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:23.818214+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:23.818214+0000","last_peered":"2024-09-16T04:50:23.818214+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:23.818214+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14669068,"num_objects":1791,"num_object_clones":0,"num_object_copies":7164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-422}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10872","reported_seq":15434,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:26.340236+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:26.340236+0000","last_peered":"2024-09-16T04:50:26.340236+0000","last_clean":"2024-09-16T04:50:26.340236+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:26.340236+0000","last_undegraded":"2024-09-16T04:50:26.340236+0000","last_fullsized":"2024-09-16T04:50:26.340236+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15668492,"num_objects":1913,"num_object_clones":0,"num_object_copies":7652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10870,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10908","reported_seq":15666,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:24.334953+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:24.334953+0000","last_peered":"2024-09-16T04:50:24.334953+0000","last_clean":"2024-09-16T04:50:24.334953+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:24.334953+0000","last_undegraded":"2024-09-16T04:50:24.334953+0000","last_fullsized":"2024-09-16T04:50:24.334953+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:25.548394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15133208,"num_objects":1848,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10905,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10881","reported_seq":10988,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:28.007937+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:28.007937+0000","last_peered":"2024-09-16T04:50:28.007937+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:28.007937+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15373580,"num_objects":1877,"num_object_clones":0,"num_object_copies":7508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10879,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-312}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10652","reported_seq":10785,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.435596+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:24.435596+0000","last_peered":"2024-09-16T04:50:24.435596+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:24.435596+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15060882,"num_objects":1839,"num_object_clones":0,"num_object_copies":7356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10649,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10570","reported_seq":10662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.493248+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:24.493248+0000","last_peered":"2024-09-16T04:50:24.493248+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:24.493248+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14424710,"num_objects":1761,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10569,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-344}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10735","reported_seq":10824,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:27.949992+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:27.949992+0000","last_peered":"2024-09-16T04:50:27.949992+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:27.949992+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14727814,"num_objects":1798,"num_object_clones":0,"num_object_copies":7192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10734,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10901","reported_seq":11034,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:24.457767+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:24.457767+0000","last_peered":"2024-09-16T04:50:24.457767+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:24.457767+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15406348,"num_objects":1881,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10899,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-317}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10640","reported_seq":10744,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:28.004868+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:28.004868+0000","last_peered":"2024-09-16T04:50:28.004868+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:28.004868+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14816524,"num_objects":1809,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10638,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-314}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10545","reported_seq":10639,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:23.931217+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:23.931217+0000","last_peered":"2024-09-16T04:50:23.931217+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:23.931217+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14431500,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10543,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-466}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":240289472,"num_objects":29338,"num_object_clones":0,"num_object_copies":117352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29338,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172149,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6606,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":479830016,"data_stored":479830016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160676,"ondisk_log_size":160676,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727604,"kb_used_data":118372,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93644236,"statfs":{"total":96636764160,"available":95891697664,"internally_reserved":0,"allocated":121212928,"data_stored":121023726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623832783},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738588,"kb_used_data":115980,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93633252,"statfs":{"total":96636764160,"available":95880450048,"internally_reserved":0,"allocated":118763520,"data_stored":118574318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,9,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582528,"kb_used_data":117680,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93789312,"statfs":{"total":96636764160,"available":96040255488,"internally_reserved":0,"allocated":120504320,"data_stored":120315118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475983567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,2,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732108,"kb_used_data":118780,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93639732,"statfs":{"total":96636764160,"available":95887085568,"internally_reserved":0,"allocated":121630720,"data_stored":121441518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":628027086},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":121151488,"data_stored":121151488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":119779328,"data_stored":119779328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":118022144,"data_stored":118022144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":120877056,"data_stored":120877056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:29.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:50:29.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10776", 'reported_seq': 10891, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:24.491806+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:50:24.491806+0000', 'last_peered': '2024-09-16T04:50:24.491806+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:50:24.491806+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15384576, 'num_objects': 1878, 'num_object_clones': 0, 'num_object_copies': 7512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1878, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10776, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -308}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:29.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:50:29.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10844", 'reported_seq': 10933, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:24.084464+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:50:24.084464+0000', 'last_peered': '2024-09-16T04:50:24.084464+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:50:24.084464+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14988556, 'num_objects': 1830, 'num_object_clones': 0, 'num_object_copies': 7320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1830, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10842, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -373}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:29.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:50:29.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10879", 'reported_seq': 10978, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:27.845128+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:50:27.845128+0000', 'last_peered': '2024-09-16T04:50:27.845128+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:50:27.845128+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14670494, 'num_objects': 1792, 'num_object_clones': 0, 'num_object_copies': 7168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1792, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10878, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -427}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:29.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:50:29.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10809", 'reported_seq': 14364, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:24.130503+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:50:24.130503+0000', 'last_peered': '2024-09-16T04:50:24.130503+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:50:24.130503+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15242508, 'num_objects': 1861, 'num_object_clones': 0, 'num_object_copies': 7444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 490, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10807, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1743, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1371}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:29.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:50:29.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10832", 'reported_seq': 10965, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:24.486882+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:50:24.486882+0000', 'last_peered': '2024-09-16T04:50:24.486882+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:50:24.486882+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15635724, 'num_objects': 1909, 'num_object_clones': 0, 'num_object_copies': 7636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10830, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -339}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:29.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:50:29.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10523", 'reported_seq': 10634, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:24.143965+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:50:24.143965+0000', 'last_peered': '2024-09-16T04:50:24.143965+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:50:24.143965+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14655488, 'num_objects': 1789, 'num_object_clones': 0, 'num_object_copies': 7156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1789, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10523, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -389}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:29.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:50:29.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10809", 'reported_seq': 10907, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:23.818214+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:50:23.818214+0000', 'last_peered': '2024-09-16T04:50:23.818214+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:50:23.818214+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14669068, 'num_objects': 1791, 'num_object_clones': 0, 'num_object_copies': 7164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1791, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10807, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -422}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:29.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:50:29.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10881", 'reported_seq': 10988, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:28.007937+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:50:28.007937+0000', 'last_peered': '2024-09-16T04:50:28.007937+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:50:28.007937+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15373580, 'num_objects': 1877, 'num_object_clones': 0, 'num_object_copies': 7508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1877, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10879, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -312}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:29.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:50:29.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10652", 'reported_seq': 10785, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:24.435596+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:50:24.435596+0000', 'last_peered': '2024-09-16T04:50:24.435596+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:50:24.435596+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15060882, 'num_objects': 1839, 'num_object_clones': 0, 'num_object_copies': 7356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1839, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10649, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:29.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:50:29.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10570", 'reported_seq': 10662, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:24.493248+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:50:24.493248+0000', 'last_peered': '2024-09-16T04:50:24.493248+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:50:24.493248+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14424710, 'num_objects': 1761, 'num_object_clones': 0, 'num_object_copies': 7044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10569, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -344}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:29.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:50:29.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10735", 'reported_seq': 10824, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:27.949992+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:50:27.949992+0000', 'last_peered': '2024-09-16T04:50:27.949992+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:50:27.949992+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14727814, 'num_objects': 1798, 'num_object_clones': 0, 'num_object_copies': 7192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10734, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -423}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:29.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:50:29.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10901", 'reported_seq': 11034, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:24.457767+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:50:24.457767+0000', 'last_peered': '2024-09-16T04:50:24.457767+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:50:24.457767+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15406348, 'num_objects': 1881, 'num_object_clones': 0, 'num_object_copies': 7524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1881, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10899, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -317}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:29.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:50:29.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10640", 'reported_seq': 10744, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:28.004868+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:50:28.004868+0000', 'last_peered': '2024-09-16T04:50:28.004868+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:50:28.004868+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14816524, 'num_objects': 1809, 'num_object_clones': 0, 'num_object_copies': 7236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1809, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10638, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -314}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:29.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:50:29.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10545", 'reported_seq': 10639, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:23.931217+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:50:23.931217+0000', 'last_peered': '2024-09-16T04:50:23.931217+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:50:23.931217+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14431500, 'num_objects': 1762, 'num_object_clones': 0, 'num_object_copies': 7048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1762, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10543, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -466}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:29.972 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:50:30.716 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:30.716 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28976/117352 objects degraded (24.692%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":643,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":29338,"data_bytes":240289472,"bytes_used":2847567872,"bytes_avail":383699488768,"bytes_total":386547056640,"degraded_objects":28976,"degraded_total":117352,"degraded_ratio":0.24691526348080989,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":186},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:50:30.740 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:50:30.741 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:31.309 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:31.309 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:31.332 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-09-16T04:50:30.743050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":234874560,"num_objects":28677,"num_object_clones":0,"num_object_copies":114708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28677,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172810,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6662,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160937,"ondisk_log_size":160937,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781512,"kb_used_data":463176,"kb_used_omap":14,"kb_used_meta":2318257,"kb_avail":374705848,"statfs":{"total":386547056640,"available":383698788352,"internally_reserved":0,"allocated":474292224,"data_stored":473535416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2373895729},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,5,8,22,12,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21471232,"num_objects":-2621,"num_object_clones":0,"num_object_copies":-10484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2621,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004567"},"pg_stats":[{"pgid":"1.f","version":"35'10851","reported_seq":10968,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:29.633352+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:29.633352+0000","last_peered":"2024-09-16T04:50:29.633352+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:29.633352+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14770176,"num_objects":1803,"num_object_clones":0,"num_object_copies":7212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10851,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-383}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10899","reported_seq":10988,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:29.017877+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:29.017877+0000","last_peered":"2024-09-16T04:50:29.017877+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:29.017877+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14537996,"num_objects":1775,"num_object_clones":0,"num_object_copies":7100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10897,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-428}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10879","reported_seq":10978,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:27.845128+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:27.845128+0000","last_peered":"2024-09-16T04:50:27.845128+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:27.845128+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14670494,"num_objects":1792,"num_object_clones":0,"num_object_copies":7168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1792,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10878,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10864","reported_seq":14532,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:29.078719+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:29.078719+0000","last_peered":"2024-09-16T04:50:29.078719+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:29.078719+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14791948,"num_objects":1806,"num_object_clones":0,"num_object_copies":7224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10862,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1799,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1372}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10904","reported_seq":11037,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:29.309160+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:29.309160+0000","last_peered":"2024-09-16T04:50:29.309160+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:29.309160+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15045900,"num_objects":1837,"num_object_clones":0,"num_object_copies":7348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10902,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10588","reported_seq":10699,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:29.542181+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:29.542181+0000","last_peered":"2024-09-16T04:50:29.542181+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:29.542181+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14123008,"num_objects":1724,"num_object_clones":0,"num_object_copies":6896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10588,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10868","reported_seq":10965,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:28.968585+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:28.968585+0000","last_peered":"2024-09-16T04:50:28.968585+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:28.968585+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14185740,"num_objects":1732,"num_object_clones":0,"num_object_copies":6928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10866,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-481}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10872","reported_seq":15434,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:26.340236+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:26.340236+0000","last_peered":"2024-09-16T04:50:26.340236+0000","last_clean":"2024-09-16T04:50:26.340236+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:26.340236+0000","last_undegraded":"2024-09-16T04:50:26.340236+0000","last_fullsized":"2024-09-16T04:50:26.340236+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15668492,"num_objects":1913,"num_object_clones":0,"num_object_copies":7652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10870,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'10966","reported_seq":15724,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:29.542854+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:29.542854+0000","last_peered":"2024-09-16T04:50:29.542854+0000","last_clean":"2024-09-16T04:50:29.542854+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:29.542854+0000","last_undegraded":"2024-09-16T04:50:29.542854+0000","last_fullsized":"2024-09-16T04:50:29.542854+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:31.499823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14658072,"num_objects":1790,"num_object_clones":0,"num_object_copies":7160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10881","reported_seq":10988,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:28.007937+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:28.007937+0000","last_peered":"2024-09-16T04:50:28.007937+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:28.007937+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15373580,"num_objects":1877,"num_object_clones":0,"num_object_copies":7508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10879,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-312}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10708","reported_seq":10841,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:29.441854+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:29.441854+0000","last_peered":"2024-09-16T04:50:29.441854+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:29.441854+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14602130,"num_objects":1783,"num_object_clones":0,"num_object_copies":7132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10705,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-372}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10632","reported_seq":10725,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:29.530946+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:29.530946+0000","last_peered":"2024-09-16T04:50:29.530946+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:29.530946+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13916806,"num_objects":1699,"num_object_clones":0,"num_object_copies":6796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10631,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-406}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10735","reported_seq":10824,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:27.949992+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:27.949992+0000","last_peered":"2024-09-16T04:50:27.949992+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:27.949992+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14727814,"num_objects":1798,"num_object_clones":0,"num_object_copies":7192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10734,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10961","reported_seq":11094,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:29.560903+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:29.560903+0000","last_peered":"2024-09-16T04:50:29.560903+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:29.560903+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14914828,"num_objects":1821,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10959,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-377}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10640","reported_seq":10744,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:28.004868+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:28.004868+0000","last_peered":"2024-09-16T04:50:28.004868+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:28.004868+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14816524,"num_objects":1809,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10638,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-314}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10589","reported_seq":10684,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:29.032866+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:29.032866+0000","last_peered":"2024-09-16T04:50:29.032866+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:29.032866+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14071052,"num_objects":1718,"num_object_clones":0,"num_object_copies":6872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10587,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-510}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":234874560,"num_objects":28677,"num_object_clones":0,"num_object_copies":114708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28677,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172810,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6662,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471830528,"data_stored":471830528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160937,"ondisk_log_size":160937,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727916,"kb_used_data":114524,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93643924,"statfs":{"total":96636764160,"available":95891378176,"internally_reserved":0,"allocated":117272576,"data_stored":117083374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628092623},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738588,"kb_used_data":115980,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93633252,"statfs":{"total":96636764160,"available":95880450048,"internally_reserved":0,"allocated":118763520,"data_stored":118574318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,9,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582528,"kb_used_data":117680,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93789312,"statfs":{"total":96636764160,"available":96040255488,"internally_reserved":0,"allocated":120504320,"data_stored":120315118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475983567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,2,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247931,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732480,"kb_used_data":114992,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93639360,"statfs":{"total":96636764160,"available":95886704640,"internally_reserved":0,"allocated":117751808,"data_stored":117562606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":632286926},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,1,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":117211136,"data_stored":117211136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":119779328,"data_stored":119779328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":118022144,"data_stored":118022144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":116817920,"data_stored":116817920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:34.334 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:34.969 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:34.969 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:34.992 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-09-16T04:50:34.744576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":226887360,"num_objects":27702,"num_object_clones":0,"num_object_copies":110808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173785,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6733,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2784180,"kb_used_data":447784,"kb_used_omap":14,"kb_used_meta":2336305,"kb_avail":374703180,"statfs":{"total":386547056640,"available":383696056320,"internally_reserved":0,"allocated":458530816,"data_stored":457774008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2392376881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,12,0,0,8,19,16,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23494656,"num_objects":-2868,"num_object_clones":0,"num_object_copies":-11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2868,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004820"},"pg_stats":[{"pgid":"1.f","version":"35'10914","reported_seq":11031,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.594311+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:34.594311+0000","last_peered":"2024-09-16T04:50:34.594311+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:34.594311+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14254080,"num_objects":1740,"num_object_clones":0,"num_object_copies":6960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10914,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-446}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10962","reported_seq":11051,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.001916+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:34.001916+0000","last_peered":"2024-09-16T04:50:34.001916+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:34.001916+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14021900,"num_objects":1712,"num_object_clones":0,"num_object_copies":6848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10960,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-491}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10948","reported_seq":11045,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:33.121790+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:33.121790+0000","last_peered":"2024-09-16T04:50:33.121790+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:33.121790+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14105246,"num_objects":1723,"num_object_clones":0,"num_object_copies":6892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1723,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10947,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10908","reported_seq":14718,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.162340+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:34.162340+0000","last_peered":"2024-09-16T04:50:34.162340+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:34.162340+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14431500,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10906,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1870,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1399}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10965","reported_seq":11098,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.412489+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:34.412489+0000","last_peered":"2024-09-16T04:50:34.412489+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:34.412489+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14546188,"num_objects":1776,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10647","reported_seq":10758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.543929+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:34.543929+0000","last_peered":"2024-09-16T04:50:34.543929+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:34.543929+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13639680,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10647,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10934","reported_seq":11032,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.061051+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:34.061051+0000","last_peered":"2024-09-16T04:50:34.061051+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:34.061051+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13645068,"num_objects":1666,"num_object_clones":0,"num_object_copies":6664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10932,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-547}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10930","reported_seq":15492,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:31.795905+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:31.795905+0000","last_peered":"2024-09-16T04:50:31.795905+0000","last_clean":"2024-09-16T04:50:31.795905+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:31.795905+0000","last_undegraded":"2024-09-16T04:50:31.795905+0000","last_fullsized":"2024-09-16T04:50:31.795905+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:32.234512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15193356,"num_objects":1855,"num_object_clones":0,"num_object_copies":7420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10928,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11015","reported_seq":15772,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:34.599937+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:34.599937+0000","last_peered":"2024-09-16T04:50:34.599937+0000","last_clean":"2024-09-16T04:50:34.599937+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:34.599937+0000","last_undegraded":"2024-09-16T04:50:34.599937+0000","last_fullsized":"2024-09-16T04:50:34.599937+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:37.456287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14256664,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11012,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10931","reported_seq":11038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:33.011247+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:33.011247+0000","last_peered":"2024-09-16T04:50:33.011247+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:33.011247+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14963980,"num_objects":1827,"num_object_clones":0,"num_object_copies":7308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10929,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-362}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10782","reported_seq":10914,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.698708+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:34.698708+0000","last_peered":"2024-09-16T04:50:34.698708+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:34.698708+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13995922,"num_objects":1709,"num_object_clones":0,"num_object_copies":6836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10779,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-446}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10687","reported_seq":10780,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.650964+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:34.650964+0000","last_peered":"2024-09-16T04:50:34.650964+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:34.650964+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13466246,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10686,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-461}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10790","reported_seq":10879,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:33.084191+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:33.084191+0000","last_peered":"2024-09-16T04:50:33.084191+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:33.084191+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14277254,"num_objects":1743,"num_object_clones":0,"num_object_copies":6972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10789,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-478}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11029","reported_seq":11162,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.325271+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:34.325271+0000","last_peered":"2024-09-16T04:50:34.325271+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:34.325271+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14357772,"num_objects":1753,"num_object_clones":0,"num_object_copies":7012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11027,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-445}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10705","reported_seq":10809,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:33.095908+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:33.095908+0000","last_peered":"2024-09-16T04:50:33.095908+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:33.095908+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14284044,"num_objects":1744,"num_object_clones":0,"num_object_copies":6976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10703,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10665","reported_seq":10760,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.226733+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:34.226733+0000","last_peered":"2024-09-16T04:50:34.226733+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:34.226733+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13448460,"num_objects":1642,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":226887360,"num_objects":27702,"num_object_clones":0,"num_object_copies":110808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173785,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6733,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":455897088,"data_stored":455897088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728244,"kb_used_data":110688,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93643596,"statfs":{"total":96636764160,"available":95891042304,"internally_reserved":0,"allocated":113344512,"data_stored":113155310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738956,"kb_used_data":112188,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93632884,"statfs":{"total":96636764160,"available":95880073216,"internally_reserved":0,"allocated":114880512,"data_stored":114691310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,12,0,0,0,5,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600804,"kb_used_data":113748,"kb_used_omap":4,"kb_used_meta":487035,"kb_avail":93771036,"statfs":{"total":96636764160,"available":96021540864,"internally_reserved":0,"allocated":116477952,"data_stored":116288750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":498724559},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716176,"kb_used_data":111160,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93655664,"statfs":{"total":96636764160,"available":95903399936,"internally_reserved":0,"allocated":113827840,"data_stored":113638638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":619507406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113172480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":115802112,"data_stored":115802112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114089984,"data_stored":114089984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":112832512,"data_stored":112832512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:34.993 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:35.605 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:35.605 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:35.625 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-09-16T04:50:34.744576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":226887360,"num_objects":27702,"num_object_clones":0,"num_object_copies":110808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173785,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6733,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2784180,"kb_used_data":447784,"kb_used_omap":14,"kb_used_meta":2336305,"kb_avail":374703180,"statfs":{"total":386547056640,"available":383696056320,"internally_reserved":0,"allocated":458530816,"data_stored":457774008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2392376881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,12,0,0,8,19,16,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23494656,"num_objects":-2868,"num_object_clones":0,"num_object_copies":-11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2868,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004820"},"pg_stats":[{"pgid":"1.f","version":"35'10914","reported_seq":11031,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.594311+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:34.594311+0000","last_peered":"2024-09-16T04:50:34.594311+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:34.594311+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14254080,"num_objects":1740,"num_object_clones":0,"num_object_copies":6960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10914,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-446}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10962","reported_seq":11051,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.001916+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:34.001916+0000","last_peered":"2024-09-16T04:50:34.001916+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:34.001916+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14021900,"num_objects":1712,"num_object_clones":0,"num_object_copies":6848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10960,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-491}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10948","reported_seq":11045,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:33.121790+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:33.121790+0000","last_peered":"2024-09-16T04:50:33.121790+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:33.121790+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14105246,"num_objects":1723,"num_object_clones":0,"num_object_copies":6892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1723,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10947,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10908","reported_seq":14718,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.162340+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:34.162340+0000","last_peered":"2024-09-16T04:50:34.162340+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:34.162340+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14431500,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10906,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1870,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1399}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10965","reported_seq":11098,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.412489+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:34.412489+0000","last_peered":"2024-09-16T04:50:34.412489+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:34.412489+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14546188,"num_objects":1776,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10647","reported_seq":10758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.543929+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:34.543929+0000","last_peered":"2024-09-16T04:50:34.543929+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:34.543929+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13639680,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10647,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10934","reported_seq":11032,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.061051+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:34.061051+0000","last_peered":"2024-09-16T04:50:34.061051+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:34.061051+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13645068,"num_objects":1666,"num_object_clones":0,"num_object_copies":6664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10932,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-547}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10930","reported_seq":15492,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:31.795905+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:31.795905+0000","last_peered":"2024-09-16T04:50:31.795905+0000","last_clean":"2024-09-16T04:50:31.795905+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:31.795905+0000","last_undegraded":"2024-09-16T04:50:31.795905+0000","last_fullsized":"2024-09-16T04:50:31.795905+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:32.234512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15193356,"num_objects":1855,"num_object_clones":0,"num_object_copies":7420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10928,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11015","reported_seq":15772,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:34.599937+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:34.599937+0000","last_peered":"2024-09-16T04:50:34.599937+0000","last_clean":"2024-09-16T04:50:34.599937+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:34.599937+0000","last_undegraded":"2024-09-16T04:50:34.599937+0000","last_fullsized":"2024-09-16T04:50:34.599937+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:37.456287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14256664,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11012,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10931","reported_seq":11038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:33.011247+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:33.011247+0000","last_peered":"2024-09-16T04:50:33.011247+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:33.011247+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14963980,"num_objects":1827,"num_object_clones":0,"num_object_copies":7308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10929,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-362}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10782","reported_seq":10914,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.698708+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:34.698708+0000","last_peered":"2024-09-16T04:50:34.698708+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:34.698708+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13995922,"num_objects":1709,"num_object_clones":0,"num_object_copies":6836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10779,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-446}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10687","reported_seq":10780,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.650964+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:34.650964+0000","last_peered":"2024-09-16T04:50:34.650964+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:34.650964+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13466246,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10686,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-461}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10790","reported_seq":10879,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:33.084191+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:33.084191+0000","last_peered":"2024-09-16T04:50:33.084191+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:33.084191+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14277254,"num_objects":1743,"num_object_clones":0,"num_object_copies":6972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10789,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-478}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11029","reported_seq":11162,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.325271+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:34.325271+0000","last_peered":"2024-09-16T04:50:34.325271+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:34.325271+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14357772,"num_objects":1753,"num_object_clones":0,"num_object_copies":7012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11027,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-445}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10705","reported_seq":10809,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:33.095908+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:33.095908+0000","last_peered":"2024-09-16T04:50:33.095908+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:33.095908+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14284044,"num_objects":1744,"num_object_clones":0,"num_object_copies":6976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10703,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10665","reported_seq":10760,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.226733+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:34.226733+0000","last_peered":"2024-09-16T04:50:34.226733+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:34.226733+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13448460,"num_objects":1642,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":226887360,"num_objects":27702,"num_object_clones":0,"num_object_copies":110808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173785,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6733,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":455897088,"data_stored":455897088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728244,"kb_used_data":110688,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93643596,"statfs":{"total":96636764160,"available":95891042304,"internally_reserved":0,"allocated":113344512,"data_stored":113155310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738956,"kb_used_data":112188,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93632884,"statfs":{"total":96636764160,"available":95880073216,"internally_reserved":0,"allocated":114880512,"data_stored":114691310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,12,0,0,0,5,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600804,"kb_used_data":113748,"kb_used_omap":4,"kb_used_meta":487035,"kb_avail":93771036,"statfs":{"total":96636764160,"available":96021540864,"internally_reserved":0,"allocated":116477952,"data_stored":116288750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":498724559},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716176,"kb_used_data":111160,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93655664,"statfs":{"total":96636764160,"available":95903399936,"internally_reserved":0,"allocated":113827840,"data_stored":113638638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":619507406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113172480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":115802112,"data_stored":115802112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114089984,"data_stored":114089984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":112832512,"data_stored":112832512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:35.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:50:35.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10914", 'reported_seq': 11031, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:34.594311+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:50:34.594311+0000', 'last_peered': '2024-09-16T04:50:34.594311+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:50:34.594311+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14254080, 'num_objects': 1740, 'num_object_clones': 0, 'num_object_copies': 6960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1740, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10914, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -446}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:35.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:50:35.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'10962", 'reported_seq': 11051, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:34.001916+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:50:34.001916+0000', 'last_peered': '2024-09-16T04:50:34.001916+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:50:34.001916+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14021900, 'num_objects': 1712, 'num_object_clones': 0, 'num_object_copies': 6848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1712, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10960, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -491}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:50:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10948", 'reported_seq': 11045, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:33.121790+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:50:33.121790+0000', 'last_peered': '2024-09-16T04:50:33.121790+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:50:33.121790+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14105246, 'num_objects': 1723, 'num_object_clones': 0, 'num_object_copies': 6892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1723, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10947, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -496}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:50:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10908", 'reported_seq': 14718, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:34.162340+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:50:34.162340+0000', 'last_peered': '2024-09-16T04:50:34.162340+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:50:34.162340+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14431500, 'num_objects': 1762, 'num_object_clones': 0, 'num_object_copies': 7048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 363, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1762, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10906, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1870, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1399}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:50:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10965", 'reported_seq': 11098, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:34.412489+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:50:34.412489+0000', 'last_peered': '2024-09-16T04:50:34.412489+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:50:34.412489+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14546188, 'num_objects': 1776, 'num_object_clones': 0, 'num_object_copies': 7104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10963, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -472}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:50:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10647", 'reported_seq': 10758, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:34.543929+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:50:34.543929+0000', 'last_peered': '2024-09-16T04:50:34.543929+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:50:34.543929+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13639680, 'num_objects': 1665, 'num_object_clones': 0, 'num_object_copies': 6660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10647, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -513}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:50:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10934", 'reported_seq': 11032, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:34.061051+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:50:34.061051+0000', 'last_peered': '2024-09-16T04:50:34.061051+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:50:34.061051+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13645068, 'num_objects': 1666, 'num_object_clones': 0, 'num_object_copies': 6664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1666, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10932, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -547}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:35.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:50:35.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10931", 'reported_seq': 11038, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:33.011247+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:50:33.011247+0000', 'last_peered': '2024-09-16T04:50:33.011247+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:50:33.011247+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14963980, 'num_objects': 1827, 'num_object_clones': 0, 'num_object_copies': 7308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1827, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10929, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -362}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:35.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:50:35.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10782", 'reported_seq': 10914, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:34.698708+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:50:34.698708+0000', 'last_peered': '2024-09-16T04:50:34.698708+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:50:34.698708+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13995922, 'num_objects': 1709, 'num_object_clones': 0, 'num_object_copies': 6836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10779, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -446}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:35.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:50:35.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10687", 'reported_seq': 10780, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:34.650964+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:50:34.650964+0000', 'last_peered': '2024-09-16T04:50:34.650964+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:50:34.650964+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13466246, 'num_objects': 1644, 'num_object_clones': 0, 'num_object_copies': 6576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1644, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10686, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -461}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:35.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:50:35.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10790", 'reported_seq': 10879, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:33.084191+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:50:33.084191+0000', 'last_peered': '2024-09-16T04:50:33.084191+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:50:33.084191+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14277254, 'num_objects': 1743, 'num_object_clones': 0, 'num_object_copies': 6972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10789, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -478}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:35.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:50:35.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11029", 'reported_seq': 11162, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:34.325271+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:50:34.325271+0000', 'last_peered': '2024-09-16T04:50:34.325271+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:50:34.325271+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14357772, 'num_objects': 1753, 'num_object_clones': 0, 'num_object_copies': 7012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1753, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11027, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -445}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:35.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:50:35.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10705", 'reported_seq': 10809, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:33.095908+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:50:33.095908+0000', 'last_peered': '2024-09-16T04:50:33.095908+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:50:33.095908+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14284044, 'num_objects': 1744, 'num_object_clones': 0, 'num_object_copies': 6976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10703, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -379}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:35.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:50:35.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10665", 'reported_seq': 10760, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:34.226733+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:50:34.226733+0000', 'last_peered': '2024-09-16T04:50:34.226733+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:50:34.226733+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13448460, 'num_objects': 1642, 'num_object_clones': 0, 'num_object_copies': 6568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10663, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -586}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:35.629 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:50:36.348 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:36.348 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28849/110808 objects degraded (26.035%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":648,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":27702,"data_bytes":226887360,"bytes_used":2851000320,"bytes_avail":383696056320,"bytes_total":386547056640,"degraded_objects":28849,"degraded_total":110808,"degraded_ratio":0.26035123817774891,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":188,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":238},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:50:36.373 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:50:36.374 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:36.974 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:36.974 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:36.999 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-16T04:50:36.745477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":226887360,"num_objects":27702,"num_object_clones":0,"num_object_copies":110808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173785,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6733,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2784180,"kb_used_data":447784,"kb_used_omap":14,"kb_used_meta":2336305,"kb_avail":374703180,"statfs":{"total":386547056640,"available":383696056320,"internally_reserved":0,"allocated":458530816,"data_stored":457774008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2392376881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,12,0,0,8,19,16,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15785984,"num_objects":-1927,"num_object_clones":0,"num_object_copies":-7708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1927,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005062"},"pg_stats":[{"pgid":"1.f","version":"35'10914","reported_seq":11031,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.594311+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:34.594311+0000","last_peered":"2024-09-16T04:50:34.594311+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:34.594311+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14254080,"num_objects":1740,"num_object_clones":0,"num_object_copies":6960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10914,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-446}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10962","reported_seq":11051,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.001916+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:34.001916+0000","last_peered":"2024-09-16T04:50:34.001916+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:34.001916+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14021900,"num_objects":1712,"num_object_clones":0,"num_object_copies":6848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10960,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-491}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10948","reported_seq":11045,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:33.121790+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:33.121790+0000","last_peered":"2024-09-16T04:50:33.121790+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:33.121790+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14105246,"num_objects":1723,"num_object_clones":0,"num_object_copies":6892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1723,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10947,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10908","reported_seq":14718,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.162340+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:34.162340+0000","last_peered":"2024-09-16T04:50:34.162340+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:34.162340+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14431500,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10906,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1870,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1399}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10965","reported_seq":11098,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.412489+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:34.412489+0000","last_peered":"2024-09-16T04:50:34.412489+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:34.412489+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14546188,"num_objects":1776,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10647","reported_seq":10758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.543929+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:34.543929+0000","last_peered":"2024-09-16T04:50:34.543929+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:34.543929+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13639680,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10647,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10934","reported_seq":11032,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.061051+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:34.061051+0000","last_peered":"2024-09-16T04:50:34.061051+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:34.061051+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13645068,"num_objects":1666,"num_object_clones":0,"num_object_copies":6664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10932,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-547}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10930","reported_seq":15492,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:31.795905+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:31.795905+0000","last_peered":"2024-09-16T04:50:31.795905+0000","last_clean":"2024-09-16T04:50:31.795905+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:31.795905+0000","last_undegraded":"2024-09-16T04:50:31.795905+0000","last_fullsized":"2024-09-16T04:50:31.795905+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:32.234512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15193356,"num_objects":1855,"num_object_clones":0,"num_object_copies":7420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10928,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11015","reported_seq":15772,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:34.599937+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:34.599937+0000","last_peered":"2024-09-16T04:50:34.599937+0000","last_clean":"2024-09-16T04:50:34.599937+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:34.599937+0000","last_undegraded":"2024-09-16T04:50:34.599937+0000","last_fullsized":"2024-09-16T04:50:34.599937+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:37.456287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14256664,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11012,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10931","reported_seq":11038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:33.011247+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:33.011247+0000","last_peered":"2024-09-16T04:50:33.011247+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:33.011247+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14963980,"num_objects":1827,"num_object_clones":0,"num_object_copies":7308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10929,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-362}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10782","reported_seq":10914,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.698708+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:34.698708+0000","last_peered":"2024-09-16T04:50:34.698708+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:34.698708+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13995922,"num_objects":1709,"num_object_clones":0,"num_object_copies":6836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10779,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-446}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10687","reported_seq":10780,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.650964+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:34.650964+0000","last_peered":"2024-09-16T04:50:34.650964+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:34.650964+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13466246,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10686,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-461}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10790","reported_seq":10879,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:33.084191+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:33.084191+0000","last_peered":"2024-09-16T04:50:33.084191+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:33.084191+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14277254,"num_objects":1743,"num_object_clones":0,"num_object_copies":6972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10789,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-478}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11029","reported_seq":11162,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.325271+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:34.325271+0000","last_peered":"2024-09-16T04:50:34.325271+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:34.325271+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14357772,"num_objects":1753,"num_object_clones":0,"num_object_copies":7012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11027,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-445}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10705","reported_seq":10809,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:33.095908+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:33.095908+0000","last_peered":"2024-09-16T04:50:33.095908+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:33.095908+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14284044,"num_objects":1744,"num_object_clones":0,"num_object_copies":6976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10703,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10665","reported_seq":10760,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.226733+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:34.226733+0000","last_peered":"2024-09-16T04:50:34.226733+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:34.226733+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13448460,"num_objects":1642,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":226887360,"num_objects":27702,"num_object_clones":0,"num_object_copies":110808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173785,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6733,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":455897088,"data_stored":455897088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728244,"kb_used_data":110688,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93643596,"statfs":{"total":96636764160,"available":95891042304,"internally_reserved":0,"allocated":113344512,"data_stored":113155310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738956,"kb_used_data":112188,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93632884,"statfs":{"total":96636764160,"available":95880073216,"internally_reserved":0,"allocated":114880512,"data_stored":114691310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,12,0,0,0,5,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600804,"kb_used_data":113748,"kb_used_omap":4,"kb_used_meta":487035,"kb_avail":93771036,"statfs":{"total":96636764160,"available":96021540864,"internally_reserved":0,"allocated":116477952,"data_stored":116288750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":498724559},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716176,"kb_used_data":111160,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93655664,"statfs":{"total":96636764160,"available":95903399936,"internally_reserved":0,"allocated":113827840,"data_stored":113638638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":619507406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113172480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":115802112,"data_stored":115802112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114089984,"data_stored":114089984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":112832512,"data_stored":112832512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:40.002 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:40.604 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:40.605 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:40.624 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2024-09-16T04:50:38.746169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":224364224,"num_objects":27394,"num_object_clones":0,"num_object_copies":109576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27394,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174093,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6733,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2784716,"kb_used_data":439968,"kb_used_omap":14,"kb_used_meta":2344625,"kb_avail":374702644,"statfs":{"total":386547056640,"available":383695507456,"internally_reserved":0,"allocated":450527232,"data_stored":449770424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2400896561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,17,15,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17973248,"num_objects":-2194,"num_object_clones":0,"num_object_copies":-8776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2194,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004822"},"pg_stats":[{"pgid":"1.f","version":"35'10914","reported_seq":11031,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.594311+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:34.594311+0000","last_peered":"2024-09-16T04:50:34.594311+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:34.594311+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14254080,"num_objects":1740,"num_object_clones":0,"num_object_copies":6960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10914,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-446}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10962","reported_seq":11051,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.001916+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:34.001916+0000","last_peered":"2024-09-16T04:50:34.001916+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:34.001916+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14021900,"num_objects":1712,"num_object_clones":0,"num_object_copies":6848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10960,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-491}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11001","reported_seq":11098,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:37.946598+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:37.946598+0000","last_peered":"2024-09-16T04:50:37.946598+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:37.946598+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13671070,"num_objects":1670,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1670,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11000,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-549}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10908","reported_seq":14718,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.162340+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:34.162340+0000","last_peered":"2024-09-16T04:50:34.162340+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:34.162340+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14431500,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10906,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1870,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1399}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10965","reported_seq":11098,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.412489+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:34.412489+0000","last_peered":"2024-09-16T04:50:34.412489+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:34.412489+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14546188,"num_objects":1776,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10647","reported_seq":10758,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.543929+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:34.543929+0000","last_peered":"2024-09-16T04:50:34.543929+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:34.543929+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13639680,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10647,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10934","reported_seq":11032,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.061051+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:34.061051+0000","last_peered":"2024-09-16T04:50:34.061051+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:34.061051+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13645068,"num_objects":1666,"num_object_clones":0,"num_object_copies":6664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10932,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-547}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10984","reported_seq":15547,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:36.778101+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:36.778101+0000","last_peered":"2024-09-16T04:50:36.778101+0000","last_clean":"2024-09-16T04:50:36.778101+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:36.778101+0000","last_undegraded":"2024-09-16T04:50:36.778101+0000","last_fullsized":"2024-09-16T04:50:36.778101+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:37.425915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14750988,"num_objects":1801,"num_object_clones":0,"num_object_copies":7204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11015","reported_seq":15772,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:34.599937+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:34.599937+0000","last_peered":"2024-09-16T04:50:34.599937+0000","last_clean":"2024-09-16T04:50:34.599937+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:34.599937+0000","last_undegraded":"2024-09-16T04:50:34.599937+0000","last_fullsized":"2024-09-16T04:50:34.599937+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:37.456287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14256664,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11012,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10997","reported_seq":11103,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:38.209835+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:38.209835+0000","last_peered":"2024-09-16T04:50:38.209835+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:38.209835+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14423308,"num_objects":1761,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10995,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-428}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10782","reported_seq":10914,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.698708+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:34.698708+0000","last_peered":"2024-09-16T04:50:34.698708+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:34.698708+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13995922,"num_objects":1709,"num_object_clones":0,"num_object_copies":6836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10779,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-446}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10687","reported_seq":10780,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.650964+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:34.650964+0000","last_peered":"2024-09-16T04:50:34.650964+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:34.650964+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13466246,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10686,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-461}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10850","reported_seq":10938,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:38.128269+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:38.128269+0000","last_peered":"2024-09-16T04:50:38.128269+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:38.128269+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13785734,"num_objects":1683,"num_object_clones":0,"num_object_copies":6732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10849,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-538}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11029","reported_seq":11162,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.325271+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:34.325271+0000","last_peered":"2024-09-16T04:50:34.325271+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:34.325271+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14357772,"num_objects":1753,"num_object_clones":0,"num_object_copies":7012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11027,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-445}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10780","reported_seq":10883,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:38.115751+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:38.115751+0000","last_peered":"2024-09-16T04:50:38.115751+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:38.115751+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13669644,"num_objects":1669,"num_object_clones":0,"num_object_copies":6676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10778,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-454}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10665","reported_seq":10760,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:34.226733+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:34.226733+0000","last_peered":"2024-09-16T04:50:34.226733+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:34.226733+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13448460,"num_objects":1642,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":224364224,"num_objects":27394,"num_object_clones":0,"num_object_copies":109576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27394,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174093,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6733,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":447807488,"data_stored":447807488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728244,"kb_used_data":110688,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93643596,"statfs":{"total":96636764160,"available":95891042304,"internally_reserved":0,"allocated":113344512,"data_stored":113155310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739308,"kb_used_data":108368,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93632532,"statfs":{"total":96636764160,"available":95879712768,"internally_reserved":0,"allocated":110968832,"data_stored":110779630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600988,"kb_used_data":109752,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93770852,"statfs":{"total":96636764160,"available":96021352448,"internally_reserved":0,"allocated":112386048,"data_stored":112196846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":502984399},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716176,"kb_used_data":111160,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93655664,"statfs":{"total":96636764160,"available":95903399936,"internally_reserved":0,"allocated":113827840,"data_stored":113638638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":619507406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113172480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111710208,"data_stored":111710208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110092288,"data_stored":110092288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":112832512,"data_stored":112832512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:40.625 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:41.199 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:41.200 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:41.232 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-16T04:50:40.746851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":218736320,"num_objects":26707,"num_object_clones":0,"num_object_copies":106828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174780,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6804,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160907,"ondisk_log_size":160907,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2768708,"kb_used_data":432292,"kb_used_omap":14,"kb_used_meta":2336305,"kb_avail":374718652,"statfs":{"total":386547056640,"available":383711899648,"internally_reserved":0,"allocated":442667008,"data_stored":441910200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2392376881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,1,1,3,25,14,22,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21553152,"num_objects":-2631,"num_object_clones":0,"num_object_copies":-10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2631,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004692"},"pg_stats":[{"pgid":"1.f","version":"35'10985","reported_seq":11102,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.662438+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:39.662438+0000","last_peered":"2024-09-16T04:50:39.662438+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:39.662438+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13672448,"num_objects":1669,"num_object_clones":0,"num_object_copies":6676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10985,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11013","reported_seq":11102,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.124964+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:39.124964+0000","last_peered":"2024-09-16T04:50:39.124964+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:39.124964+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13604108,"num_objects":1661,"num_object_clones":0,"num_object_copies":6644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11011,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-542}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11001","reported_seq":11098,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:37.946598+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:37.946598+0000","last_peered":"2024-09-16T04:50:37.946598+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:37.946598+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13671070,"num_objects":1670,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1670,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11000,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-549}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10968","reported_seq":14920,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.313153+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:39.313153+0000","last_peered":"2024-09-16T04:50:39.313153+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:39.313153+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13939980,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10966,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11026","reported_seq":11159,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.650864+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:39.650864+0000","last_peered":"2024-09-16T04:50:39.650864+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:39.650864+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14046476,"num_objects":1715,"num_object_clones":0,"num_object_copies":6860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11024,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10709","reported_seq":10820,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.795605+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:39.795605+0000","last_peered":"2024-09-16T04:50:39.795605+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:39.795605+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13131776,"num_objects":1603,"num_object_clones":0,"num_object_copies":6412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10709,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10996","reported_seq":11093,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.328408+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:39.328408+0000","last_peered":"2024-09-16T04:50:39.328408+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:39.328408+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13137164,"num_objects":1604,"num_object_clones":0,"num_object_copies":6416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10994,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-609}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10984","reported_seq":15547,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:36.778101+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:36.778101+0000","last_peered":"2024-09-16T04:50:36.778101+0000","last_clean":"2024-09-16T04:50:36.778101+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:36.778101+0000","last_undegraded":"2024-09-16T04:50:36.778101+0000","last_fullsized":"2024-09-16T04:50:36.778101+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:37.425915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14750988,"num_objects":1801,"num_object_clones":0,"num_object_copies":7204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11088","reported_seq":15846,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:39.781197+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:39.781197+0000","last_peered":"2024-09-16T04:50:39.781197+0000","last_clean":"2024-09-16T04:50:39.781197+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:39.781197+0000","last_undegraded":"2024-09-16T04:50:39.781197+0000","last_fullsized":"2024-09-16T04:50:39.781197+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:43.437176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13658648,"num_objects":1668,"num_object_clones":0,"num_object_copies":6672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11085,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10997","reported_seq":11103,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:38.209835+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:38.209835+0000","last_peered":"2024-09-16T04:50:38.209835+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:38.209835+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14423308,"num_objects":1761,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10995,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-428}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10846","reported_seq":10979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.753526+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:39.753526+0000","last_peered":"2024-09-16T04:50:39.753526+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:39.753526+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13471634,"num_objects":1645,"num_object_clones":0,"num_object_copies":6580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10843,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-510}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10750","reported_seq":10841,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.732777+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:39.732777+0000","last_peered":"2024-09-16T04:50:39.732777+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:39.732777+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12950150,"num_objects":1581,"num_object_clones":0,"num_object_copies":6324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10749,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-524}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10850","reported_seq":10938,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:38.128269+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:38.128269+0000","last_peered":"2024-09-16T04:50:38.128269+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:38.128269+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13785734,"num_objects":1683,"num_object_clones":0,"num_object_copies":6732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10849,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-538}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11087","reported_seq":11220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.665496+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:39.665496+0000","last_peered":"2024-09-16T04:50:39.665496+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:39.665496+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13882636,"num_objects":1695,"num_object_clones":0,"num_object_copies":6780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11085,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-503}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10780","reported_seq":10883,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:38.115751+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:38.115751+0000","last_peered":"2024-09-16T04:50:38.115751+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:38.115751+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13669644,"num_objects":1669,"num_object_clones":0,"num_object_copies":6676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10778,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-454}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10727","reported_seq":10822,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.226629+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:39.226629+0000","last_peered":"2024-09-16T04:50:39.226629+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:39.226629+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12940556,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10725,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-648}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":218736320,"num_objects":26707,"num_object_clones":0,"num_object_copies":106828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174780,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6804,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":439754752,"data_stored":439754752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160907,"ondisk_log_size":160907,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711928,"kb_used_data":106856,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93659912,"statfs":{"total":96636764160,"available":95907749888,"internally_reserved":0,"allocated":109420544,"data_stored":109231342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619572943},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,0,11,1,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739308,"kb_used_data":108368,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93632532,"statfs":{"total":96636764160,"available":95879712768,"internally_reserved":0,"allocated":110968832,"data_stored":110779630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600988,"kb_used_data":109752,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93770852,"statfs":{"total":96636764160,"available":96021352448,"internally_reserved":0,"allocated":112386048,"data_stored":112196846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":502984399},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716484,"kb_used_data":107316,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93655356,"statfs":{"total":96636764160,"available":95903084544,"internally_reserved":0,"allocated":109891584,"data_stored":109702382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":623767246},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,8,6,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":109174784,"data_stored":109174784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111710208,"data_stored":111710208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110092288,"data_stored":110092288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":108777472,"data_stored":108777472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:41.233 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:50:41.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10985", 'reported_seq': 11102, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:39.662438+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:50:39.662438+0000', 'last_peered': '2024-09-16T04:50:39.662438+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:50:39.662438+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13672448, 'num_objects': 1669, 'num_object_clones': 0, 'num_object_copies': 6676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1669, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10985, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -517}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:41.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:50:41.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11013", 'reported_seq': 11102, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:39.124964+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:50:39.124964+0000', 'last_peered': '2024-09-16T04:50:39.124964+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:50:39.124964+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13604108, 'num_objects': 1661, 'num_object_clones': 0, 'num_object_copies': 6644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1661, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11011, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -542}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:41.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:50:41.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11001", 'reported_seq': 11098, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:37.946598+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:50:37.946598+0000', 'last_peered': '2024-09-16T04:50:37.946598+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:50:37.946598+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13671070, 'num_objects': 1670, 'num_object_clones': 0, 'num_object_copies': 6680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1670, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11000, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -549}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:41.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:50:41.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10968", 'reported_seq': 14920, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:39.313153+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:50:39.313153+0000', 'last_peered': '2024-09-16T04:50:39.313153+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:50:39.313153+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13939980, 'num_objects': 1702, 'num_object_clones': 0, 'num_object_copies': 6808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 292, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1702, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10966, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1941, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1410}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:41.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:50:41.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11026", 'reported_seq': 11159, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:39.650864+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:50:39.650864+0000', 'last_peered': '2024-09-16T04:50:39.650864+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:50:39.650864+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14046476, 'num_objects': 1715, 'num_object_clones': 0, 'num_object_copies': 6860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11024, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -533}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:41.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:50:41.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10709", 'reported_seq': 10820, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:39.795605+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:50:39.795605+0000', 'last_peered': '2024-09-16T04:50:39.795605+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:50:39.795605+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13131776, 'num_objects': 1603, 'num_object_clones': 0, 'num_object_copies': 6412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10709, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -575}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:41.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:50:41.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10996", 'reported_seq': 11093, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:39.328408+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:50:39.328408+0000', 'last_peered': '2024-09-16T04:50:39.328408+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:50:39.328408+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13137164, 'num_objects': 1604, 'num_object_clones': 0, 'num_object_copies': 6416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10994, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -609}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:41.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:50:41.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10997", 'reported_seq': 11103, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:38.209835+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:50:38.209835+0000', 'last_peered': '2024-09-16T04:50:38.209835+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:50:38.209835+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14423308, 'num_objects': 1761, 'num_object_clones': 0, 'num_object_copies': 7044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10995, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -428}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:41.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:50:41.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10846", 'reported_seq': 10979, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:39.753526+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:50:39.753526+0000', 'last_peered': '2024-09-16T04:50:39.753526+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:50:39.753526+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13471634, 'num_objects': 1645, 'num_object_clones': 0, 'num_object_copies': 6580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1645, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10843, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -510}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:41.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:50:41.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10750", 'reported_seq': 10841, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:39.732777+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:50:39.732777+0000', 'last_peered': '2024-09-16T04:50:39.732777+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:50:39.732777+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12950150, 'num_objects': 1581, 'num_object_clones': 0, 'num_object_copies': 6324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10749, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -524}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:41.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:50:41.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10850", 'reported_seq': 10938, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:38.128269+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:50:38.128269+0000', 'last_peered': '2024-09-16T04:50:38.128269+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:50:38.128269+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13785734, 'num_objects': 1683, 'num_object_clones': 0, 'num_object_copies': 6732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1683, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10849, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -538}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:41.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:50:41.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11087", 'reported_seq': 11220, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:39.665496+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:50:39.665496+0000', 'last_peered': '2024-09-16T04:50:39.665496+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:50:39.665496+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13882636, 'num_objects': 1695, 'num_object_clones': 0, 'num_object_copies': 6780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1695, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11085, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -503}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:41.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:50:41.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10780", 'reported_seq': 10883, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:38.115751+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:50:38.115751+0000', 'last_peered': '2024-09-16T04:50:38.115751+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:50:38.115751+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13669644, 'num_objects': 1669, 'num_object_clones': 0, 'num_object_copies': 6676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1669, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10778, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -454}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:41.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:50:41.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10727", 'reported_seq': 10822, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:39.226629+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:50:39.226629+0000', 'last_peered': '2024-09-16T04:50:39.226629+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:50:39.226629+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12940556, 'num_objects': 1580, 'num_object_clones': 0, 'num_object_copies': 6320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1580, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10725, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -648}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:41.237 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:50:41.907 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:42.035 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28778/106828 objects degraded (26.939%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":654,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":26707,"data_bytes":218736320,"bytes_used":2835156992,"bytes_avail":383711899648,"bytes_total":386547056640,"degraded_objects":28778,"degraded_total":106828,"degraded_ratio":0.26938630321638524,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":198,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":219},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:50:42.036 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:50:42.036 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:42.639 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:42.640 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:42.667 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-16T04:50:40.746851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":218736320,"num_objects":26707,"num_object_clones":0,"num_object_copies":106828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174780,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6804,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160907,"ondisk_log_size":160907,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2768708,"kb_used_data":432292,"kb_used_omap":14,"kb_used_meta":2336305,"kb_avail":374718652,"statfs":{"total":386547056640,"available":383711899648,"internally_reserved":0,"allocated":442667008,"data_stored":441910200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2392376881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,1,1,3,25,14,22,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21553152,"num_objects":-2631,"num_object_clones":0,"num_object_copies":-10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2631,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004692"},"pg_stats":[{"pgid":"1.f","version":"35'10985","reported_seq":11102,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.662438+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:39.662438+0000","last_peered":"2024-09-16T04:50:39.662438+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:39.662438+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13672448,"num_objects":1669,"num_object_clones":0,"num_object_copies":6676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10985,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11013","reported_seq":11102,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.124964+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:39.124964+0000","last_peered":"2024-09-16T04:50:39.124964+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:39.124964+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13604108,"num_objects":1661,"num_object_clones":0,"num_object_copies":6644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11011,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-542}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11001","reported_seq":11098,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:37.946598+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:37.946598+0000","last_peered":"2024-09-16T04:50:37.946598+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:37.946598+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13671070,"num_objects":1670,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1670,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11000,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-549}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10968","reported_seq":14920,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.313153+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:39.313153+0000","last_peered":"2024-09-16T04:50:39.313153+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:39.313153+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13939980,"num_objects":1702,"num_object_clones":0,"num_object_copies":6808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10966,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11026","reported_seq":11159,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.650864+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:39.650864+0000","last_peered":"2024-09-16T04:50:39.650864+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:39.650864+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14046476,"num_objects":1715,"num_object_clones":0,"num_object_copies":6860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11024,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10709","reported_seq":10820,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.795605+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:39.795605+0000","last_peered":"2024-09-16T04:50:39.795605+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:39.795605+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13131776,"num_objects":1603,"num_object_clones":0,"num_object_copies":6412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10709,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10996","reported_seq":11093,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.328408+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:39.328408+0000","last_peered":"2024-09-16T04:50:39.328408+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:39.328408+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13137164,"num_objects":1604,"num_object_clones":0,"num_object_copies":6416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10994,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-609}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10984","reported_seq":15547,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:36.778101+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:36.778101+0000","last_peered":"2024-09-16T04:50:36.778101+0000","last_clean":"2024-09-16T04:50:36.778101+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:36.778101+0000","last_undegraded":"2024-09-16T04:50:36.778101+0000","last_fullsized":"2024-09-16T04:50:36.778101+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:37.425915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14750988,"num_objects":1801,"num_object_clones":0,"num_object_copies":7204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11088","reported_seq":15846,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:39.781197+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:39.781197+0000","last_peered":"2024-09-16T04:50:39.781197+0000","last_clean":"2024-09-16T04:50:39.781197+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:39.781197+0000","last_undegraded":"2024-09-16T04:50:39.781197+0000","last_fullsized":"2024-09-16T04:50:39.781197+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:43.437176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13658648,"num_objects":1668,"num_object_clones":0,"num_object_copies":6672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11085,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10997","reported_seq":11103,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:38.209835+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:38.209835+0000","last_peered":"2024-09-16T04:50:38.209835+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:38.209835+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14423308,"num_objects":1761,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10995,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-428}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10846","reported_seq":10979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.753526+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:39.753526+0000","last_peered":"2024-09-16T04:50:39.753526+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:39.753526+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13471634,"num_objects":1645,"num_object_clones":0,"num_object_copies":6580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10843,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-510}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10750","reported_seq":10841,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.732777+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:39.732777+0000","last_peered":"2024-09-16T04:50:39.732777+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:39.732777+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12950150,"num_objects":1581,"num_object_clones":0,"num_object_copies":6324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10749,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-524}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10850","reported_seq":10938,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:38.128269+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:38.128269+0000","last_peered":"2024-09-16T04:50:38.128269+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:38.128269+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13785734,"num_objects":1683,"num_object_clones":0,"num_object_copies":6732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10849,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-538}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11087","reported_seq":11220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.665496+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:39.665496+0000","last_peered":"2024-09-16T04:50:39.665496+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:39.665496+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13882636,"num_objects":1695,"num_object_clones":0,"num_object_copies":6780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11085,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-503}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10780","reported_seq":10883,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:38.115751+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:38.115751+0000","last_peered":"2024-09-16T04:50:38.115751+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:38.115751+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13669644,"num_objects":1669,"num_object_clones":0,"num_object_copies":6676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10778,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-454}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10727","reported_seq":10822,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.226629+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:39.226629+0000","last_peered":"2024-09-16T04:50:39.226629+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:39.226629+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12940556,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10725,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-648}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":218736320,"num_objects":26707,"num_object_clones":0,"num_object_copies":106828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174780,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6804,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":439754752,"data_stored":439754752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160907,"ondisk_log_size":160907,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711928,"kb_used_data":106856,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93659912,"statfs":{"total":96636764160,"available":95907749888,"internally_reserved":0,"allocated":109420544,"data_stored":109231342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619572943},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,0,11,1,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739308,"kb_used_data":108368,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93632532,"statfs":{"total":96636764160,"available":95879712768,"internally_reserved":0,"allocated":110968832,"data_stored":110779630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600988,"kb_used_data":109752,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93770852,"statfs":{"total":96636764160,"available":96021352448,"internally_reserved":0,"allocated":112386048,"data_stored":112196846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":502984399},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716484,"kb_used_data":107316,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93655356,"statfs":{"total":96636764160,"available":95903084544,"internally_reserved":0,"allocated":109891584,"data_stored":109702382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":623767246},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,8,6,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":109174784,"data_stored":109174784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111710208,"data_stored":111710208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110092288,"data_stored":110092288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":108777472,"data_stored":108777472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:45.669 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:46.298 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:46.298 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:46.326 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-09-16T04:50:44.748607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":214091456,"num_objects":26140,"num_object_clones":0,"num_object_copies":104560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175347,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6865,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160874,"ondisk_log_size":160874,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2807168,"kb_used_data":420748,"kb_used_omap":14,"kb_used_meta":2386289,"kb_avail":374680192,"statfs":{"total":386547056640,"available":383672516608,"internally_reserved":0,"allocated":430845952,"data_stored":430089144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2443560497},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,7,4,16,15,20,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20307968,"num_objects":-2479,"num_object_clones":0,"num_object_copies":-9916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2479,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004828"},"pg_stats":[{"pgid":"1.f","version":"35'10985","reported_seq":11102,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.662438+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:39.662438+0000","last_peered":"2024-09-16T04:50:39.662438+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:39.662438+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13672448,"num_objects":1669,"num_object_clones":0,"num_object_copies":6676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10985,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11085","reported_seq":11174,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.342732+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:44.342732+0000","last_peered":"2024-09-16T04:50:44.342732+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:44.342732+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13014284,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11083,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11060","reported_seq":11158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:43.160102+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:43.160102+0000","last_peered":"2024-09-16T04:50:43.160102+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:43.160102+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13187742,"num_objects":1611,"num_object_clones":0,"num_object_copies":6444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11059,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-608}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11029","reported_seq":15102,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.366981+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:44.366981+0000","last_peered":"2024-09-16T04:50:44.366981+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:44.366981+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13440268,"num_objects":1641,"num_object_clones":0,"num_object_copies":6564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11027,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11026","reported_seq":11159,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.650864+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:39.650864+0000","last_peered":"2024-09-16T04:50:39.650864+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:39.650864+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14046476,"num_objects":1715,"num_object_clones":0,"num_object_copies":6860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11024,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10709","reported_seq":10820,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.795605+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:39.795605+0000","last_peered":"2024-09-16T04:50:39.795605+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:39.795605+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13131776,"num_objects":1603,"num_object_clones":0,"num_object_copies":6412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10709,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11065","reported_seq":11163,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.267708+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:44.267708+0000","last_peered":"2024-09-16T04:50:44.267708+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:44.267708+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12571916,"num_objects":1535,"num_object_clones":0,"num_object_copies":6140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11063,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-678}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11038","reported_seq":15601,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:41.863232+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:41.863232+0000","last_peered":"2024-09-16T04:50:41.863232+0000","last_clean":"2024-09-16T04:50:41.863232+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:41.863232+0000","last_undegraded":"2024-09-16T04:50:41.863232+0000","last_fullsized":"2024-09-16T04:50:41.863232+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:42.474806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14308620,"num_objects":1747,"num_object_clones":0,"num_object_copies":6988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11036,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11088","reported_seq":15846,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:39.781197+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:39.781197+0000","last_peered":"2024-09-16T04:50:39.781197+0000","last_clean":"2024-09-16T04:50:39.781197+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:39.781197+0000","last_undegraded":"2024-09-16T04:50:39.781197+0000","last_fullsized":"2024-09-16T04:50:39.781197+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:43.437176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13658648,"num_objects":1668,"num_object_clones":0,"num_object_copies":6672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11085,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11070","reported_seq":11178,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:43.289235+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:43.289235+0000","last_peered":"2024-09-16T04:50:43.289235+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:43.289235+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13825292,"num_objects":1688,"num_object_clones":0,"num_object_copies":6752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11068,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-501}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10846","reported_seq":10979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.753526+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:39.753526+0000","last_peered":"2024-09-16T04:50:39.753526+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:39.753526+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13471634,"num_objects":1645,"num_object_clones":0,"num_object_copies":6580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10843,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-510}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10750","reported_seq":10841,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.732777+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:39.732777+0000","last_peered":"2024-09-16T04:50:39.732777+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:39.732777+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12950150,"num_objects":1581,"num_object_clones":0,"num_object_copies":6324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10749,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-524}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10904","reported_seq":10993,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:43.053899+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:43.053899+0000","last_peered":"2024-09-16T04:50:43.053899+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:43.053899+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13343366,"num_objects":1629,"num_object_clones":0,"num_object_copies":6516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10903,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11087","reported_seq":11220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:39.665496+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:39.665496+0000","last_peered":"2024-09-16T04:50:39.665496+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:39.665496+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13882636,"num_objects":1695,"num_object_clones":0,"num_object_copies":6780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11085,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-503}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10838","reported_seq":10942,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:42.913243+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:42.913243+0000","last_peered":"2024-09-16T04:50:42.913243+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:42.913243+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13194508,"num_objects":1611,"num_object_clones":0,"num_object_copies":6444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10836,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-512}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10794","reported_seq":10889,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.361878+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:44.361878+0000","last_peered":"2024-09-16T04:50:44.361878+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:44.361878+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12391692,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10792,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-715}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":214091456,"num_objects":26140,"num_object_clones":0,"num_object_copies":104560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175347,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6865,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":427667456,"data_stored":427667456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160874,"ondisk_log_size":160874,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711928,"kb_used_data":106856,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93659912,"statfs":{"total":96636764160,"available":95907749888,"internally_reserved":0,"allocated":109420544,"data_stored":109231342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619572943},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,0,11,1,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757724,"kb_used_data":104576,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93614116,"statfs":{"total":96636764160,"available":95860854784,"internally_reserved":0,"allocated":107085824,"data_stored":106896622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602664,"kb_used_data":105852,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93769176,"statfs":{"total":96636764160,"available":96019636224,"internally_reserved":0,"allocated":108392448,"data_stored":108203246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508686031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734852,"kb_used_data":103464,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93636988,"statfs":{"total":96636764160,"available":95884275712,"internally_reserved":0,"allocated":105947136,"data_stored":105757934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":646508238},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":105189376,"data_stored":105189376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107601920,"data_stored":107601920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106098688,"data_stored":106098688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":108777472,"data_stored":108777472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:46.328 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:46.926 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:46.926 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:46.958 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-16T04:50:46.749563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":210511552,"num_objects":25703,"num_object_clones":0,"num_object_copies":102812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25703,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175784,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6865,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160911,"ondisk_log_size":160911,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2825516,"kb_used_data":416876,"kb_used_omap":14,"kb_used_meta":2408497,"kb_avail":374661844,"statfs":{"total":386547056640,"available":383653728256,"internally_reserved":0,"allocated":426881024,"data_stored":426124216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2466301489},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,11,6,7,15,15,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16375808,"num_objects":-1999,"num_object_clones":0,"num_object_copies":-7996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1999,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004986"},"pg_stats":[{"pgid":"1.f","version":"35'11051","reported_seq":11166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.729073+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:44.729073+0000","last_peered":"2024-09-16T04:50:44.729073+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:44.729073+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13131776,"num_objects":1603,"num_object_clones":0,"num_object_copies":6412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11051,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11085","reported_seq":11174,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.342732+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:44.342732+0000","last_peered":"2024-09-16T04:50:44.342732+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:44.342732+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13014284,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11083,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11060","reported_seq":11158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:43.160102+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:43.160102+0000","last_peered":"2024-09-16T04:50:43.160102+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:43.160102+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13187742,"num_objects":1611,"num_object_clones":0,"num_object_copies":6444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11059,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-608}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11029","reported_seq":15102,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.366981+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:44.366981+0000","last_peered":"2024-09-16T04:50:44.366981+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:44.366981+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13440268,"num_objects":1641,"num_object_clones":0,"num_object_copies":6564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11027,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11089","reported_seq":11222,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.911611+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:44.911611+0000","last_peered":"2024-09-16T04:50:44.911611+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:44.911611+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13530380,"num_objects":1652,"num_object_clones":0,"num_object_copies":6608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11087,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-596}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10771","reported_seq":10882,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.864960+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:44.864960+0000","last_peered":"2024-09-16T04:50:44.864960+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:44.864960+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12623872,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10771,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-637}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11065","reported_seq":11163,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.267708+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:44.267708+0000","last_peered":"2024-09-16T04:50:44.267708+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:44.267708+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12571916,"num_objects":1535,"num_object_clones":0,"num_object_copies":6140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11063,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-678}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11038","reported_seq":15601,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:41.863232+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:41.863232+0000","last_peered":"2024-09-16T04:50:41.863232+0000","last_clean":"2024-09-16T04:50:41.863232+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:41.863232+0000","last_undegraded":"2024-09-16T04:50:41.863232+0000","last_fullsized":"2024-09-16T04:50:41.863232+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:42.474806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14308620,"num_objects":1747,"num_object_clones":0,"num_object_copies":6988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11036,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11166","reported_seq":15924,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:44.866109+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:44.866109+0000","last_peered":"2024-09-16T04:50:44.866109+0000","last_clean":"2024-09-16T04:50:44.866109+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:44.866109+0000","last_undegraded":"2024-09-16T04:50:44.866109+0000","last_fullsized":"2024-09-16T04:50:44.866109+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:49.450807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13019672,"num_objects":1590,"num_object_clones":0,"num_object_copies":6360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11163,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11070","reported_seq":11178,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:43.289235+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:43.289235+0000","last_peered":"2024-09-16T04:50:43.289235+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:43.289235+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13825292,"num_objects":1688,"num_object_clones":0,"num_object_copies":6752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11068,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-501}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10906","reported_seq":11039,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.822247+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:44.822247+0000","last_peered":"2024-09-16T04:50:44.822247+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:44.822247+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12980114,"num_objects":1585,"num_object_clones":0,"num_object_copies":6340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10903,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10793","reported_seq":10886,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.870343+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:44.870343+0000","last_peered":"2024-09-16T04:50:44.870343+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:44.870343+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12597894,"num_objects":1538,"num_object_clones":0,"num_object_copies":6152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10792,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-567}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10904","reported_seq":10993,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:43.053899+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:43.053899+0000","last_peered":"2024-09-16T04:50:43.053899+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:43.053899+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13343366,"num_objects":1629,"num_object_clones":0,"num_object_copies":6516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10903,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11152","reported_seq":11285,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.815430+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:44.815430+0000","last_peered":"2024-09-16T04:50:44.815430+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:44.815430+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13350156,"num_objects":1630,"num_object_clones":0,"num_object_copies":6520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11150,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-568}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10838","reported_seq":10942,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:42.913243+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:42.913243+0000","last_peered":"2024-09-16T04:50:42.913243+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:42.913243+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13194508,"num_objects":1611,"num_object_clones":0,"num_object_copies":6444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10836,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-512}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10794","reported_seq":10889,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.361878+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:44.361878+0000","last_peered":"2024-09-16T04:50:44.361878+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:44.361878+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12391692,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10792,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-715}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":210511552,"num_objects":25703,"num_object_clones":0,"num_object_copies":102812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25703,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175784,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6865,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":423636992,"data_stored":423636992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160911,"ondisk_log_size":160911,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730276,"kb_used_data":102984,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93641564,"statfs":{"total":96636764160,"available":95888961536,"internally_reserved":0,"allocated":105455616,"data_stored":105266414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,4,2,2,1,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757724,"kb_used_data":104576,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93614116,"statfs":{"total":96636764160,"available":95860854784,"internally_reserved":0,"allocated":107085824,"data_stored":106896622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602664,"kb_used_data":105852,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93769176,"statfs":{"total":96636764160,"available":96019636224,"internally_reserved":0,"allocated":108392448,"data_stored":108203246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508686031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734852,"kb_used_data":103464,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93636988,"statfs":{"total":96636764160,"available":95884275712,"internally_reserved":0,"allocated":105947136,"data_stored":105757934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":646508238},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":105189376,"data_stored":105189376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107601920,"data_stored":107601920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106098688,"data_stored":106098688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":104747008,"data_stored":104747008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:46.959 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:50:46.959 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11051", 'reported_seq': 11166, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:44.729073+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:50:44.729073+0000', 'last_peered': '2024-09-16T04:50:44.729073+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:50:44.729073+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13131776, 'num_objects': 1603, 'num_object_clones': 0, 'num_object_copies': 6412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11051, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -583}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:46.959 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:50:46.959 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11085", 'reported_seq': 11174, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:44.342732+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:50:44.342732+0000', 'last_peered': '2024-09-16T04:50:44.342732+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:50:44.342732+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13014284, 'num_objects': 1589, 'num_object_clones': 0, 'num_object_copies': 6356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11083, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -614}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:46.959 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:50:46.960 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11060", 'reported_seq': 11158, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:43.160102+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:50:43.160102+0000', 'last_peered': '2024-09-16T04:50:43.160102+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:50:43.160102+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13187742, 'num_objects': 1611, 'num_object_clones': 0, 'num_object_copies': 6444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1611, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11059, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -608}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:46.960 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:50:46.960 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11029", 'reported_seq': 15102, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:44.366981+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:50:44.366981+0000', 'last_peered': '2024-09-16T04:50:44.366981+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:50:44.366981+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13440268, 'num_objects': 1641, 'num_object_clones': 0, 'num_object_copies': 6564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 231, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1641, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11027, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2002, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1410}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:46.960 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:50:46.960 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11089", 'reported_seq': 11222, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:44.911611+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:50:44.911611+0000', 'last_peered': '2024-09-16T04:50:44.911611+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:50:44.911611+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13530380, 'num_objects': 1652, 'num_object_clones': 0, 'num_object_copies': 6608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1652, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11087, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -596}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:46.960 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:50:46.960 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10771", 'reported_seq': 10882, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:44.864960+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:50:44.864960+0000', 'last_peered': '2024-09-16T04:50:44.864960+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:50:44.864960+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12623872, 'num_objects': 1541, 'num_object_clones': 0, 'num_object_copies': 6164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10771, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -637}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:46.960 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:50:46.960 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11065", 'reported_seq': 11163, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:44.267708+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:50:44.267708+0000', 'last_peered': '2024-09-16T04:50:44.267708+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:50:44.267708+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12571916, 'num_objects': 1535, 'num_object_clones': 0, 'num_object_copies': 6140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1535, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11063, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -678}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:46.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:50:46.961 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11070", 'reported_seq': 11178, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:43.289235+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:50:43.289235+0000', 'last_peered': '2024-09-16T04:50:43.289235+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:50:43.289235+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13825292, 'num_objects': 1688, 'num_object_clones': 0, 'num_object_copies': 6752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11068, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -501}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:46.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:50:46.961 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10906", 'reported_seq': 11039, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:44.822247+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:50:44.822247+0000', 'last_peered': '2024-09-16T04:50:44.822247+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:50:44.822247+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12980114, 'num_objects': 1585, 'num_object_clones': 0, 'num_object_copies': 6340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10903, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -570}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:46.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:50:46.961 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10793", 'reported_seq': 10886, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:44.870343+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:50:44.870343+0000', 'last_peered': '2024-09-16T04:50:44.870343+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:50:44.870343+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12597894, 'num_objects': 1538, 'num_object_clones': 0, 'num_object_copies': 6152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1538, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10792, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -567}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:46.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:50:46.961 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10904", 'reported_seq': 10993, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:43.053899+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:50:43.053899+0000', 'last_peered': '2024-09-16T04:50:43.053899+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:50:43.053899+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13343366, 'num_objects': 1629, 'num_object_clones': 0, 'num_object_copies': 6516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1629, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10903, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -592}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:46.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:50:46.961 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11152", 'reported_seq': 11285, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:44.815430+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:50:44.815430+0000', 'last_peered': '2024-09-16T04:50:44.815430+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:50:44.815430+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13350156, 'num_objects': 1630, 'num_object_clones': 0, 'num_object_copies': 6520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11150, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -568}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:46.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:50:46.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10838", 'reported_seq': 10942, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:42.913243+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:50:42.913243+0000', 'last_peered': '2024-09-16T04:50:42.913243+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:50:42.913243+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13194508, 'num_objects': 1611, 'num_object_clones': 0, 'num_object_copies': 6444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1611, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10836, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -512}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:46.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:50:46.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10794", 'reported_seq': 10889, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:44.361878+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:50:44.361878+0000', 'last_peered': '2024-09-16T04:50:44.361878+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:50:44.361878+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12391692, 'num_objects': 1513, 'num_object_clones': 0, 'num_object_copies': 6052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10792, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -715}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:46.962 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:50:47.730 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:47.731 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28717/102812 objects degraded (27.932%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":660,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":25703,"data_bytes":210511552,"bytes_used":2893328384,"bytes_avail":383653728256,"bytes_total":386547056640,"degraded_objects":28717,"degraded_total":102812,"degraded_ratio":0.27931564408823872,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":132,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":166},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:50:47.754 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:50:47.754 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:48.327 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:48.328 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:48.359 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-16T04:50:46.749563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":210511552,"num_objects":25703,"num_object_clones":0,"num_object_copies":102812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25703,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175784,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6865,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160911,"ondisk_log_size":160911,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2825516,"kb_used_data":416876,"kb_used_omap":14,"kb_used_meta":2408497,"kb_avail":374661844,"statfs":{"total":386547056640,"available":383653728256,"internally_reserved":0,"allocated":426881024,"data_stored":426124216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2466301489},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,11,6,7,15,15,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16375808,"num_objects":-1999,"num_object_clones":0,"num_object_copies":-7996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1999,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004986"},"pg_stats":[{"pgid":"1.f","version":"35'11051","reported_seq":11166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.729073+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:44.729073+0000","last_peered":"2024-09-16T04:50:44.729073+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:44.729073+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13131776,"num_objects":1603,"num_object_clones":0,"num_object_copies":6412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11051,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11085","reported_seq":11174,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.342732+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:44.342732+0000","last_peered":"2024-09-16T04:50:44.342732+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:44.342732+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13014284,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11083,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11060","reported_seq":11158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:43.160102+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:43.160102+0000","last_peered":"2024-09-16T04:50:43.160102+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:43.160102+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13187742,"num_objects":1611,"num_object_clones":0,"num_object_copies":6444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11059,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-608}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11029","reported_seq":15102,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.366981+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:44.366981+0000","last_peered":"2024-09-16T04:50:44.366981+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:44.366981+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13440268,"num_objects":1641,"num_object_clones":0,"num_object_copies":6564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11027,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11089","reported_seq":11222,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.911611+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:44.911611+0000","last_peered":"2024-09-16T04:50:44.911611+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:44.911611+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13530380,"num_objects":1652,"num_object_clones":0,"num_object_copies":6608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11087,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-596}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10771","reported_seq":10882,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.864960+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:44.864960+0000","last_peered":"2024-09-16T04:50:44.864960+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:44.864960+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12623872,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10771,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-637}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11065","reported_seq":11163,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.267708+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:44.267708+0000","last_peered":"2024-09-16T04:50:44.267708+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:44.267708+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12571916,"num_objects":1535,"num_object_clones":0,"num_object_copies":6140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11063,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-678}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11038","reported_seq":15601,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:41.863232+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:41.863232+0000","last_peered":"2024-09-16T04:50:41.863232+0000","last_clean":"2024-09-16T04:50:41.863232+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:41.863232+0000","last_undegraded":"2024-09-16T04:50:41.863232+0000","last_fullsized":"2024-09-16T04:50:41.863232+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:42.474806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14308620,"num_objects":1747,"num_object_clones":0,"num_object_copies":6988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11036,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11166","reported_seq":15924,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:44.866109+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:44.866109+0000","last_peered":"2024-09-16T04:50:44.866109+0000","last_clean":"2024-09-16T04:50:44.866109+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:44.866109+0000","last_undegraded":"2024-09-16T04:50:44.866109+0000","last_fullsized":"2024-09-16T04:50:44.866109+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:49.450807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13019672,"num_objects":1590,"num_object_clones":0,"num_object_copies":6360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11163,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11070","reported_seq":11178,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:43.289235+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:43.289235+0000","last_peered":"2024-09-16T04:50:43.289235+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:43.289235+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13825292,"num_objects":1688,"num_object_clones":0,"num_object_copies":6752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11068,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-501}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10906","reported_seq":11039,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.822247+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:44.822247+0000","last_peered":"2024-09-16T04:50:44.822247+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:44.822247+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12980114,"num_objects":1585,"num_object_clones":0,"num_object_copies":6340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10903,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10793","reported_seq":10886,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.870343+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:44.870343+0000","last_peered":"2024-09-16T04:50:44.870343+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:44.870343+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12597894,"num_objects":1538,"num_object_clones":0,"num_object_copies":6152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10792,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-567}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10904","reported_seq":10993,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:43.053899+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:43.053899+0000","last_peered":"2024-09-16T04:50:43.053899+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:43.053899+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13343366,"num_objects":1629,"num_object_clones":0,"num_object_copies":6516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10903,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11152","reported_seq":11285,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.815430+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:44.815430+0000","last_peered":"2024-09-16T04:50:44.815430+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:44.815430+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13350156,"num_objects":1630,"num_object_clones":0,"num_object_copies":6520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11150,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-568}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10838","reported_seq":10942,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:42.913243+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:42.913243+0000","last_peered":"2024-09-16T04:50:42.913243+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:42.913243+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13194508,"num_objects":1611,"num_object_clones":0,"num_object_copies":6444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10836,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-512}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10794","reported_seq":10889,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:44.361878+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:44.361878+0000","last_peered":"2024-09-16T04:50:44.361878+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:44.361878+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12391692,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10792,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-715}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":210511552,"num_objects":25703,"num_object_clones":0,"num_object_copies":102812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25703,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175784,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6865,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":423636992,"data_stored":423636992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160911,"ondisk_log_size":160911,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730276,"kb_used_data":102984,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93641564,"statfs":{"total":96636764160,"available":95888961536,"internally_reserved":0,"allocated":105455616,"data_stored":105266414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,4,2,2,1,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757724,"kb_used_data":104576,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93614116,"statfs":{"total":96636764160,"available":95860854784,"internally_reserved":0,"allocated":107085824,"data_stored":106896622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602664,"kb_used_data":105852,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93769176,"statfs":{"total":96636764160,"available":96019636224,"internally_reserved":0,"allocated":108392448,"data_stored":108203246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508686031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734852,"kb_used_data":103464,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93636988,"statfs":{"total":96636764160,"available":95884275712,"internally_reserved":0,"allocated":105947136,"data_stored":105757934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":646508238},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":105189376,"data_stored":105189376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107601920,"data_stored":107601920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106098688,"data_stored":106098688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":104747008,"data_stored":104747008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:51.361 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:51.915 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:51.916 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:51.935 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-09-16T04:50:50.751286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":202745536,"num_objects":24755,"num_object_clones":0,"num_object_copies":99020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176732,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6926,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160659,"ondisk_log_size":160659,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805684,"kb_used_data":401240,"kb_used_omap":14,"kb_used_meta":2404337,"kb_avail":374681676,"statfs":{"total":386547056640,"available":383674036224,"internally_reserved":0,"allocated":410869760,"data_stored":410112952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2462041649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,2,7,12,12,16,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21618688,"num_objects":-2639,"num_object_clones":0,"num_object_copies":-10556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2639,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005117"},"pg_stats":[{"pgid":"1.f","version":"35'11105","reported_seq":11221,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.990176+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:49.990176+0000","last_peered":"2024-09-16T04:50:49.990176+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:49.990176+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12689408,"num_objects":1549,"num_object_clones":0,"num_object_copies":6196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11105,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-637}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11153","reported_seq":11242,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.263212+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:49.263212+0000","last_peered":"2024-09-16T04:50:49.263212+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:49.263212+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12457228,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11151,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-682}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11119","reported_seq":11218,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:48.287315+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:48.287315+0000","last_peered":"2024-09-16T04:50:48.287315+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:48.287315+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12704414,"num_objects":1552,"num_object_clones":0,"num_object_copies":6208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11118,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-667}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11098","reported_seq":15295,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.437208+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:49.437208+0000","last_peered":"2024-09-16T04:50:49.437208+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:49.437208+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12875020,"num_objects":1572,"num_object_clones":0,"num_object_copies":6288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11096,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2063,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11148","reported_seq":11281,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.991728+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:49.991728+0000","last_peered":"2024-09-16T04:50:49.991728+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:49.991728+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13047052,"num_objects":1593,"num_object_clones":0,"num_object_copies":6372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11146,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10827","reported_seq":10938,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.958895+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:49.958895+0000","last_peered":"2024-09-16T04:50:49.958895+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:49.958895+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12165120,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10827,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11122","reported_seq":11220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.451822+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:49.451822+0000","last_peered":"2024-09-16T04:50:49.451822+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:49.451822+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12104972,"num_objects":1478,"num_object_clones":0,"num_object_copies":5912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11120,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-735}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11086","reported_seq":15649,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:46.750812+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:46.750812+0000","last_peered":"2024-09-16T04:50:46.750812+0000","last_clean":"2024-09-16T04:50:46.750812+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:46.750812+0000","last_undegraded":"2024-09-16T04:50:46.750812+0000","last_fullsized":"2024-09-16T04:50:46.750812+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:47.495809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13915404,"num_objects":1699,"num_object_clones":0,"num_object_copies":6796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11084,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11216","reported_seq":15972,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:50.000038+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:50.000038+0000","last_peered":"2024-09-16T04:50:50.000038+0000","last_clean":"2024-09-16T04:50:50.000038+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:50.000038+0000","last_undegraded":"2024-09-16T04:50:50.000038+0000","last_fullsized":"2024-09-16T04:50:50.000038+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:54.526181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12610072,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11213,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11130","reported_seq":11238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:48.284875+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:48.284875+0000","last_peered":"2024-09-16T04:50:48.284875+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:48.284875+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13333772,"num_objects":1628,"num_object_clones":0,"num_object_copies":6512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11128,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-561}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10981","reported_seq":11114,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.975441+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:49.975441+0000","last_peered":"2024-09-16T04:50:49.975441+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:49.975441+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12365714,"num_objects":1510,"num_object_clones":0,"num_object_copies":6040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10978,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-645}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10849","reported_seq":10942,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.984794+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:49.984794+0000","last_peered":"2024-09-16T04:50:49.984794+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:49.984794+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12139142,"num_objects":1482,"num_object_clones":0,"num_object_copies":5928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10848,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10957","reported_seq":11045,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:48.018818+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:48.018818+0000","last_peered":"2024-09-16T04:50:48.018818+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:48.018818+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12909190,"num_objects":1576,"num_object_clones":0,"num_object_copies":6304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10956,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-645}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11210","reported_seq":11343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.942233+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:49.942233+0000","last_peered":"2024-09-16T04:50:49.942233+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:49.942233+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12875020,"num_objects":1572,"num_object_clones":0,"num_object_copies":6288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11208,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-626}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10909","reported_seq":11012,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:48.082489+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:48.082489+0000","last_peered":"2024-09-16T04:50:48.082489+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:48.082489+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12612876,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10907,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-583}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10849","reported_seq":10944,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.494116+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:49.494116+0000","last_peered":"2024-09-16T04:50:49.494116+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:49.494116+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11941132,"num_objects":1458,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10847,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-770}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":202745536,"num_objects":24755,"num_object_clones":0,"num_object_copies":99020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176732,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6926,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":407408640,"data_stored":407408640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160659,"ondisk_log_size":160659,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730488,"kb_used_data":99036,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93641352,"statfs":{"total":96636764160,"available":95888744448,"internally_reserved":0,"allocated":101412864,"data_stored":101223662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646573775},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,4,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741348,"kb_used_data":100676,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93630492,"statfs":{"total":96636764160,"available":95877623808,"internally_reserved":0,"allocated":103092224,"data_stored":102903022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,5,5,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602888,"kb_used_data":101944,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93768952,"statfs":{"total":96636764160,"available":96019406848,"internally_reserved":0,"allocated":104390656,"data_stored":104201454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512945871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,4,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730960,"kb_used_data":99584,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93640880,"statfs":{"total":96636764160,"available":95888261120,"internally_reserved":0,"allocated":101974016,"data_stored":101784814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":646508238},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1,1,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":101097472,"data_stored":101097472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103563264,"data_stored":103563264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102105088,"data_stored":102105088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":100642816,"data_stored":100642816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:51.938 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:52.536 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:52.537 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:52.567 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-09-16T04:50:50.751286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":202745536,"num_objects":24755,"num_object_clones":0,"num_object_copies":99020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176732,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6926,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160659,"ondisk_log_size":160659,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805684,"kb_used_data":401240,"kb_used_omap":14,"kb_used_meta":2404337,"kb_avail":374681676,"statfs":{"total":386547056640,"available":383674036224,"internally_reserved":0,"allocated":410869760,"data_stored":410112952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2462041649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,2,7,12,12,16,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21618688,"num_objects":-2639,"num_object_clones":0,"num_object_copies":-10556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2639,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005117"},"pg_stats":[{"pgid":"1.f","version":"35'11105","reported_seq":11221,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.990176+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:49.990176+0000","last_peered":"2024-09-16T04:50:49.990176+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:49.990176+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12689408,"num_objects":1549,"num_object_clones":0,"num_object_copies":6196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11105,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-637}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11153","reported_seq":11242,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.263212+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:49.263212+0000","last_peered":"2024-09-16T04:50:49.263212+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:49.263212+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12457228,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11151,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-682}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11119","reported_seq":11218,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:48.287315+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:48.287315+0000","last_peered":"2024-09-16T04:50:48.287315+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:48.287315+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12704414,"num_objects":1552,"num_object_clones":0,"num_object_copies":6208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11118,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-667}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11098","reported_seq":15295,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.437208+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:49.437208+0000","last_peered":"2024-09-16T04:50:49.437208+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:49.437208+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12875020,"num_objects":1572,"num_object_clones":0,"num_object_copies":6288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11096,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2063,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11148","reported_seq":11281,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.991728+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:49.991728+0000","last_peered":"2024-09-16T04:50:49.991728+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:49.991728+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13047052,"num_objects":1593,"num_object_clones":0,"num_object_copies":6372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11146,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10827","reported_seq":10938,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.958895+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:49.958895+0000","last_peered":"2024-09-16T04:50:49.958895+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:49.958895+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12165120,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10827,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11122","reported_seq":11220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.451822+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:49.451822+0000","last_peered":"2024-09-16T04:50:49.451822+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:49.451822+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12104972,"num_objects":1478,"num_object_clones":0,"num_object_copies":5912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11120,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-735}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11086","reported_seq":15649,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:46.750812+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:46.750812+0000","last_peered":"2024-09-16T04:50:46.750812+0000","last_clean":"2024-09-16T04:50:46.750812+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:46.750812+0000","last_undegraded":"2024-09-16T04:50:46.750812+0000","last_fullsized":"2024-09-16T04:50:46.750812+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:47.495809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13915404,"num_objects":1699,"num_object_clones":0,"num_object_copies":6796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11084,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11216","reported_seq":15972,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:50.000038+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:50.000038+0000","last_peered":"2024-09-16T04:50:50.000038+0000","last_clean":"2024-09-16T04:50:50.000038+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:50.000038+0000","last_undegraded":"2024-09-16T04:50:50.000038+0000","last_fullsized":"2024-09-16T04:50:50.000038+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:54.526181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12610072,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11213,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11130","reported_seq":11238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:48.284875+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:48.284875+0000","last_peered":"2024-09-16T04:50:48.284875+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:48.284875+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13333772,"num_objects":1628,"num_object_clones":0,"num_object_copies":6512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11128,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-561}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10981","reported_seq":11114,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.975441+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:49.975441+0000","last_peered":"2024-09-16T04:50:49.975441+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:49.975441+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12365714,"num_objects":1510,"num_object_clones":0,"num_object_copies":6040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10978,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-645}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10849","reported_seq":10942,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.984794+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:49.984794+0000","last_peered":"2024-09-16T04:50:49.984794+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:49.984794+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12139142,"num_objects":1482,"num_object_clones":0,"num_object_copies":5928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10848,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10957","reported_seq":11045,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:48.018818+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:48.018818+0000","last_peered":"2024-09-16T04:50:48.018818+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:48.018818+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12909190,"num_objects":1576,"num_object_clones":0,"num_object_copies":6304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10956,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-645}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11210","reported_seq":11343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.942233+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:49.942233+0000","last_peered":"2024-09-16T04:50:49.942233+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:49.942233+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12875020,"num_objects":1572,"num_object_clones":0,"num_object_copies":6288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11208,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-626}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10909","reported_seq":11012,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:48.082489+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:48.082489+0000","last_peered":"2024-09-16T04:50:48.082489+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:48.082489+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12612876,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10907,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-583}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10849","reported_seq":10944,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.494116+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:49.494116+0000","last_peered":"2024-09-16T04:50:49.494116+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:49.494116+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11941132,"num_objects":1458,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10847,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-770}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":202745536,"num_objects":24755,"num_object_clones":0,"num_object_copies":99020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176732,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6926,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":407408640,"data_stored":407408640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160659,"ondisk_log_size":160659,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730488,"kb_used_data":99036,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93641352,"statfs":{"total":96636764160,"available":95888744448,"internally_reserved":0,"allocated":101412864,"data_stored":101223662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646573775},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,4,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741348,"kb_used_data":100676,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93630492,"statfs":{"total":96636764160,"available":95877623808,"internally_reserved":0,"allocated":103092224,"data_stored":102903022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,5,5,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602888,"kb_used_data":101944,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93768952,"statfs":{"total":96636764160,"available":96019406848,"internally_reserved":0,"allocated":104390656,"data_stored":104201454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512945871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,4,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730960,"kb_used_data":99584,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93640880,"statfs":{"total":96636764160,"available":95888261120,"internally_reserved":0,"allocated":101974016,"data_stored":101784814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":646508238},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1,1,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":101097472,"data_stored":101097472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103563264,"data_stored":103563264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102105088,"data_stored":102105088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":100642816,"data_stored":100642816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:52.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:50:52.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11105", 'reported_seq': 11221, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:49.990176+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:50:49.990176+0000', 'last_peered': '2024-09-16T04:50:49.990176+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:50:49.990176+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12689408, 'num_objects': 1549, 'num_object_clones': 0, 'num_object_copies': 6196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1549, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11105, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -637}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:52.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:50:52.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11153", 'reported_seq': 11242, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:49.263212+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:50:49.263212+0000', 'last_peered': '2024-09-16T04:50:49.263212+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:50:49.263212+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12457228, 'num_objects': 1521, 'num_object_clones': 0, 'num_object_copies': 6084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11151, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -682}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:52.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:50:52.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11119", 'reported_seq': 11218, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:48.287315+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:50:48.287315+0000', 'last_peered': '2024-09-16T04:50:48.287315+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:50:48.287315+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12704414, 'num_objects': 1552, 'num_object_clones': 0, 'num_object_copies': 6208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1552, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11118, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -667}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:52.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:50:52.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11098", 'reported_seq': 15295, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:49.437208+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:50:49.437208+0000', 'last_peered': '2024-09-16T04:50:49.437208+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:50:49.437208+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12875020, 'num_objects': 1572, 'num_object_clones': 0, 'num_object_copies': 6288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 170, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11096, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2063, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1402}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:52.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:50:52.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11148", 'reported_seq': 11281, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:49.991728+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:50:49.991728+0000', 'last_peered': '2024-09-16T04:50:49.991728+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:50:49.991728+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13047052, 'num_objects': 1593, 'num_object_clones': 0, 'num_object_copies': 6372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11146, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -655}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:52.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:50:52.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10827", 'reported_seq': 10938, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:49.958895+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:50:49.958895+0000', 'last_peered': '2024-09-16T04:50:49.958895+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:50:49.958895+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12165120, 'num_objects': 1485, 'num_object_clones': 0, 'num_object_copies': 5940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1485, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10827, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -693}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:52.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:50:52.570 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11122", 'reported_seq': 11220, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:49.451822+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:50:49.451822+0000', 'last_peered': '2024-09-16T04:50:49.451822+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:50:49.451822+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12104972, 'num_objects': 1478, 'num_object_clones': 0, 'num_object_copies': 5912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1478, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11120, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -735}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:52.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:50:52.570 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11130", 'reported_seq': 11238, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:48.284875+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:50:48.284875+0000', 'last_peered': '2024-09-16T04:50:48.284875+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:50:48.284875+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13333772, 'num_objects': 1628, 'num_object_clones': 0, 'num_object_copies': 6512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11128, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -561}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:52.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:50:52.570 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10981", 'reported_seq': 11114, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:49.975441+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:50:49.975441+0000', 'last_peered': '2024-09-16T04:50:49.975441+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:50:49.975441+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12365714, 'num_objects': 1510, 'num_object_clones': 0, 'num_object_copies': 6040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1510, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10978, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -645}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:52.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:50:52.570 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10849", 'reported_seq': 10942, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:49.984794+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:50:49.984794+0000', 'last_peered': '2024-09-16T04:50:49.984794+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:50:49.984794+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12139142, 'num_objects': 1482, 'num_object_clones': 0, 'num_object_copies': 5928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10848, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -623}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:52.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:50:52.570 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10957", 'reported_seq': 11045, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:48.018818+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:50:48.018818+0000', 'last_peered': '2024-09-16T04:50:48.018818+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:50:48.018818+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12909190, 'num_objects': 1576, 'num_object_clones': 0, 'num_object_copies': 6304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10956, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -645}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:52.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:50:52.571 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11210", 'reported_seq': 11343, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:49.942233+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:50:49.942233+0000', 'last_peered': '2024-09-16T04:50:49.942233+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:50:49.942233+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12875020, 'num_objects': 1572, 'num_object_clones': 0, 'num_object_copies': 6288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11208, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -626}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:52.571 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:50:52.571 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10909", 'reported_seq': 11012, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:48.082489+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:50:48.082489+0000', 'last_peered': '2024-09-16T04:50:48.082489+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:50:48.082489+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12612876, 'num_objects': 1540, 'num_object_clones': 0, 'num_object_copies': 6160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10907, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -583}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:52.571 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:50:52.571 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10849", 'reported_seq': 10944, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:49.494116+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:50:49.494116+0000', 'last_peered': '2024-09-16T04:50:49.494116+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:50:49.494116+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11941132, 'num_objects': 1458, 'num_object_clones': 0, 'num_object_copies': 5832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1458, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10847, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -770}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:52.571 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:50:53.280 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:53.280 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28656/99020 objects degraded (28.940%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":665,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":24755,"data_bytes":202745536,"bytes_used":2873020416,"bytes_avail":383674036224,"bytes_total":386547056640,"degraded_objects":28656,"degraded_total":99020,"degraded_ratio":0.28939608159967684,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":193,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":219},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:50:53.309 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:50:53.309 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:53.921 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:53.921 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:53.946 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-16T04:50:52.752202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":202081984,"num_objects":24674,"num_object_clones":0,"num_object_copies":98696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24674,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176813,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6926,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160640,"ondisk_log_size":160640,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805916,"kb_used_data":397312,"kb_used_omap":14,"kb_used_meta":2408497,"kb_avail":374681444,"statfs":{"total":386547056640,"available":383673798656,"internally_reserved":0,"allocated":406847488,"data_stored":406090680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2466301489},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,4,15,9,14,10,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16654336,"num_objects":-2033,"num_object_clones":0,"num_object_copies":-8132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2033,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005350"},"pg_stats":[{"pgid":"1.f","version":"35'11105","reported_seq":11221,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.990176+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:49.990176+0000","last_peered":"2024-09-16T04:50:49.990176+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:49.990176+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12689408,"num_objects":1549,"num_object_clones":0,"num_object_copies":6196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11105,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-637}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11153","reported_seq":11242,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.263212+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:49.263212+0000","last_peered":"2024-09-16T04:50:49.263212+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:49.263212+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12457228,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11151,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-682}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11119","reported_seq":11218,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:48.287315+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:48.287315+0000","last_peered":"2024-09-16T04:50:48.287315+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:48.287315+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12704414,"num_objects":1552,"num_object_clones":0,"num_object_copies":6208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11118,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-667}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11098","reported_seq":15295,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.437208+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:49.437208+0000","last_peered":"2024-09-16T04:50:49.437208+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:49.437208+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12875020,"num_objects":1572,"num_object_clones":0,"num_object_copies":6288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11096,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2063,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11148","reported_seq":11281,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.991728+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:49.991728+0000","last_peered":"2024-09-16T04:50:49.991728+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:49.991728+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13047052,"num_objects":1593,"num_object_clones":0,"num_object_copies":6372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11146,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10827","reported_seq":10938,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.958895+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:49.958895+0000","last_peered":"2024-09-16T04:50:49.958895+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:49.958895+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12165120,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10827,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11122","reported_seq":11220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.451822+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:49.451822+0000","last_peered":"2024-09-16T04:50:49.451822+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:49.451822+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12104972,"num_objects":1478,"num_object_clones":0,"num_object_copies":5912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11120,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-735}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11167","reported_seq":15730,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:52.227245+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:52.227245+0000","last_peered":"2024-09-16T04:50:52.227245+0000","last_clean":"2024-09-16T04:50:52.227245+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:52.227245+0000","last_undegraded":"2024-09-16T04:50:52.227245+0000","last_fullsized":"2024-09-16T04:50:52.227245+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:52.522805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13251852,"num_objects":1618,"num_object_clones":0,"num_object_copies":6472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11216","reported_seq":15972,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:50.000038+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:50.000038+0000","last_peered":"2024-09-16T04:50:50.000038+0000","last_clean":"2024-09-16T04:50:50.000038+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:50.000038+0000","last_undegraded":"2024-09-16T04:50:50.000038+0000","last_fullsized":"2024-09-16T04:50:50.000038+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:54.526181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12610072,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11213,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11130","reported_seq":11238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:48.284875+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:48.284875+0000","last_peered":"2024-09-16T04:50:48.284875+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:48.284875+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13333772,"num_objects":1628,"num_object_clones":0,"num_object_copies":6512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11128,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-561}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10981","reported_seq":11114,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.975441+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:49.975441+0000","last_peered":"2024-09-16T04:50:49.975441+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:49.975441+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12365714,"num_objects":1510,"num_object_clones":0,"num_object_copies":6040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10978,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-645}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10849","reported_seq":10942,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.984794+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:49.984794+0000","last_peered":"2024-09-16T04:50:49.984794+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:49.984794+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12139142,"num_objects":1482,"num_object_clones":0,"num_object_copies":5928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10848,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10957","reported_seq":11045,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:48.018818+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:48.018818+0000","last_peered":"2024-09-16T04:50:48.018818+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:48.018818+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12909190,"num_objects":1576,"num_object_clones":0,"num_object_copies":6304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10956,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-645}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11210","reported_seq":11343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.942233+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:49.942233+0000","last_peered":"2024-09-16T04:50:49.942233+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:49.942233+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12875020,"num_objects":1572,"num_object_clones":0,"num_object_copies":6288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11208,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-626}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10909","reported_seq":11012,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:48.082489+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:48.082489+0000","last_peered":"2024-09-16T04:50:48.082489+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:48.082489+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12612876,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10907,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-583}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10849","reported_seq":10944,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:49.494116+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:49.494116+0000","last_peered":"2024-09-16T04:50:49.494116+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:49.494116+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11941132,"num_objects":1458,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10847,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-770}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":202081984,"num_objects":24674,"num_object_clones":0,"num_object_copies":98696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24674,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176813,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6926,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":403333120,"data_stored":403333120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160640,"ondisk_log_size":160640,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730488,"kb_used_data":99036,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93641352,"statfs":{"total":96636764160,"available":95888744448,"internally_reserved":0,"allocated":101412864,"data_stored":101223662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646573775},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,4,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741348,"kb_used_data":100676,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93630492,"statfs":{"total":96636764160,"available":95877623808,"internally_reserved":0,"allocated":103092224,"data_stored":102903022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,5,5,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603120,"kb_used_data":98016,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93768720,"statfs":{"total":96636764160,"available":96019169280,"internally_reserved":0,"allocated":100368384,"data_stored":100179182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517205711},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,1,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730960,"kb_used_data":99584,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93640880,"statfs":{"total":96636764160,"available":95888261120,"internally_reserved":0,"allocated":101974016,"data_stored":101784814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":646508238},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1,1,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":101097472,"data_stored":101097472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99487744,"data_stored":99487744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102105088,"data_stored":102105088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":100642816,"data_stored":100642816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:56.948 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:57.577 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:57.578 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:57.613 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-16T04:50:56.753768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":194635456,"num_objects":23765,"num_object_clones":0,"num_object_copies":95060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23765,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177722,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6992,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2860964,"kb_used_data":385756,"kb_used_omap":14,"kb_used_meta":2475121,"kb_avail":374626396,"statfs":{"total":386547056640,"available":383617429504,"internally_reserved":0,"allocated":395014144,"data_stored":394257336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2534524465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,12,5,14,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19456000,"num_objects":-2375,"num_object_clones":0,"num_object_copies":-9500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2375,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005160"},"pg_stats":[{"pgid":"1.f","version":"35'11167","reported_seq":11284,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.221121+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:55.221121+0000","last_peered":"2024-09-16T04:50:55.221121+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:55.221121+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12181504,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11167,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11216","reported_seq":11305,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:54.056527+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:54.056527+0000","last_peered":"2024-09-16T04:50:54.056527+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:54.056527+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11941132,"num_objects":1458,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11214,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11172","reported_seq":11271,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:53.321170+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:53.321170+0000","last_peered":"2024-09-16T04:50:53.321170+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:53.321170+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12270238,"num_objects":1499,"num_object_clones":0,"num_object_copies":5996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11171,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-720}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11160","reported_seq":15486,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:54.597238+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:54.597238+0000","last_peered":"2024-09-16T04:50:54.597238+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:54.597238+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12367116,"num_objects":1510,"num_object_clones":0,"num_object_copies":6040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11158,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11214","reported_seq":11346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.191796+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:55.191796+0000","last_peered":"2024-09-16T04:50:55.191796+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:55.191796+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12506380,"num_objects":1527,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11212,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10894","reported_seq":11004,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.035629+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:55.035629+0000","last_peered":"2024-09-16T04:50:55.035629+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:55.035629+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11616256,"num_objects":1418,"num_object_clones":0,"num_object_copies":5672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11174","reported_seq":11271,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:54.494080+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:54.494080+0000","last_peered":"2024-09-16T04:50:54.494080+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:54.494080+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11678988,"num_objects":1426,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11172,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-787}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11167","reported_seq":15730,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:52.227245+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:52.227245+0000","last_peered":"2024-09-16T04:50:52.227245+0000","last_clean":"2024-09-16T04:50:52.227245+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:52.227245+0000","last_undegraded":"2024-09-16T04:50:52.227245+0000","last_fullsized":"2024-09-16T04:50:52.227245+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:52.522805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13251852,"num_objects":1618,"num_object_clones":0,"num_object_copies":6472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11270","reported_seq":16028,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:55.141529+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:55.141529+0000","last_peered":"2024-09-16T04:50:55.141529+0000","last_clean":"2024-09-16T04:50:55.141529+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:55.141529+0000","last_undegraded":"2024-09-16T04:50:55.141529+0000","last_fullsized":"2024-09-16T04:50:55.141529+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:59.585804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12167704,"num_objects":1486,"num_object_clones":0,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11267,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11194","reported_seq":11302,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:53.203993+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:53.203993+0000","last_peered":"2024-09-16T04:50:53.203993+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:53.203993+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12809484,"num_objects":1564,"num_object_clones":0,"num_object_copies":6256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11192,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-625}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11042","reported_seq":11175,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.216029+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:55.216029+0000","last_peered":"2024-09-16T04:50:55.216029+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:55.216029+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11866002,"num_objects":1449,"num_object_clones":0,"num_object_copies":5796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11039,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10910","reported_seq":11003,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.159196+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:55.159196+0000","last_peered":"2024-09-16T04:50:55.159196+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:55.159196+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11639430,"num_objects":1421,"num_object_clones":0,"num_object_copies":5684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10909,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-684}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11013","reported_seq":11101,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:53.321997+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:53.321997+0000","last_peered":"2024-09-16T04:50:53.321997+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:53.321997+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12450438,"num_objects":1520,"num_object_clones":0,"num_object_copies":6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11012,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-701}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11274","reported_seq":11407,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.189660+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:55.189660+0000","last_peered":"2024-09-16T04:50:55.189660+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:55.189660+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12350732,"num_objects":1508,"num_object_clones":0,"num_object_copies":6032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11272,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-690}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10976","reported_seq":11080,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:53.232629+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:53.232629+0000","last_peered":"2024-09-16T04:50:53.232629+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:53.232629+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12064012,"num_objects":1473,"num_object_clones":0,"num_object_copies":5892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10974,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-650}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10906","reported_seq":11001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:54.481030+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:54.481030+0000","last_peered":"2024-09-16T04:50:54.481030+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:54.481030+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11474188,"num_objects":1401,"num_object_clones":0,"num_object_copies":5604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10904,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-827}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":194635456,"num_objects":23765,"num_object_clones":0,"num_object_copies":95060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23765,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177722,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6992,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":391225344,"data_stored":391225344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748820,"kb_used_data":95172,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93623020,"statfs":{"total":96636764160,"available":95869972480,"internally_reserved":0,"allocated":97456128,"data_stored":97266926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669314767},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759608,"kb_used_data":96736,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93612232,"statfs":{"total":96636764160,"available":95858925568,"internally_reserved":0,"allocated":99057664,"data_stored":98868462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603120,"kb_used_data":98016,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93768720,"statfs":{"total":96636764160,"available":96019169280,"internally_reserved":0,"allocated":100368384,"data_stored":100179182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517205711},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,1,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247936,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749416,"kb_used_data":95832,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93622424,"statfs":{"total":96636764160,"available":95869362176,"internally_reserved":0,"allocated":98131968,"data_stored":97942766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":669249230},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97132544,"data_stored":97132544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99487744,"data_stored":99487744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98066432,"data_stored":98066432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":96538624,"data_stored":96538624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:57.615 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:58.178 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:58.179 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:58.203 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-16T04:50:56.753768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":194635456,"num_objects":23765,"num_object_clones":0,"num_object_copies":95060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23765,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177722,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6992,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2860964,"kb_used_data":385756,"kb_used_omap":14,"kb_used_meta":2475121,"kb_avail":374626396,"statfs":{"total":386547056640,"available":383617429504,"internally_reserved":0,"allocated":395014144,"data_stored":394257336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2534524465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,12,5,14,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19456000,"num_objects":-2375,"num_object_clones":0,"num_object_copies":-9500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2375,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005160"},"pg_stats":[{"pgid":"1.f","version":"35'11167","reported_seq":11284,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.221121+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:55.221121+0000","last_peered":"2024-09-16T04:50:55.221121+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:55.221121+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12181504,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11167,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11216","reported_seq":11305,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:54.056527+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:54.056527+0000","last_peered":"2024-09-16T04:50:54.056527+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:54.056527+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11941132,"num_objects":1458,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11214,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11172","reported_seq":11271,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:53.321170+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:53.321170+0000","last_peered":"2024-09-16T04:50:53.321170+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:53.321170+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12270238,"num_objects":1499,"num_object_clones":0,"num_object_copies":5996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11171,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-720}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11160","reported_seq":15486,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:54.597238+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:54.597238+0000","last_peered":"2024-09-16T04:50:54.597238+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:54.597238+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12367116,"num_objects":1510,"num_object_clones":0,"num_object_copies":6040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11158,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11214","reported_seq":11346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.191796+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:55.191796+0000","last_peered":"2024-09-16T04:50:55.191796+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:55.191796+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12506380,"num_objects":1527,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11212,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10894","reported_seq":11004,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.035629+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:55.035629+0000","last_peered":"2024-09-16T04:50:55.035629+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:55.035629+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11616256,"num_objects":1418,"num_object_clones":0,"num_object_copies":5672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11174","reported_seq":11271,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:54.494080+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:54.494080+0000","last_peered":"2024-09-16T04:50:54.494080+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:54.494080+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11678988,"num_objects":1426,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11172,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-787}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11167","reported_seq":15730,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:52.227245+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:52.227245+0000","last_peered":"2024-09-16T04:50:52.227245+0000","last_clean":"2024-09-16T04:50:52.227245+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:52.227245+0000","last_undegraded":"2024-09-16T04:50:52.227245+0000","last_fullsized":"2024-09-16T04:50:52.227245+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:52.522805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13251852,"num_objects":1618,"num_object_clones":0,"num_object_copies":6472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11270","reported_seq":16028,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:55.141529+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:55.141529+0000","last_peered":"2024-09-16T04:50:55.141529+0000","last_clean":"2024-09-16T04:50:55.141529+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:55.141529+0000","last_undegraded":"2024-09-16T04:50:55.141529+0000","last_fullsized":"2024-09-16T04:50:55.141529+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:59.585804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12167704,"num_objects":1486,"num_object_clones":0,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11267,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11194","reported_seq":11302,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:53.203993+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:53.203993+0000","last_peered":"2024-09-16T04:50:53.203993+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:53.203993+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12809484,"num_objects":1564,"num_object_clones":0,"num_object_copies":6256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11192,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-625}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11042","reported_seq":11175,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.216029+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:55.216029+0000","last_peered":"2024-09-16T04:50:55.216029+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:55.216029+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11866002,"num_objects":1449,"num_object_clones":0,"num_object_copies":5796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11039,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10910","reported_seq":11003,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.159196+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:55.159196+0000","last_peered":"2024-09-16T04:50:55.159196+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:55.159196+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11639430,"num_objects":1421,"num_object_clones":0,"num_object_copies":5684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10909,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-684}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11013","reported_seq":11101,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:53.321997+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:53.321997+0000","last_peered":"2024-09-16T04:50:53.321997+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:53.321997+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12450438,"num_objects":1520,"num_object_clones":0,"num_object_copies":6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11012,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-701}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11274","reported_seq":11407,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.189660+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:55.189660+0000","last_peered":"2024-09-16T04:50:55.189660+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:55.189660+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12350732,"num_objects":1508,"num_object_clones":0,"num_object_copies":6032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11272,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-690}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10976","reported_seq":11080,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:53.232629+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:53.232629+0000","last_peered":"2024-09-16T04:50:53.232629+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:53.232629+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12064012,"num_objects":1473,"num_object_clones":0,"num_object_copies":5892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10974,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-650}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10906","reported_seq":11001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:54.481030+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:54.481030+0000","last_peered":"2024-09-16T04:50:54.481030+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:54.481030+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11474188,"num_objects":1401,"num_object_clones":0,"num_object_copies":5604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10904,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-827}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":194635456,"num_objects":23765,"num_object_clones":0,"num_object_copies":95060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23765,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177722,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6992,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":391225344,"data_stored":391225344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748820,"kb_used_data":95172,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93623020,"statfs":{"total":96636764160,"available":95869972480,"internally_reserved":0,"allocated":97456128,"data_stored":97266926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669314767},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759608,"kb_used_data":96736,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93612232,"statfs":{"total":96636764160,"available":95858925568,"internally_reserved":0,"allocated":99057664,"data_stored":98868462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603120,"kb_used_data":98016,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93768720,"statfs":{"total":96636764160,"available":96019169280,"internally_reserved":0,"allocated":100368384,"data_stored":100179182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517205711},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,1,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247936,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749416,"kb_used_data":95832,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93622424,"statfs":{"total":96636764160,"available":95869362176,"internally_reserved":0,"allocated":98131968,"data_stored":97942766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":669249230},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97132544,"data_stored":97132544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99487744,"data_stored":99487744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98066432,"data_stored":98066432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":96538624,"data_stored":96538624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:50:58.204 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:50:58.205 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11167", 'reported_seq': 11284, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:55.221121+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:50:55.221121+0000', 'last_peered': '2024-09-16T04:50:55.221121+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:50:55.221121+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12181504, 'num_objects': 1487, 'num_object_clones': 0, 'num_object_copies': 5948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1487, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11167, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -699}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:58.205 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:50:58.205 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11216", 'reported_seq': 11305, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:54.056527+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:50:54.056527+0000', 'last_peered': '2024-09-16T04:50:54.056527+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:50:54.056527+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11941132, 'num_objects': 1458, 'num_object_clones': 0, 'num_object_copies': 5832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1458, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11214, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -745}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:58.205 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:50:58.205 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11172", 'reported_seq': 11271, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:53.321170+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:50:53.321170+0000', 'last_peered': '2024-09-16T04:50:53.321170+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:50:53.321170+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12270238, 'num_objects': 1499, 'num_object_clones': 0, 'num_object_copies': 5996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1499, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11171, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -720}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:58.205 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:50:58.205 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11160", 'reported_seq': 15486, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:54.597238+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:50:54.597238+0000', 'last_peered': '2024-09-16T04:50:54.597238+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:50:54.597238+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12367116, 'num_objects': 1510, 'num_object_clones': 0, 'num_object_copies': 6040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1510, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11158, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2129, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1406}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:58.205 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:50:58.205 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11214", 'reported_seq': 11346, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:55.191796+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:50:55.191796+0000', 'last_peered': '2024-09-16T04:50:55.191796+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:50:55.191796+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12506380, 'num_objects': 1527, 'num_object_clones': 0, 'num_object_copies': 6108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11212, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -721}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:58.206 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:50:58.206 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10894", 'reported_seq': 11004, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:55.035629+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:50:55.035629+0000', 'last_peered': '2024-09-16T04:50:55.035629+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:50:55.035629+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11616256, 'num_objects': 1418, 'num_object_clones': 0, 'num_object_copies': 5672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1418, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10894, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -760}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:58.206 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:50:58.206 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11174", 'reported_seq': 11271, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:54.494080+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:50:54.494080+0000', 'last_peered': '2024-09-16T04:50:54.494080+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:50:54.494080+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11678988, 'num_objects': 1426, 'num_object_clones': 0, 'num_object_copies': 5704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1426, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11172, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -787}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:58.206 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:50:58.206 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11194", 'reported_seq': 11302, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:53.203993+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:50:53.203993+0000', 'last_peered': '2024-09-16T04:50:53.203993+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:50:53.203993+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12809484, 'num_objects': 1564, 'num_object_clones': 0, 'num_object_copies': 6256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11192, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -625}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:58.206 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:50:58.206 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'11042", 'reported_seq': 11175, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:55.216029+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:50:55.216029+0000', 'last_peered': '2024-09-16T04:50:55.216029+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:50:55.216029+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11866002, 'num_objects': 1449, 'num_object_clones': 0, 'num_object_copies': 5796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11039, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -706}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:58.207 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:50:58.207 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10910", 'reported_seq': 11003, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:55.159196+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:50:55.159196+0000', 'last_peered': '2024-09-16T04:50:55.159196+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:50:55.159196+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11639430, 'num_objects': 1421, 'num_object_clones': 0, 'num_object_copies': 5684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10909, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -684}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:58.207 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:50:58.207 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11013", 'reported_seq': 11101, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:53.321997+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:50:53.321997+0000', 'last_peered': '2024-09-16T04:50:53.321997+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:50:53.321997+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12450438, 'num_objects': 1520, 'num_object_clones': 0, 'num_object_copies': 6080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1520, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11012, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -701}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:58.207 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:50:58.207 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11274", 'reported_seq': 11407, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:55.189660+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:50:55.189660+0000', 'last_peered': '2024-09-16T04:50:55.189660+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:50:55.189660+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12350732, 'num_objects': 1508, 'num_object_clones': 0, 'num_object_copies': 6032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11272, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -690}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:50:58.207 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:50:58.207 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10976", 'reported_seq': 11080, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:53.232629+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:50:53.232629+0000', 'last_peered': '2024-09-16T04:50:53.232629+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:50:53.232629+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12064012, 'num_objects': 1473, 'num_object_clones': 0, 'num_object_copies': 5892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1473, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10974, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -650}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:50:58.207 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:50:58.207 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10906", 'reported_seq': 11001, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:54.481030+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:50:54.481030+0000', 'last_peered': '2024-09-16T04:50:54.481030+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:50:54.481030+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11474188, 'num_objects': 1401, 'num_object_clones': 0, 'num_object_copies': 5604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1401, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10904, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -827}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:50:58.208 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:50:58.932 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:58.932 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28590/95060 objects degraded (30.076%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":671,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":23765,"data_bytes":194635456,"bytes_used":2929627136,"bytes_avail":383617429504,"bytes_total":386547056640,"degraded_objects":28590,"degraded_total":95060,"degraded_ratio":0.30075741636860931,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":197},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:50:58.962 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:50:58.962 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:50:59.607 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:50:59.608 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:50:59.647 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-16T04:50:58.754137+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":191923904,"num_objects":23434,"num_object_clones":0,"num_object_copies":93736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23434,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178053,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6992,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160780,"ondisk_log_size":160780,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2861588,"kb_used_data":378068,"kb_used_omap":14,"kb_used_meta":2483441,"kb_avail":374625772,"statfs":{"total":386547056640,"available":383616790528,"internally_reserved":0,"allocated":387141632,"data_stored":386384824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2543044145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,2,3,8,20,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18587648,"num_objects":-2269,"num_object_clones":0,"num_object_copies":-9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2269,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004574"},"pg_stats":[{"pgid":"1.f","version":"35'11167","reported_seq":11284,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.221121+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:50:55.221121+0000","last_peered":"2024-09-16T04:50:55.221121+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:50:55.221121+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12181504,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11167,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11216","reported_seq":11305,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:54.056527+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:54.056527+0000","last_peered":"2024-09-16T04:50:54.056527+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:54.056527+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11941132,"num_objects":1458,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11214,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11239","reported_seq":11338,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:58.333559+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:58.333559+0000","last_peered":"2024-09-16T04:50:58.333559+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:58.333559+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11721374,"num_objects":1432,"num_object_clones":0,"num_object_copies":5728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1432,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11238,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-787}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11160","reported_seq":15486,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:54.597238+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:54.597238+0000","last_peered":"2024-09-16T04:50:54.597238+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:54.597238+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12367116,"num_objects":1510,"num_object_clones":0,"num_object_copies":6040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11158,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11214","reported_seq":11346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.191796+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:50:55.191796+0000","last_peered":"2024-09-16T04:50:55.191796+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:50:55.191796+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12506380,"num_objects":1527,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11212,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10894","reported_seq":11004,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.035629+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:50:55.035629+0000","last_peered":"2024-09-16T04:50:55.035629+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:50:55.035629+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11616256,"num_objects":1418,"num_object_clones":0,"num_object_copies":5672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11174","reported_seq":11271,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:54.494080+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:54.494080+0000","last_peered":"2024-09-16T04:50:54.494080+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:54.494080+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11678988,"num_objects":1426,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11172,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-787}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11240","reported_seq":15802,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:57.041085+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:50:57.041085+0000","last_peered":"2024-09-16T04:50:57.041085+0000","last_clean":"2024-09-16T04:50:57.041085+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:50:57.041085+0000","last_undegraded":"2024-09-16T04:50:57.041085+0000","last_fullsized":"2024-09-16T04:50:57.041085+0000","mapping_epoch":32,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:58.462180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12653836,"num_objects":1545,"num_object_clones":0,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11238,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11270","reported_seq":16028,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:50:55.141529+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:50:55.141529+0000","last_peered":"2024-09-16T04:50:55.141529+0000","last_clean":"2024-09-16T04:50:55.141529+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:50:55.141529+0000","last_undegraded":"2024-09-16T04:50:55.141529+0000","last_fullsized":"2024-09-16T04:50:55.141529+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:50:59.585804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12167704,"num_objects":1486,"num_object_clones":0,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11267,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11271","reported_seq":11379,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:58.383976+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:58.383976+0000","last_peered":"2024-09-16T04:50:58.383976+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:58.383976+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12178700,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11269,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-702}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11042","reported_seq":11175,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.216029+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:50:55.216029+0000","last_peered":"2024-09-16T04:50:55.216029+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:50:55.216029+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11866002,"num_objects":1449,"num_object_clones":0,"num_object_copies":5796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11039,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10910","reported_seq":11003,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.159196+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:50:55.159196+0000","last_peered":"2024-09-16T04:50:55.159196+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:50:55.159196+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11639430,"num_objects":1421,"num_object_clones":0,"num_object_copies":5684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10909,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-684}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11074","reported_seq":11163,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:58.416129+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:58.416129+0000","last_peered":"2024-09-16T04:50:58.416129+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:58.416129+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11950726,"num_objects":1459,"num_object_clones":0,"num_object_copies":5836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11073,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-762}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11274","reported_seq":11407,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:55.189660+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:50:55.189660+0000","last_peered":"2024-09-16T04:50:55.189660+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:50:55.189660+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12350732,"num_objects":1508,"num_object_clones":0,"num_object_copies":6032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11272,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-690}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11029","reported_seq":11133,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:58.339432+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:58.339432+0000","last_peered":"2024-09-16T04:50:58.339432+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:58.339432+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11629836,"num_objects":1420,"num_object_clones":0,"num_object_copies":5680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11027,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-703}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10906","reported_seq":11001,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:54.481030+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:54.481030+0000","last_peered":"2024-09-16T04:50:54.481030+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:54.481030+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11474188,"num_objects":1401,"num_object_clones":0,"num_object_copies":5604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10904,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-827}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":191923904,"num_objects":23434,"num_object_clones":0,"num_object_copies":93736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23434,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178053,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6992,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":383172608,"data_stored":383172608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160780,"ondisk_log_size":160780,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748820,"kb_used_data":95172,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93623020,"statfs":{"total":96636764160,"available":95869972480,"internally_reserved":0,"allocated":97456128,"data_stored":97266926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669314767},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759844,"kb_used_data":92820,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93611996,"statfs":{"total":96636764160,"available":95858683904,"internally_reserved":0,"allocated":95047680,"data_stored":94858478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,1,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603508,"kb_used_data":94244,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93768332,"statfs":{"total":96636764160,"available":96018771968,"internally_reserved":0,"allocated":96505856,"data_stored":96316654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521465551},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,2,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247936,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749416,"kb_used_data":95832,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93622424,"statfs":{"total":96636764160,"available":95869362176,"internally_reserved":0,"allocated":98131968,"data_stored":97942766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":669249230},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97132544,"data_stored":97132544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":95444992,"data_stored":95444992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94056448,"data_stored":94056448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":96538624,"data_stored":96538624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:02.650 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:03.244 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:03.244 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:03.268 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-09-16T04:51:02.758434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":185910976,"num_objects":22700,"num_object_clones":0,"num_object_copies":90800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22700,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178787,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7047,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160714,"ondisk_log_size":160714,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2812556,"kb_used_data":366460,"kb_used_omap":14,"kb_used_meta":2446001,"kb_avail":374674804,"statfs":{"total":386547056640,"available":383666999296,"internally_reserved":0,"allocated":375255040,"data_stored":374498232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2504705585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,7,3,13,9,11,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16834560,"num_objects":-2055,"num_object_clones":0,"num_object_copies":-8220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2055,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007147"},"pg_stats":[{"pgid":"1.f","version":"35'11233","reported_seq":11350,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.280466+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:00.280466+0000","last_peered":"2024-09-16T04:51:00.280466+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:00.280466+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11640832,"num_objects":1421,"num_object_clones":0,"num_object_copies":5684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11233,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11272","reported_seq":11360,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:59.638371+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:59.638371+0000","last_peered":"2024-09-16T04:50:59.638371+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:59.638371+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11482380,"num_objects":1402,"num_object_clones":0,"num_object_copies":5608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11270,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-801}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11239","reported_seq":11338,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:58.333559+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:58.333559+0000","last_peered":"2024-09-16T04:50:58.333559+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:58.333559+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11721374,"num_objects":1432,"num_object_clones":0,"num_object_copies":5728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1432,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11238,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-787}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11218","reported_seq":15657,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:59.698155+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:59.698155+0000","last_peered":"2024-09-16T04:50:59.698155+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:59.698155+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11891980,"num_objects":1452,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":49,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11216,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11275","reported_seq":11409,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.073991+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:00.073991+0000","last_peered":"2024-09-16T04:51:00.073991+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:00.073991+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12006668,"num_objects":1466,"num_object_clones":0,"num_object_copies":5864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11273,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10947","reported_seq":11055,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.207855+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:00.207855+0000","last_peered":"2024-09-16T04:51:00.207855+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:00.207855+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11182080,"num_objects":1365,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10947,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-813}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11237","reported_seq":11334,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:59.683860+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:59.683860+0000","last_peered":"2024-09-16T04:50:59.683860+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:59.683860+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11162892,"num_objects":1363,"num_object_clones":0,"num_object_copies":5452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-850}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11313","reported_seq":15875,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:51:02.394923+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:02.394923+0000","last_peered":"2024-09-16T04:51:02.394923+0000","last_clean":"2024-09-16T04:51:02.394923+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:02.394923+0000","last_undegraded":"2024-09-16T04:51:02.394923+0000","last_fullsized":"2024-09-16T04:51:02.394923+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:04.453343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055820,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11311,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11335","reported_seq":16093,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:51:00.152547+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:00.152547+0000","last_peered":"2024-09-16T04:51:00.152547+0000","last_clean":"2024-09-16T04:51:00.152547+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:00.152547+0000","last_undegraded":"2024-09-16T04:51:00.152547+0000","last_fullsized":"2024-09-16T04:51:00.152547+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:04.607898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11635224,"num_objects":1421,"num_object_clones":0,"num_object_copies":5684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11332,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11271","reported_seq":11379,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:58.383976+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:58.383976+0000","last_peered":"2024-09-16T04:50:58.383976+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:58.383976+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12178700,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11269,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-702}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11102","reported_seq":11235,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.152267+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:00.152267+0000","last_peered":"2024-09-16T04:51:00.152267+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:00.152267+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11374482,"num_objects":1389,"num_object_clones":0,"num_object_copies":5556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11099,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10978","reported_seq":11071,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.277914+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:00.277914+0000","last_peered":"2024-09-16T04:51:00.277914+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:00.277914+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11082374,"num_objects":1353,"num_object_clones":0,"num_object_copies":5412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10977,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-752}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11074","reported_seq":11163,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:58.416129+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:58.416129+0000","last_peered":"2024-09-16T04:50:58.416129+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:58.416129+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11950726,"num_objects":1459,"num_object_clones":0,"num_object_copies":5836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11073,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-762}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11330","reported_seq":11462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.220150+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:00.220150+0000","last_peered":"2024-09-16T04:51:00.220150+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:00.220150+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11891980,"num_objects":1452,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11328,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-746}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11029","reported_seq":11133,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:58.339432+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:58.339432+0000","last_peered":"2024-09-16T04:50:58.339432+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:58.339432+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11629836,"num_objects":1420,"num_object_clones":0,"num_object_copies":5680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11027,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-703}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10961","reported_seq":11056,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:59.661304+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:59.661304+0000","last_peered":"2024-09-16T04:50:59.661304+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:59.661304+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11023628,"num_objects":1346,"num_object_clones":0,"num_object_copies":5384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10959,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-882}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":185910976,"num_objects":22700,"num_object_clones":0,"num_object_copies":90800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22700,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178787,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7047,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":371089408,"data_stored":371089408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160714,"ondisk_log_size":160714,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732500,"kb_used_data":91332,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93639340,"statfs":{"total":96636764160,"available":95886684160,"internally_reserved":0,"allocated":93523968,"data_stored":93334766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":656535247},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,3,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759844,"kb_used_data":92820,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93611996,"statfs":{"total":96636764160,"available":95858683904,"internally_reserved":0,"allocated":95047680,"data_stored":94858478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,1,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587140,"kb_used_data":90344,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93784700,"statfs":{"total":96636764160,"available":96035532800,"internally_reserved":0,"allocated":92512256,"data_stored":92323054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508686031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733072,"kb_used_data":91964,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93638768,"statfs":{"total":96636764160,"available":95886098432,"internally_reserved":0,"allocated":94171136,"data_stored":93981934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":656469710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":93065216,"data_stored":93065216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91369472,"data_stored":91369472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94056448,"data_stored":94056448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":92598272,"data_stored":92598272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:03.269 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:03.903 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:03.904 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:03.923 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-09-16T04:51:03.499410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":185910976,"num_objects":22700,"num_object_clones":0,"num_object_copies":90800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22700,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178787,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7047,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160714,"ondisk_log_size":160714,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2812556,"kb_used_data":366460,"kb_used_omap":14,"kb_used_meta":2446001,"kb_avail":374674804,"statfs":{"total":386547056640,"available":383666999296,"internally_reserved":0,"allocated":375255040,"data_stored":374498232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2504705585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,7,3,13,9,11,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16171008,"num_objects":-1974,"num_object_clones":0,"num_object_copies":-7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1974,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.747208"},"pg_stats":[{"pgid":"1.f","version":"35'11233","reported_seq":11350,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.280466+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:00.280466+0000","last_peered":"2024-09-16T04:51:00.280466+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:00.280466+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11640832,"num_objects":1421,"num_object_clones":0,"num_object_copies":5684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11233,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11272","reported_seq":11360,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:59.638371+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:59.638371+0000","last_peered":"2024-09-16T04:50:59.638371+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:59.638371+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11482380,"num_objects":1402,"num_object_clones":0,"num_object_copies":5608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11270,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-801}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11239","reported_seq":11338,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:58.333559+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:50:58.333559+0000","last_peered":"2024-09-16T04:50:58.333559+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:50:58.333559+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11721374,"num_objects":1432,"num_object_clones":0,"num_object_copies":5728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1432,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11238,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-787}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11218","reported_seq":15657,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:59.698155+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:59.698155+0000","last_peered":"2024-09-16T04:50:59.698155+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:59.698155+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11891980,"num_objects":1452,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":49,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11216,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11275","reported_seq":11409,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.073991+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:00.073991+0000","last_peered":"2024-09-16T04:51:00.073991+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:00.073991+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12006668,"num_objects":1466,"num_object_clones":0,"num_object_copies":5864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11273,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10947","reported_seq":11055,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.207855+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:00.207855+0000","last_peered":"2024-09-16T04:51:00.207855+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:00.207855+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11182080,"num_objects":1365,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10947,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-813}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11237","reported_seq":11334,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:59.683860+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:59.683860+0000","last_peered":"2024-09-16T04:50:59.683860+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:59.683860+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11162892,"num_objects":1363,"num_object_clones":0,"num_object_copies":5452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-850}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11313","reported_seq":15875,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:51:02.394923+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:02.394923+0000","last_peered":"2024-09-16T04:51:02.394923+0000","last_clean":"2024-09-16T04:51:02.394923+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:02.394923+0000","last_undegraded":"2024-09-16T04:51:02.394923+0000","last_fullsized":"2024-09-16T04:51:02.394923+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:04.453343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055820,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11311,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11335","reported_seq":16093,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:51:00.152547+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:00.152547+0000","last_peered":"2024-09-16T04:51:00.152547+0000","last_clean":"2024-09-16T04:51:00.152547+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:00.152547+0000","last_undegraded":"2024-09-16T04:51:00.152547+0000","last_fullsized":"2024-09-16T04:51:00.152547+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:04.607898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11635224,"num_objects":1421,"num_object_clones":0,"num_object_copies":5684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11332,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11271","reported_seq":11379,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:58.383976+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:50:58.383976+0000","last_peered":"2024-09-16T04:50:58.383976+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:50:58.383976+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12178700,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11269,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-702}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11102","reported_seq":11235,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.152267+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:00.152267+0000","last_peered":"2024-09-16T04:51:00.152267+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:00.152267+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11374482,"num_objects":1389,"num_object_clones":0,"num_object_copies":5556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11099,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10978","reported_seq":11071,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.277914+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:00.277914+0000","last_peered":"2024-09-16T04:51:00.277914+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:00.277914+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11082374,"num_objects":1353,"num_object_clones":0,"num_object_copies":5412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10977,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-752}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11074","reported_seq":11163,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:58.416129+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:50:58.416129+0000","last_peered":"2024-09-16T04:50:58.416129+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:50:58.416129+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11950726,"num_objects":1459,"num_object_clones":0,"num_object_copies":5836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11073,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-762}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11330","reported_seq":11462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.220150+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:00.220150+0000","last_peered":"2024-09-16T04:51:00.220150+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:00.220150+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11891980,"num_objects":1452,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11328,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-746}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11029","reported_seq":11133,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:58.339432+0000","last_change":"2024-09-16T04:41:27.780105+0000","last_active":"2024-09-16T04:50:58.339432+0000","last_peered":"2024-09-16T04:50:58.339432+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:50:58.339432+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11629836,"num_objects":1420,"num_object_clones":0,"num_object_copies":5680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11027,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-703}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10961","reported_seq":11056,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:59.661304+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:59.661304+0000","last_peered":"2024-09-16T04:50:59.661304+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:59.661304+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11023628,"num_objects":1346,"num_object_clones":0,"num_object_copies":5384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10959,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-882}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":185910976,"num_objects":22700,"num_object_clones":0,"num_object_copies":90800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22700,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178787,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7047,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":371089408,"data_stored":371089408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160714,"ondisk_log_size":160714,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732500,"kb_used_data":91332,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93639340,"statfs":{"total":96636764160,"available":95886684160,"internally_reserved":0,"allocated":93523968,"data_stored":93334766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":656535247},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,3,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759844,"kb_used_data":92820,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93611996,"statfs":{"total":96636764160,"available":95858683904,"internally_reserved":0,"allocated":95047680,"data_stored":94858478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,1,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587140,"kb_used_data":90344,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93784700,"statfs":{"total":96636764160,"available":96035532800,"internally_reserved":0,"allocated":92512256,"data_stored":92323054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508686031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733072,"kb_used_data":91964,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93638768,"statfs":{"total":96636764160,"available":95886098432,"internally_reserved":0,"allocated":94171136,"data_stored":93981934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":656469710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":93065216,"data_stored":93065216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91369472,"data_stored":91369472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94056448,"data_stored":94056448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":92598272,"data_stored":92598272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:03.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:51:03.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11233", 'reported_seq': 11350, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:00.280466+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:51:00.280466+0000', 'last_peered': '2024-09-16T04:51:00.280466+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:51:00.280466+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11640832, 'num_objects': 1421, 'num_object_clones': 0, 'num_object_copies': 5684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11233, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -765}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:03.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:51:03.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'11272", 'reported_seq': 11360, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:59.638371+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:50:59.638371+0000', 'last_peered': '2024-09-16T04:50:59.638371+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:50:59.638371+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11482380, 'num_objects': 1402, 'num_object_clones': 0, 'num_object_copies': 5608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1402, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11270, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -801}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:03.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:51:03.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11239", 'reported_seq': 11338, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:58.333559+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:50:58.333559+0000', 'last_peered': '2024-09-16T04:50:58.333559+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:50:58.333559+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11721374, 'num_objects': 1432, 'num_object_clones': 0, 'num_object_copies': 5728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1432, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11238, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -787}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:03.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-16T04:51:03.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11218", 'reported_seq': 15657, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:59.698155+0000', 'last_change': '2024-09-16T04:47:50.634194+0000', 'last_active': '2024-09-16T04:50:59.698155+0000', 'last_peered': '2024-09-16T04:50:59.698155+0000', 'last_clean': '2024-09-16T04:40:22.815410+0000', 'last_became_active': '2024-09-16T04:41:27.738432+0000', 'last_became_peered': '2024-09-16T04:41:27.738432+0000', 'last_unstale': '2024-09-16T04:50:59.698155+0000', 'last_undegraded': '2024-09-16T04:41:27.524646+0000', 'last_fullsized': '2024-09-16T04:41:27.482483+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11891980, 'num_objects': 1452, 'num_object_clones': 0, 'num_object_copies': 5808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 49, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11216, 'num_write_kb': 50669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2184, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:03.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:51:03.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11275", 'reported_seq': 11409, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:00.073991+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:51:00.073991+0000', 'last_peered': '2024-09-16T04:51:00.073991+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:51:00.073991+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12006668, 'num_objects': 1466, 'num_object_clones': 0, 'num_object_copies': 5864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11273, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -782}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:03.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:51:03.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10947", 'reported_seq': 11055, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:00.207855+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:51:00.207855+0000', 'last_peered': '2024-09-16T04:51:00.207855+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:51:00.207855+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11182080, 'num_objects': 1365, 'num_object_clones': 0, 'num_object_copies': 5460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1365, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10947, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -813}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:03.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:51:03.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11237", 'reported_seq': 11334, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:59.683860+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:50:59.683860+0000', 'last_peered': '2024-09-16T04:50:59.683860+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:50:59.683860+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11162892, 'num_objects': 1363, 'num_object_clones': 0, 'num_object_copies': 5452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11235, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -850}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:03.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:51:03.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11271", 'reported_seq': 11379, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:58.383976+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:50:58.383976+0000', 'last_peered': '2024-09-16T04:50:58.383976+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:50:58.383976+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12178700, 'num_objects': 1487, 'num_object_clones': 0, 'num_object_copies': 5948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1487, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11269, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -702}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:03.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:51:03.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'11102", 'reported_seq': 11235, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:00.152267+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:51:00.152267+0000', 'last_peered': '2024-09-16T04:51:00.152267+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:51:00.152267+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11374482, 'num_objects': 1389, 'num_object_clones': 0, 'num_object_copies': 5556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11099, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -766}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:03.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:51:03.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10978", 'reported_seq': 11071, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:00.277914+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:51:00.277914+0000', 'last_peered': '2024-09-16T04:51:00.277914+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:51:00.277914+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11082374, 'num_objects': 1353, 'num_object_clones': 0, 'num_object_copies': 5412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1353, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10977, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -752}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:03.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:51:03.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11074", 'reported_seq': 11163, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:58.416129+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:50:58.416129+0000', 'last_peered': '2024-09-16T04:50:58.416129+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:50:58.416129+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11950726, 'num_objects': 1459, 'num_object_clones': 0, 'num_object_copies': 5836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1459, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11073, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -762}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:03.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:51:03.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11330", 'reported_seq': 11462, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:00.220150+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:51:00.220150+0000', 'last_peered': '2024-09-16T04:51:00.220150+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:51:00.220150+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11891980, 'num_objects': 1452, 'num_object_clones': 0, 'num_object_copies': 5808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11328, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -746}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:03.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:51:03.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11029", 'reported_seq': 11133, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:58.339432+0000', 'last_change': '2024-09-16T04:41:27.780105+0000', 'last_active': '2024-09-16T04:50:58.339432+0000', 'last_peered': '2024-09-16T04:50:58.339432+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:50:58.339432+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11629836, 'num_objects': 1420, 'num_object_clones': 0, 'num_object_copies': 5680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1420, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11027, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -703}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:03.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:51:03.927 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10961", 'reported_seq': 11056, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:50:59.661304+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:50:59.661304+0000', 'last_peered': '2024-09-16T04:50:59.661304+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:50:59.661304+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11023628, 'num_objects': 1346, 'num_object_clones': 0, 'num_object_copies': 5384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10959, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -882}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:03.927 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:51:04.654 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:04.654 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28535/90800 objects degraded (31.426%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":676,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":22700,"data_bytes":185910976,"bytes_used":2880057344,"bytes_avail":383666999296,"bytes_total":386547056640,"degraded_objects":28535,"degraded_total":90800,"degraded_ratio":0.31426211453744496,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":171},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:51:04.682 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:51:04.682 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:05.283 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:05.284 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:05.310 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-16T04:51:04.759172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":183871168,"num_objects":22451,"num_object_clones":0,"num_object_copies":89804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179036,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7051,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160763,"ondisk_log_size":160763,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2812064,"kb_used_data":361804,"kb_used_omap":14,"kb_used_meta":2450161,"kb_avail":374675296,"statfs":{"total":386547056640,"available":383667503104,"internally_reserved":0,"allocated":370487296,"data_stored":369725845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2508965425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,4,2,18,10,6,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10764288,"num_objects":-1314,"num_object_clones":0,"num_object_copies":-5256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-59,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1314,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005404"},"pg_stats":[{"pgid":"1.f","version":"35'11233","reported_seq":11350,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.280466+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:00.280466+0000","last_peered":"2024-09-16T04:51:00.280466+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:00.280466+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11640832,"num_objects":1421,"num_object_clones":0,"num_object_copies":5684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11233,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11272","reported_seq":11360,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:59.638371+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:50:59.638371+0000","last_peered":"2024-09-16T04:50:59.638371+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:50:59.638371+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11482380,"num_objects":1402,"num_object_clones":0,"num_object_copies":5608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11270,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-801}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'11294","reported_seq":11393,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:03.452075+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:03.452075+0000","last_peered":"2024-09-16T04:51:03.452075+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:03.452075+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11270814,"num_objects":1377,"num_object_clones":0,"num_object_copies":5508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1377,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11293,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-842}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11218","reported_seq":15657,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:59.698155+0000","last_change":"2024-09-16T04:47:50.634194+0000","last_active":"2024-09-16T04:50:59.698155+0000","last_peered":"2024-09-16T04:50:59.698155+0000","last_clean":"2024-09-16T04:40:22.815410+0000","last_became_active":"2024-09-16T04:41:27.738432+0000","last_became_peered":"2024-09-16T04:41:27.738432+0000","last_unstale":"2024-09-16T04:50:59.698155+0000","last_undegraded":"2024-09-16T04:41:27.524646+0000","last_fullsized":"2024-09-16T04:41:27.482483+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11891980,"num_objects":1452,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":49,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11216,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11275","reported_seq":11409,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.073991+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:00.073991+0000","last_peered":"2024-09-16T04:51:00.073991+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:00.073991+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12006668,"num_objects":1466,"num_object_clones":0,"num_object_copies":5864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11273,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10947","reported_seq":11055,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.207855+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:00.207855+0000","last_peered":"2024-09-16T04:51:00.207855+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:00.207855+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11182080,"num_objects":1365,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10947,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-813}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11237","reported_seq":11334,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:59.683860+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:50:59.683860+0000","last_peered":"2024-09-16T04:50:59.683860+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:50:59.683860+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11162892,"num_objects":1363,"num_object_clones":0,"num_object_copies":5452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-850}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11313","reported_seq":15875,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:51:02.394923+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:02.394923+0000","last_peered":"2024-09-16T04:51:02.394923+0000","last_clean":"2024-09-16T04:51:02.394923+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:02.394923+0000","last_undegraded":"2024-09-16T04:51:02.394923+0000","last_fullsized":"2024-09-16T04:51:02.394923+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:04.453343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055820,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11311,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"35'11335","reported_seq":16093,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:51:00.152547+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:00.152547+0000","last_peered":"2024-09-16T04:51:00.152547+0000","last_clean":"2024-09-16T04:51:00.152547+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:00.152547+0000","last_undegraded":"2024-09-16T04:51:00.152547+0000","last_fullsized":"2024-09-16T04:51:00.152547+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:04.607898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11635224,"num_objects":1421,"num_object_clones":0,"num_object_copies":5684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11332,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11347","reported_seq":11455,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:03.532334+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:03.532334+0000","last_peered":"2024-09-16T04:51:03.532334+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:03.532334+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11556108,"num_objects":1411,"num_object_clones":0,"num_object_copies":5644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11345,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-778}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11102","reported_seq":11235,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.152267+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:00.152267+0000","last_peered":"2024-09-16T04:51:00.152267+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:00.152267+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11374482,"num_objects":1389,"num_object_clones":0,"num_object_copies":5556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11099,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10978","reported_seq":11071,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.277914+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:00.277914+0000","last_peered":"2024-09-16T04:51:00.277914+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:00.277914+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11082374,"num_objects":1353,"num_object_clones":0,"num_object_copies":5412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10977,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-752}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11132","reported_seq":11221,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:03.258517+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:03.258517+0000","last_peered":"2024-09-16T04:51:03.258517+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:03.258517+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11475590,"num_objects":1401,"num_object_clones":0,"num_object_copies":5604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11131,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-820}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11330","reported_seq":11462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:00.220150+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:00.220150+0000","last_peered":"2024-09-16T04:51:00.220150+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:00.220150+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11891980,"num_objects":1452,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11328,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-746}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11089","reported_seq":11202,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:03.547546+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:03.547546+0000","last_peered":"2024-09-16T04:51:03.547546+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:03.547546+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11138316,"num_objects":1360,"num_object_clones":0,"num_object_copies":5440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11087,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-761}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10961","reported_seq":11056,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:50:59.661304+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:50:59.661304+0000","last_peered":"2024-09-16T04:50:59.661304+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:50:59.661304+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11023628,"num_objects":1346,"num_object_clones":0,"num_object_copies":5384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10959,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-882}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":183871168,"num_objects":22451,"num_object_clones":0,"num_object_copies":89804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179036,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7051,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":366985216,"data_stored":366985216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160763,"ondisk_log_size":160763,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732500,"kb_used_data":91332,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93639340,"statfs":{"total":96636764160,"available":95886684160,"internally_reserved":0,"allocated":93523968,"data_stored":93334766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":656535247},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,3,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":759352,"kb_used_data":88164,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93612488,"statfs":{"total":96636764160,"available":95859187712,"internally_reserved":0,"allocated":90279936,"data_stored":90086091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,6,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587140,"kb_used_data":90344,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93784700,"statfs":{"total":96636764160,"available":96035532800,"internally_reserved":0,"allocated":92512256,"data_stored":92323054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508686031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733072,"kb_used_data":91964,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93638768,"statfs":{"total":96636764160,"available":95886098432,"internally_reserved":0,"allocated":94171136,"data_stored":93981934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":656469710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":93065216,"data_stored":93065216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91369472,"data_stored":91369472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89952256,"data_stored":89952256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":92598272,"data_stored":92598272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:08.312 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:08.952 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:08.953 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:08.976 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-16T04:51:08.760711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":176154304,"num_objects":21509,"num_object_clones":0,"num_object_copies":86036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179978,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7153,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2795904,"kb_used_data":345640,"kb_used_omap":14,"kb_used_meta":2450161,"kb_avail":374691456,"statfs":{"total":386547056640,"available":383684050944,"internally_reserved":0,"allocated":353935360,"data_stored":353145988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2508965425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,3,14,16,12,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10354688,"num_objects":-1264,"num_object_clones":0,"num_object_copies":-5056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1264,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005898"},"pg_stats":[{"pgid":"1.f","version":"37'11280","reported_seq":11401,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:05.104832+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:05.104832+0000","last_peered":"2024-09-16T04:51:05.104832+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:05.104832+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11255808,"num_objects":1374,"num_object_clones":0,"num_object_copies":5496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11280,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-812}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11329","reported_seq":11422,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:04.542704+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:04.542704+0000","last_peered":"2024-09-16T04:51:04.542704+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:04.542704+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11015436,"num_objects":1345,"num_object_clones":0,"num_object_copies":5380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11327,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-858}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11345","reported_seq":11448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.553396+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:08.553396+0000","last_peered":"2024-09-16T04:51:08.553396+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:08.553396+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10853022,"num_objects":1326,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11344,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-893}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11309","reported_seq":15858,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:07.386983+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:07.386983+0000","last_peered":"2024-09-16T04:51:07.386983+0000","last_clean":"2024-09-16T04:51:07.386983+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:07.386983+0000","last_undegraded":"2024-09-16T04:51:07.386983+0000","last_fullsized":"2024-09-16T04:51:07.386983+0000","mapping_epoch":36,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:10.469568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11146508,"num_objects":1361,"num_object_clones":0,"num_object_copies":5444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11307,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11330","reported_seq":11468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:05.214107+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:05.214107+0000","last_peered":"2024-09-16T04:51:05.214107+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:05.214107+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11556108,"num_objects":1411,"num_object_clones":0,"num_object_copies":5644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11328,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-837}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11004","reported_seq":11117,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:05.297997+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:05.297997+0000","last_peered":"2024-09-16T04:51:05.297997+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:05.297997+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10715136,"num_objects":1308,"num_object_clones":0,"num_object_copies":5232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11004,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-870}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11293","reported_seq":11396,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:04.586148+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:04.586148+0000","last_peered":"2024-09-16T04:51:04.586148+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:04.586148+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10704140,"num_objects":1307,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11291,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-906}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11384","reported_seq":15951,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:07.436703+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:07.436703+0000","last_peered":"2024-09-16T04:51:07.436703+0000","last_clean":"2024-09-16T04:51:07.436703+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:07.436703+0000","last_undegraded":"2024-09-16T04:51:07.436703+0000","last_fullsized":"2024-09-16T04:51:07.436703+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:09.499841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11474188,"num_objects":1401,"num_object_clones":0,"num_object_copies":5604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11382,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11393","reported_seq":16155,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:05.070405+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:05.070405+0000","last_peered":"2024-09-16T04:51:05.070405+0000","last_clean":"2024-09-16T04:51:05.070405+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:05.070405+0000","last_undegraded":"2024-09-16T04:51:05.070405+0000","last_fullsized":"2024-09-16T04:51:05.070405+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11160088,"num_objects":1363,"num_object_clones":0,"num_object_copies":5452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11390,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11408","reported_seq":11520,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.559426+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:08.559426+0000","last_peered":"2024-09-16T04:51:08.559426+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:08.559426+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11056396,"num_objects":1350,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11406,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-839}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11161","reported_seq":11298,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:05.154007+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:05.154007+0000","last_peered":"2024-09-16T04:51:05.154007+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:05.154007+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10891154,"num_objects":1330,"num_object_clones":0,"num_object_copies":5320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11158,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-825}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11034","reported_seq":11131,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:05.145151+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:05.145151+0000","last_peered":"2024-09-16T04:51:05.145151+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:05.145151+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10623622,"num_objects":1297,"num_object_clones":0,"num_object_copies":5188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11033,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-808}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11193","reported_seq":11286,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.271264+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:08.271264+0000","last_peered":"2024-09-16T04:51:08.271264+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:08.271264+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10975878,"num_objects":1340,"num_object_clones":0,"num_object_copies":5360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11192,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-881}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11384","reported_seq":11521,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:04.777117+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:04.777117+0000","last_peered":"2024-09-16T04:51:04.777117+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:04.777117+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11449612,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11382,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-800}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11143","reported_seq":11366,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.586481+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:08.586481+0000","last_peered":"2024-09-16T04:51:08.586481+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:08.586481+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10695948,"num_objects":1306,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11141,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-762}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11015","reported_seq":11114,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:04.685361+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:04.685361+0000","last_peered":"2024-09-16T04:51:04.685361+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:04.685361+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10581260,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11013,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":176154304,"num_objects":21509,"num_object_clones":0,"num_object_copies":86036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179978,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7153,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":351371264,"data_stored":351371264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732792,"kb_used_data":87456,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93639048,"statfs":{"total":96636764160,"available":95886385152,"internally_reserved":0,"allocated":89554944,"data_stored":89356435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660795087},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,0,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743104,"kb_used_data":84400,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93628736,"statfs":{"total":96636764160,"available":95875825664,"internally_reserved":0,"allocated":86425600,"data_stored":86227091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586740,"kb_used_data":85796,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93785100,"statfs":{"total":96636764160,"available":96035942400,"internally_reserved":0,"allocated":87855104,"data_stored":87656595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512945871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,2,4,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733268,"kb_used_data":87988,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93638572,"statfs":{"total":96636764160,"available":95885897728,"internally_reserved":0,"allocated":90099712,"data_stored":89905867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":660729550},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,6,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":89133056,"data_stored":89133056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87486464,"data_stored":87486464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":85975040,"data_stored":85975040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":88776704,"data_stored":88776704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:08.978 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:09.607 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:09.608 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:09.632 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-16T04:51:08.760711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":176154304,"num_objects":21509,"num_object_clones":0,"num_object_copies":86036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179978,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7153,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2795904,"kb_used_data":345640,"kb_used_omap":14,"kb_used_meta":2450161,"kb_avail":374691456,"statfs":{"total":386547056640,"available":383684050944,"internally_reserved":0,"allocated":353935360,"data_stored":353145988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2508965425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,3,14,16,12,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10354688,"num_objects":-1264,"num_object_clones":0,"num_object_copies":-5056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1264,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005898"},"pg_stats":[{"pgid":"1.f","version":"37'11280","reported_seq":11401,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:05.104832+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:05.104832+0000","last_peered":"2024-09-16T04:51:05.104832+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:05.104832+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11255808,"num_objects":1374,"num_object_clones":0,"num_object_copies":5496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11280,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-812}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11329","reported_seq":11422,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:04.542704+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:04.542704+0000","last_peered":"2024-09-16T04:51:04.542704+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:04.542704+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11015436,"num_objects":1345,"num_object_clones":0,"num_object_copies":5380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11327,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-858}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11345","reported_seq":11448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.553396+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:08.553396+0000","last_peered":"2024-09-16T04:51:08.553396+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:08.553396+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10853022,"num_objects":1326,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11344,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-893}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11309","reported_seq":15858,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:07.386983+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:07.386983+0000","last_peered":"2024-09-16T04:51:07.386983+0000","last_clean":"2024-09-16T04:51:07.386983+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:07.386983+0000","last_undegraded":"2024-09-16T04:51:07.386983+0000","last_fullsized":"2024-09-16T04:51:07.386983+0000","mapping_epoch":36,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:10.469568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11146508,"num_objects":1361,"num_object_clones":0,"num_object_copies":5444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11307,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11330","reported_seq":11468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:05.214107+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:05.214107+0000","last_peered":"2024-09-16T04:51:05.214107+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:05.214107+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11556108,"num_objects":1411,"num_object_clones":0,"num_object_copies":5644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11328,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-837}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11004","reported_seq":11117,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:05.297997+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:05.297997+0000","last_peered":"2024-09-16T04:51:05.297997+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:05.297997+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10715136,"num_objects":1308,"num_object_clones":0,"num_object_copies":5232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11004,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-870}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11293","reported_seq":11396,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:04.586148+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:04.586148+0000","last_peered":"2024-09-16T04:51:04.586148+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:04.586148+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10704140,"num_objects":1307,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11291,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-906}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11384","reported_seq":15951,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:07.436703+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:07.436703+0000","last_peered":"2024-09-16T04:51:07.436703+0000","last_clean":"2024-09-16T04:51:07.436703+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:07.436703+0000","last_undegraded":"2024-09-16T04:51:07.436703+0000","last_fullsized":"2024-09-16T04:51:07.436703+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:09.499841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11474188,"num_objects":1401,"num_object_clones":0,"num_object_copies":5604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11382,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11393","reported_seq":16155,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:05.070405+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:05.070405+0000","last_peered":"2024-09-16T04:51:05.070405+0000","last_clean":"2024-09-16T04:51:05.070405+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:05.070405+0000","last_undegraded":"2024-09-16T04:51:05.070405+0000","last_fullsized":"2024-09-16T04:51:05.070405+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11160088,"num_objects":1363,"num_object_clones":0,"num_object_copies":5452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11390,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11408","reported_seq":11520,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.559426+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:08.559426+0000","last_peered":"2024-09-16T04:51:08.559426+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:08.559426+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11056396,"num_objects":1350,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11406,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-839}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11161","reported_seq":11298,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:05.154007+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:05.154007+0000","last_peered":"2024-09-16T04:51:05.154007+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:05.154007+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10891154,"num_objects":1330,"num_object_clones":0,"num_object_copies":5320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11158,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-825}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11034","reported_seq":11131,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:05.145151+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:05.145151+0000","last_peered":"2024-09-16T04:51:05.145151+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:05.145151+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10623622,"num_objects":1297,"num_object_clones":0,"num_object_copies":5188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11033,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-808}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11193","reported_seq":11286,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.271264+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:08.271264+0000","last_peered":"2024-09-16T04:51:08.271264+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:08.271264+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10975878,"num_objects":1340,"num_object_clones":0,"num_object_copies":5360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11192,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-881}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11384","reported_seq":11521,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:04.777117+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:04.777117+0000","last_peered":"2024-09-16T04:51:04.777117+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:04.777117+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11449612,"num_objects":1398,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11382,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-800}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11143","reported_seq":11366,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.586481+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:08.586481+0000","last_peered":"2024-09-16T04:51:08.586481+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:08.586481+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10695948,"num_objects":1306,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11141,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-762}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11015","reported_seq":11114,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:04.685361+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:04.685361+0000","last_peered":"2024-09-16T04:51:04.685361+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:04.685361+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10581260,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11013,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":176154304,"num_objects":21509,"num_object_clones":0,"num_object_copies":86036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179978,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7153,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":351371264,"data_stored":351371264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732792,"kb_used_data":87456,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93639048,"statfs":{"total":96636764160,"available":95886385152,"internally_reserved":0,"allocated":89554944,"data_stored":89356435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660795087},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,0,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743104,"kb_used_data":84400,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93628736,"statfs":{"total":96636764160,"available":95875825664,"internally_reserved":0,"allocated":86425600,"data_stored":86227091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586740,"kb_used_data":85796,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93785100,"statfs":{"total":96636764160,"available":96035942400,"internally_reserved":0,"allocated":87855104,"data_stored":87656595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512945871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,2,4,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733268,"kb_used_data":87988,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93638572,"statfs":{"total":96636764160,"available":95885897728,"internally_reserved":0,"allocated":90099712,"data_stored":89905867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":660729550},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,6,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":89133056,"data_stored":89133056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87486464,"data_stored":87486464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":85975040,"data_stored":85975040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":88776704,"data_stored":88776704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:09.633 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:51:09.633 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11280", 'reported_seq': 11401, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:05.104832+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:51:05.104832+0000', 'last_peered': '2024-09-16T04:51:05.104832+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:51:05.104832+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11255808, 'num_objects': 1374, 'num_object_clones': 0, 'num_object_copies': 5496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1374, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11280, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -812}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:09.633 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:51:09.633 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11329", 'reported_seq': 11422, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:04.542704+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:51:04.542704+0000', 'last_peered': '2024-09-16T04:51:04.542704+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:51:04.542704+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11015436, 'num_objects': 1345, 'num_object_clones': 0, 'num_object_copies': 5380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11327, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -858}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:09.633 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:51:09.633 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11345", 'reported_seq': 11448, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:08.553396+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:51:08.553396+0000', 'last_peered': '2024-09-16T04:51:08.553396+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:51:08.553396+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10853022, 'num_objects': 1326, 'num_object_clones': 0, 'num_object_copies': 5304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1326, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11344, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -893}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:09.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:51:09.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11330", 'reported_seq': 11468, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:05.214107+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:51:05.214107+0000', 'last_peered': '2024-09-16T04:51:05.214107+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:51:05.214107+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11556108, 'num_objects': 1411, 'num_object_clones': 0, 'num_object_copies': 5644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1411, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11328, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -837}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:09.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:51:09.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11004", 'reported_seq': 11117, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:05.297997+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:51:05.297997+0000', 'last_peered': '2024-09-16T04:51:05.297997+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:51:05.297997+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10715136, 'num_objects': 1308, 'num_object_clones': 0, 'num_object_copies': 5232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11004, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -870}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:09.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:51:09.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11293", 'reported_seq': 11396, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:04.586148+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:51:04.586148+0000', 'last_peered': '2024-09-16T04:51:04.586148+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:51:04.586148+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10704140, 'num_objects': 1307, 'num_object_clones': 0, 'num_object_copies': 5228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11291, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -906}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:09.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:51:09.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11408", 'reported_seq': 11520, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:08.559426+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:51:08.559426+0000', 'last_peered': '2024-09-16T04:51:08.559426+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:51:08.559426+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11056396, 'num_objects': 1350, 'num_object_clones': 0, 'num_object_copies': 5400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1350, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11406, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -839}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:09.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:51:09.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11161", 'reported_seq': 11298, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:05.154007+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:51:05.154007+0000', 'last_peered': '2024-09-16T04:51:05.154007+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:51:05.154007+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10891154, 'num_objects': 1330, 'num_object_clones': 0, 'num_object_copies': 5320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11158, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -825}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:09.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:51:09.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11034", 'reported_seq': 11131, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:05.145151+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:51:05.145151+0000', 'last_peered': '2024-09-16T04:51:05.145151+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:51:05.145151+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10623622, 'num_objects': 1297, 'num_object_clones': 0, 'num_object_copies': 5188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1297, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11033, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -808}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:09.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:51:09.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11193", 'reported_seq': 11286, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:08.271264+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:51:08.271264+0000', 'last_peered': '2024-09-16T04:51:08.271264+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:51:08.271264+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10975878, 'num_objects': 1340, 'num_object_clones': 0, 'num_object_copies': 5360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11192, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:09.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:51:09.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11384", 'reported_seq': 11521, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:04.777117+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:51:04.777117+0000', 'last_peered': '2024-09-16T04:51:04.777117+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:51:04.777117+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11449612, 'num_objects': 1398, 'num_object_clones': 0, 'num_object_copies': 5592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11382, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -800}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:09.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:51:09.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11143", 'reported_seq': 11366, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:08.586481+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:51:08.586481+0000', 'last_peered': '2024-09-16T04:51:08.586481+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:51:08.586481+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10695948, 'num_objects': 1306, 'num_object_clones': 0, 'num_object_copies': 5224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2068, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11141, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 68, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -762}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:09.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:51:09.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11015", 'reported_seq': 11114, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:04.685361+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:51:04.685361+0000', 'last_peered': '2024-09-16T04:51:04.685361+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:51:04.685361+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10581260, 'num_objects': 1292, 'num_object_clones': 0, 'num_object_copies': 5168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11013, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -936}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:09.636 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:51:10.376 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:10.376 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28429/86036 objects degraded (33.043%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":682,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":21509,"data_bytes":176154304,"bytes_used":2863005696,"bytes_avail":383684050944,"bytes_total":386547056640,"degraded_objects":28429,"degraded_total":86036,"degraded_ratio":0.33043144730112978,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":157},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:51:10.402 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:51:10.402 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:10.943 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:10.944 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:10.966 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-16T04:51:10.761500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":170878656,"num_objects":20865,"num_object_clones":0,"num_object_copies":83460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20865,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180622,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7153,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160949,"ondisk_log_size":160949,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2795780,"kb_used_data":337220,"kb_used_omap":14,"kb_used_meta":2458481,"kb_avail":374691580,"statfs":{"total":386547056640,"available":383684177920,"internally_reserved":0,"allocated":345313280,"data_stored":344519244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2517485105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,1,1,4,13,11,24,20,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15032320,"num_objects":-1835,"num_object_clones":0,"num_object_copies":-7340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1835,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003066"},"pg_stats":[{"pgid":"1.f","version":"37'11346","reported_seq":11467,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.199891+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:10.199891+0000","last_peered":"2024-09-16T04:51:10.199891+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:10.199891+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10715136,"num_objects":1308,"num_object_clones":0,"num_object_copies":5232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-878}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11389","reported_seq":11482,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:09.663810+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:09.663810+0000","last_peered":"2024-09-16T04:51:09.663810+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:09.663810+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10523916,"num_objects":1285,"num_object_clones":0,"num_object_copies":5140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11387,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-918}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11345","reported_seq":11448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.553396+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:08.553396+0000","last_peered":"2024-09-16T04:51:08.553396+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:08.553396+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10853022,"num_objects":1326,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11344,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-893}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11309","reported_seq":15858,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:07.386983+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:07.386983+0000","last_peered":"2024-09-16T04:51:07.386983+0000","last_clean":"2024-09-16T04:51:07.386983+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:07.386983+0000","last_undegraded":"2024-09-16T04:51:07.386983+0000","last_fullsized":"2024-09-16T04:51:07.386983+0000","mapping_epoch":36,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:10.469568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11146508,"num_objects":1361,"num_object_clones":0,"num_object_copies":5444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11307,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11385","reported_seq":11523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.224070+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:10.224070+0000","last_peered":"2024-09-16T04:51:10.224070+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:10.224070+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11105548,"num_objects":1356,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11383,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-892}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11067","reported_seq":11182,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.131798+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:10.131798+0000","last_peered":"2024-09-16T04:51:10.131798+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:10.131798+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10199040,"num_objects":1245,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11067,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11349","reported_seq":11452,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:09.609069+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:09.609069+0000","last_peered":"2024-09-16T04:51:09.609069+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:09.609069+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10245388,"num_objects":1251,"num_object_clones":0,"num_object_copies":5004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11347,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-962}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11384","reported_seq":15951,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:07.436703+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:07.436703+0000","last_peered":"2024-09-16T04:51:07.436703+0000","last_clean":"2024-09-16T04:51:07.436703+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:07.436703+0000","last_undegraded":"2024-09-16T04:51:07.436703+0000","last_fullsized":"2024-09-16T04:51:07.436703+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:09.499841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11474188,"num_objects":1401,"num_object_clones":0,"num_object_copies":5604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11382,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11464","reported_seq":16225,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:10.332558+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:10.332558+0000","last_peered":"2024-09-16T04:51:10.332558+0000","last_clean":"2024-09-16T04:51:10.332558+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:10.332558+0000","last_undegraded":"2024-09-16T04:51:10.332558+0000","last_fullsized":"2024-09-16T04:51:10.332558+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:10.561320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10578456,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11408","reported_seq":11520,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.559426+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:08.559426+0000","last_peered":"2024-09-16T04:51:08.559426+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:08.559426+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11056396,"num_objects":1350,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11406,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-839}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11227","reported_seq":11364,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.370180+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:10.370180+0000","last_peered":"2024-09-16T04:51:10.370180+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:10.370180+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10350482,"num_objects":1264,"num_object_clones":0,"num_object_copies":5056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11224,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11090","reported_seq":11185,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.250673+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:10.250673+0000","last_peered":"2024-09-16T04:51:10.250673+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:10.250673+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10164870,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11089,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-864}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11193","reported_seq":11286,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.271264+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:08.271264+0000","last_peered":"2024-09-16T04:51:08.271264+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:08.271264+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10975878,"num_objects":1340,"num_object_clones":0,"num_object_copies":5360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11192,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-881}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11454","reported_seq":11591,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.145898+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:10.145898+0000","last_peered":"2024-09-16T04:51:10.145898+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:10.145898+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10876172,"num_objects":1328,"num_object_clones":0,"num_object_copies":5312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11452,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-870}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11143","reported_seq":11366,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.586481+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:08.586481+0000","last_peered":"2024-09-16T04:51:08.586481+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:08.586481+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10695948,"num_objects":1306,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11141,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-762}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11096","reported_seq":11195,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:09.790068+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:09.790068+0000","last_peered":"2024-09-16T04:51:09.790068+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:09.790068+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9917708,"num_objects":1211,"num_object_clones":0,"num_object_copies":4844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1017}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":170878656,"num_objects":20865,"num_object_clones":0,"num_object_copies":83460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20865,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180622,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7153,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":343232512,"data_stored":343232512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160949,"ondisk_log_size":160949,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733120,"kb_used_data":83632,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93638720,"statfs":{"total":96636764160,"available":95886049280,"internally_reserved":0,"allocated":85639168,"data_stored":85440659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":665054927},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743104,"kb_used_data":84400,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93628736,"statfs":{"total":96636764160,"available":95875825664,"internally_reserved":0,"allocated":86425600,"data_stored":86227091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586740,"kb_used_data":85796,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93785100,"statfs":{"total":96636764160,"available":96035942400,"internally_reserved":0,"allocated":87855104,"data_stored":87656595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512945871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,2,4,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732816,"kb_used_data":83392,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93639024,"statfs":{"total":96636764160,"available":95886360576,"internally_reserved":0,"allocated":85393408,"data_stored":85194899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":664989390},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,0,0,0,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85082112,"data_stored":85082112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87486464,"data_stored":87486464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":85975040,"data_stored":85975040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":84688896,"data_stored":84688896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:13.968 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:14.592 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:14.594 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:14.636 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-09-16T04:51:12.762424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":169756352,"num_objects":20728,"num_object_clones":0,"num_object_copies":82912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20728,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180759,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7153,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160986,"ondisk_log_size":160986,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2796044,"kb_used_data":333320,"kb_used_omap":14,"kb_used_meta":2462641,"kb_avail":374691316,"statfs":{"total":386547056640,"available":383683907584,"internally_reserved":0,"allocated":341319680,"data_stored":340525644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2521744945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,3,4,11,17,20,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16154624,"num_objects":-1972,"num_object_clones":0,"num_object_copies":-7888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1972,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.263013"},"pg_stats":[{"pgid":"1.f","version":"37'11346","reported_seq":11467,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.199891+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:10.199891+0000","last_peered":"2024-09-16T04:51:10.199891+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:10.199891+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10715136,"num_objects":1308,"num_object_clones":0,"num_object_copies":5232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-878}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11389","reported_seq":11482,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:09.663810+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:09.663810+0000","last_peered":"2024-09-16T04:51:09.663810+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:09.663810+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10523916,"num_objects":1285,"num_object_clones":0,"num_object_copies":5140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11387,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-918}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11345","reported_seq":11448,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.553396+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:08.553396+0000","last_peered":"2024-09-16T04:51:08.553396+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:08.553396+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10853022,"num_objects":1326,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11344,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-893}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11367","reported_seq":15917,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:12.475455+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:12.475455+0000","last_peered":"2024-09-16T04:51:12.475455+0000","last_clean":"2024-09-16T04:51:12.475455+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:12.475455+0000","last_undegraded":"2024-09-16T04:51:12.475455+0000","last_fullsized":"2024-09-16T04:51:12.475455+0000","mapping_epoch":36,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:15.540948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10671372,"num_objects":1303,"num_object_clones":0,"num_object_copies":5212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11365,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11385","reported_seq":11523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.224070+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:10.224070+0000","last_peered":"2024-09-16T04:51:10.224070+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:10.224070+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11105548,"num_objects":1356,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11383,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-892}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11067","reported_seq":11182,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.131798+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:10.131798+0000","last_peered":"2024-09-16T04:51:10.131798+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:10.131798+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10199040,"num_objects":1245,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11067,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11349","reported_seq":11452,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:09.609069+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:09.609069+0000","last_peered":"2024-09-16T04:51:09.609069+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:09.609069+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10245388,"num_objects":1251,"num_object_clones":0,"num_object_copies":5004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11347,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-962}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11463","reported_seq":16030,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:12.296737+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:12.296737+0000","last_peered":"2024-09-16T04:51:12.296737+0000","last_clean":"2024-09-16T04:51:12.296737+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:12.296737+0000","last_undegraded":"2024-09-16T04:51:12.296737+0000","last_fullsized":"2024-09-16T04:51:12.296737+0000","mapping_epoch":32,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:15.540925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10827020,"num_objects":1322,"num_object_clones":0,"num_object_copies":5288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11464","reported_seq":16225,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:10.332558+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:10.332558+0000","last_peered":"2024-09-16T04:51:10.332558+0000","last_clean":"2024-09-16T04:51:10.332558+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:10.332558+0000","last_undegraded":"2024-09-16T04:51:10.332558+0000","last_fullsized":"2024-09-16T04:51:10.332558+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:10.561320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10578456,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11408","reported_seq":11520,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.559426+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:08.559426+0000","last_peered":"2024-09-16T04:51:08.559426+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:08.559426+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11056396,"num_objects":1350,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11406,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-839}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11227","reported_seq":11364,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.370180+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:10.370180+0000","last_peered":"2024-09-16T04:51:10.370180+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:10.370180+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10350482,"num_objects":1264,"num_object_clones":0,"num_object_copies":5056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11224,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11090","reported_seq":11185,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.250673+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:10.250673+0000","last_peered":"2024-09-16T04:51:10.250673+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:10.250673+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10164870,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11089,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-864}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11193","reported_seq":11286,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.271264+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:08.271264+0000","last_peered":"2024-09-16T04:51:08.271264+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:08.271264+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10975878,"num_objects":1340,"num_object_clones":0,"num_object_copies":5360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11192,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-881}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11454","reported_seq":11591,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.145898+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:10.145898+0000","last_peered":"2024-09-16T04:51:10.145898+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:10.145898+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10876172,"num_objects":1328,"num_object_clones":0,"num_object_copies":5312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11452,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-870}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11143","reported_seq":11366,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:08.586481+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:08.586481+0000","last_peered":"2024-09-16T04:51:08.586481+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:08.586481+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10695948,"num_objects":1306,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11141,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-762}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11096","reported_seq":11195,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:09.790068+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:09.790068+0000","last_peered":"2024-09-16T04:51:09.790068+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:09.790068+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9917708,"num_objects":1211,"num_object_clones":0,"num_object_copies":4844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1017}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":169756352,"num_objects":20728,"num_object_clones":0,"num_object_copies":82912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20728,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180759,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7153,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":339189760,"data_stored":339189760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160986,"ondisk_log_size":160986,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733120,"kb_used_data":83632,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93638720,"statfs":{"total":96636764160,"available":95886049280,"internally_reserved":0,"allocated":85639168,"data_stored":85440659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":665054927},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743104,"kb_used_data":84400,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93628736,"statfs":{"total":96636764160,"available":95875825664,"internally_reserved":0,"allocated":86425600,"data_stored":86227091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587004,"kb_used_data":81896,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93784836,"statfs":{"total":96636764160,"available":96035672064,"internally_reserved":0,"allocated":83861504,"data_stored":83662995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517205711},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732816,"kb_used_data":83392,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93639024,"statfs":{"total":96636764160,"available":95886360576,"internally_reserved":0,"allocated":85393408,"data_stored":85194899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":664989390},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,0,0,0,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85082112,"data_stored":85082112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83443712,"data_stored":83443712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":85975040,"data_stored":85975040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":84688896,"data_stored":84688896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:14.641 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:15.250 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:15.250 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:15.280 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-16T04:51:14.763220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":167593664,"num_objects":20464,"num_object_clones":0,"num_object_copies":81856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181023,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7199,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161050,"ondisk_log_size":161050,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2796332,"kb_used_data":329448,"kb_used_omap":14,"kb_used_meta":2466801,"kb_avail":374691028,"statfs":{"total":386547056640,"available":383683612672,"internally_reserved":0,"allocated":337354752,"data_stored":336560716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2526004785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,1,3,2,8,19,17,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16277504,"num_objects":-1987,"num_object_clones":0,"num_object_copies":-7948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1987,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.239934"},"pg_stats":[{"pgid":"1.f","version":"37'11346","reported_seq":11467,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.199891+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:10.199891+0000","last_peered":"2024-09-16T04:51:10.199891+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:10.199891+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10715136,"num_objects":1308,"num_object_clones":0,"num_object_copies":5232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-878}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11389","reported_seq":11482,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:09.663810+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:09.663810+0000","last_peered":"2024-09-16T04:51:09.663810+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:09.663810+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10523916,"num_objects":1285,"num_object_clones":0,"num_object_copies":5140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11387,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-918}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11410","reported_seq":11513,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:13.571850+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:13.571850+0000","last_peered":"2024-09-16T04:51:13.571850+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:13.571850+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10320542,"num_objects":1261,"num_object_clones":0,"num_object_copies":5044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1261,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11409,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-958}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11367","reported_seq":15917,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:12.475455+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:12.475455+0000","last_peered":"2024-09-16T04:51:12.475455+0000","last_clean":"2024-09-16T04:51:12.475455+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:12.475455+0000","last_undegraded":"2024-09-16T04:51:12.475455+0000","last_fullsized":"2024-09-16T04:51:12.475455+0000","mapping_epoch":36,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:15.540948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10671372,"num_objects":1303,"num_object_clones":0,"num_object_copies":5212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11365,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11385","reported_seq":11523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.224070+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:10.224070+0000","last_peered":"2024-09-16T04:51:10.224070+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:10.224070+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11105548,"num_objects":1356,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11383,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-892}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11067","reported_seq":11182,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.131798+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:10.131798+0000","last_peered":"2024-09-16T04:51:10.131798+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:10.131798+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10199040,"num_objects":1245,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11067,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11349","reported_seq":11452,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:09.609069+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:09.609069+0000","last_peered":"2024-09-16T04:51:09.609069+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:09.609069+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10245388,"num_objects":1251,"num_object_clones":0,"num_object_copies":5004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11347,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-962}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11463","reported_seq":16030,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:12.296737+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:12.296737+0000","last_peered":"2024-09-16T04:51:12.296737+0000","last_clean":"2024-09-16T04:51:12.296737+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:12.296737+0000","last_undegraded":"2024-09-16T04:51:12.296737+0000","last_fullsized":"2024-09-16T04:51:12.296737+0000","mapping_epoch":32,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:15.540925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10827020,"num_objects":1322,"num_object_clones":0,"num_object_copies":5288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11464","reported_seq":16225,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:10.332558+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:10.332558+0000","last_peered":"2024-09-16T04:51:10.332558+0000","last_clean":"2024-09-16T04:51:10.332558+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:10.332558+0000","last_undegraded":"2024-09-16T04:51:10.332558+0000","last_fullsized":"2024-09-16T04:51:10.332558+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:10.561320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10578456,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11478","reported_seq":11590,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:13.446782+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:13.446782+0000","last_peered":"2024-09-16T04:51:13.446782+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:13.446782+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10482956,"num_objects":1280,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-909}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11227","reported_seq":11364,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.370180+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:10.370180+0000","last_peered":"2024-09-16T04:51:10.370180+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:10.370180+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10350482,"num_objects":1264,"num_object_clones":0,"num_object_copies":5056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11224,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11090","reported_seq":11185,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.250673+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:10.250673+0000","last_peered":"2024-09-16T04:51:10.250673+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:10.250673+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10164870,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11089,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-864}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11271","reported_seq":11364,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:13.476540+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:13.476540+0000","last_peered":"2024-09-16T04:51:13.476540+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:13.476540+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10336902,"num_objects":1262,"num_object_clones":0,"num_object_copies":5048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11270,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-959}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11454","reported_seq":11591,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.145898+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:10.145898+0000","last_peered":"2024-09-16T04:51:10.145898+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:10.145898+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10876172,"num_objects":1328,"num_object_clones":0,"num_object_copies":5312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11452,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-870}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11194","reported_seq":11509,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:13.644274+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:13.644274+0000","last_peered":"2024-09-16T04:51:13.644274+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:13.644274+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10278156,"num_objects":1255,"num_object_clones":0,"num_object_copies":5020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11192,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11096","reported_seq":11195,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:09.790068+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:09.790068+0000","last_peered":"2024-09-16T04:51:09.790068+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:09.790068+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9917708,"num_objects":1211,"num_object_clones":0,"num_object_copies":4844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1017}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":167593664,"num_objects":20464,"num_object_clones":0,"num_object_copies":81856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181023,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7199,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335192064,"data_stored":335192064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161050,"ondisk_log_size":161050,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733120,"kb_used_data":83632,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93638720,"statfs":{"total":96636764160,"available":95886049280,"internally_reserved":0,"allocated":85639168,"data_stored":85440659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":665054927},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743392,"kb_used_data":80528,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93628448,"statfs":{"total":96636764160,"available":95875530752,"internally_reserved":0,"allocated":82460672,"data_stored":82262163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587004,"kb_used_data":81896,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93784836,"statfs":{"total":96636764160,"available":96035672064,"internally_reserved":0,"allocated":83861504,"data_stored":83662995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517205711},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732816,"kb_used_data":83392,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93639024,"statfs":{"total":96636764160,"available":95886360576,"internally_reserved":0,"allocated":85393408,"data_stored":85194899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":664989390},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,0,0,0,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85082112,"data_stored":85082112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83443712,"data_stored":83443712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81977344,"data_stored":81977344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":84688896,"data_stored":84688896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:15.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:51:15.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11346", 'reported_seq': 11467, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:10.199891+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:51:10.199891+0000', 'last_peered': '2024-09-16T04:51:10.199891+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:51:10.199891+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10715136, 'num_objects': 1308, 'num_object_clones': 0, 'num_object_copies': 5232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11346, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -878}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:15.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:51:15.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11389", 'reported_seq': 11482, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:09.663810+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:51:09.663810+0000', 'last_peered': '2024-09-16T04:51:09.663810+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:51:09.663810+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10523916, 'num_objects': 1285, 'num_object_clones': 0, 'num_object_copies': 5140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1285, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11387, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -918}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:15.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:51:15.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11410", 'reported_seq': 11513, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:13.571850+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:51:13.571850+0000', 'last_peered': '2024-09-16T04:51:13.571850+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:51:13.571850+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10320542, 'num_objects': 1261, 'num_object_clones': 0, 'num_object_copies': 5044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1261, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11409, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -958}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:15.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:51:15.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11385", 'reported_seq': 11523, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:10.224070+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:51:10.224070+0000', 'last_peered': '2024-09-16T04:51:10.224070+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:51:10.224070+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11105548, 'num_objects': 1356, 'num_object_clones': 0, 'num_object_copies': 5424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11383, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -892}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:15.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:51:15.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11067", 'reported_seq': 11182, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:10.131798+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:51:10.131798+0000', 'last_peered': '2024-09-16T04:51:10.131798+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:51:10.131798+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10199040, 'num_objects': 1245, 'num_object_clones': 0, 'num_object_copies': 4980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1245, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11067, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -933}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:15.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:51:15.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11349", 'reported_seq': 11452, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:09.609069+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:51:09.609069+0000', 'last_peered': '2024-09-16T04:51:09.609069+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:51:09.609069+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10245388, 'num_objects': 1251, 'num_object_clones': 0, 'num_object_copies': 5004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11347, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -962}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:15.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:51:15.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11478", 'reported_seq': 11590, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:13.446782+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:51:13.446782+0000', 'last_peered': '2024-09-16T04:51:13.446782+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:51:13.446782+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10482956, 'num_objects': 1280, 'num_object_clones': 0, 'num_object_copies': 5120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11476, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -909}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:15.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:51:15.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11227", 'reported_seq': 11364, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:10.370180+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:51:10.370180+0000', 'last_peered': '2024-09-16T04:51:10.370180+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:51:10.370180+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10350482, 'num_objects': 1264, 'num_object_clones': 0, 'num_object_copies': 5056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11224, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -891}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:15.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:51:15.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11090", 'reported_seq': 11185, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:10.250673+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:51:10.250673+0000', 'last_peered': '2024-09-16T04:51:10.250673+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:51:10.250673+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10164870, 'num_objects': 1241, 'num_object_clones': 0, 'num_object_copies': 4964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11089, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -864}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:15.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:51:15.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11271", 'reported_seq': 11364, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:13.476540+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:51:13.476540+0000', 'last_peered': '2024-09-16T04:51:13.476540+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:51:13.476540+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10336902, 'num_objects': 1262, 'num_object_clones': 0, 'num_object_copies': 5048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11270, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -959}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:15.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:51:15.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11454", 'reported_seq': 11591, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:10.145898+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:51:10.145898+0000', 'last_peered': '2024-09-16T04:51:10.145898+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:51:10.145898+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10876172, 'num_objects': 1328, 'num_object_clones': 0, 'num_object_copies': 5312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11452, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -870}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:15.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:51:15.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11194", 'reported_seq': 11509, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:13.644274+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:51:13.644274+0000', 'last_peered': '2024-09-16T04:51:13.644274+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:51:13.644274+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10278156, 'num_objects': 1255, 'num_object_clones': 0, 'num_object_copies': 5020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2022, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11192, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 114, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -767}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:15.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:51:15.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11096", 'reported_seq': 11195, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:09.790068+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:51:09.790068+0000', 'last_peered': '2024-09-16T04:51:09.790068+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:51:09.790068+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9917708, 'num_objects': 1211, 'num_object_clones': 0, 'num_object_copies': 4844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1211, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11094, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1017}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:15.284 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:51:16.003 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:16.003 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28383/81856 objects degraded (34.674%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":688,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20464,"data_bytes":167593664,"bytes_used":2863443968,"bytes_avail":383683612672,"bytes_total":386547056640,"degraded_objects":28383,"degraded_total":81856,"degraded_ratio":0.34674306098514462,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":194},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:51:16.038 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:51:16.039 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:16.692 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:16.693 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:16.708 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-16T04:51:14.763220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":167593664,"num_objects":20464,"num_object_clones":0,"num_object_copies":81856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181023,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7199,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161050,"ondisk_log_size":161050,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2796332,"kb_used_data":329448,"kb_used_omap":14,"kb_used_meta":2466801,"kb_avail":374691028,"statfs":{"total":386547056640,"available":383683612672,"internally_reserved":0,"allocated":337354752,"data_stored":336560716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2526004785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,1,3,2,8,19,17,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16277504,"num_objects":-1987,"num_object_clones":0,"num_object_copies":-7948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1987,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.239934"},"pg_stats":[{"pgid":"1.f","version":"37'11346","reported_seq":11467,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.199891+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:10.199891+0000","last_peered":"2024-09-16T04:51:10.199891+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:10.199891+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10715136,"num_objects":1308,"num_object_clones":0,"num_object_copies":5232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-878}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11389","reported_seq":11482,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:09.663810+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:09.663810+0000","last_peered":"2024-09-16T04:51:09.663810+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:09.663810+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10523916,"num_objects":1285,"num_object_clones":0,"num_object_copies":5140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11387,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-918}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11410","reported_seq":11513,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:13.571850+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:13.571850+0000","last_peered":"2024-09-16T04:51:13.571850+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:13.571850+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10320542,"num_objects":1261,"num_object_clones":0,"num_object_copies":5044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1261,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11409,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-958}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11367","reported_seq":15917,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:12.475455+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:12.475455+0000","last_peered":"2024-09-16T04:51:12.475455+0000","last_clean":"2024-09-16T04:51:12.475455+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:12.475455+0000","last_undegraded":"2024-09-16T04:51:12.475455+0000","last_fullsized":"2024-09-16T04:51:12.475455+0000","mapping_epoch":36,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:15.540948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10671372,"num_objects":1303,"num_object_clones":0,"num_object_copies":5212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11365,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11385","reported_seq":11523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.224070+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:10.224070+0000","last_peered":"2024-09-16T04:51:10.224070+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:10.224070+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11105548,"num_objects":1356,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11383,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-892}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11067","reported_seq":11182,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.131798+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:10.131798+0000","last_peered":"2024-09-16T04:51:10.131798+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:10.131798+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10199040,"num_objects":1245,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11067,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-933}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11349","reported_seq":11452,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:09.609069+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:09.609069+0000","last_peered":"2024-09-16T04:51:09.609069+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:09.609069+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10245388,"num_objects":1251,"num_object_clones":0,"num_object_copies":5004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11347,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-962}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11463","reported_seq":16030,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:12.296737+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:12.296737+0000","last_peered":"2024-09-16T04:51:12.296737+0000","last_clean":"2024-09-16T04:51:12.296737+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:12.296737+0000","last_undegraded":"2024-09-16T04:51:12.296737+0000","last_fullsized":"2024-09-16T04:51:12.296737+0000","mapping_epoch":32,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:15.540925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10827020,"num_objects":1322,"num_object_clones":0,"num_object_copies":5288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11464","reported_seq":16225,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:10.332558+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:10.332558+0000","last_peered":"2024-09-16T04:51:10.332558+0000","last_clean":"2024-09-16T04:51:10.332558+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:10.332558+0000","last_undegraded":"2024-09-16T04:51:10.332558+0000","last_fullsized":"2024-09-16T04:51:10.332558+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:10.561320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10578456,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11478","reported_seq":11590,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:13.446782+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:13.446782+0000","last_peered":"2024-09-16T04:51:13.446782+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:13.446782+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10482956,"num_objects":1280,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-909}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11227","reported_seq":11364,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.370180+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:10.370180+0000","last_peered":"2024-09-16T04:51:10.370180+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:10.370180+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10350482,"num_objects":1264,"num_object_clones":0,"num_object_copies":5056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11224,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11090","reported_seq":11185,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.250673+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:10.250673+0000","last_peered":"2024-09-16T04:51:10.250673+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:10.250673+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10164870,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11089,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-864}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11271","reported_seq":11364,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:13.476540+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:13.476540+0000","last_peered":"2024-09-16T04:51:13.476540+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:13.476540+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10336902,"num_objects":1262,"num_object_clones":0,"num_object_copies":5048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11270,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-959}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11454","reported_seq":11591,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:10.145898+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:10.145898+0000","last_peered":"2024-09-16T04:51:10.145898+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:10.145898+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10876172,"num_objects":1328,"num_object_clones":0,"num_object_copies":5312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11452,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-870}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11194","reported_seq":11509,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:13.644274+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:13.644274+0000","last_peered":"2024-09-16T04:51:13.644274+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:13.644274+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10278156,"num_objects":1255,"num_object_clones":0,"num_object_copies":5020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11192,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11096","reported_seq":11195,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:09.790068+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:09.790068+0000","last_peered":"2024-09-16T04:51:09.790068+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:09.790068+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9917708,"num_objects":1211,"num_object_clones":0,"num_object_copies":4844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1017}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":167593664,"num_objects":20464,"num_object_clones":0,"num_object_copies":81856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181023,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7199,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335192064,"data_stored":335192064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161050,"ondisk_log_size":161050,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733120,"kb_used_data":83632,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93638720,"statfs":{"total":96636764160,"available":95886049280,"internally_reserved":0,"allocated":85639168,"data_stored":85440659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":665054927},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743392,"kb_used_data":80528,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93628448,"statfs":{"total":96636764160,"available":95875530752,"internally_reserved":0,"allocated":82460672,"data_stored":82262163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587004,"kb_used_data":81896,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93784836,"statfs":{"total":96636764160,"available":96035672064,"internally_reserved":0,"allocated":83861504,"data_stored":83662995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517205711},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732816,"kb_used_data":83392,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93639024,"statfs":{"total":96636764160,"available":95886360576,"internally_reserved":0,"allocated":85393408,"data_stored":85194899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":664989390},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,0,0,0,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85082112,"data_stored":85082112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83443712,"data_stored":83443712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81977344,"data_stored":81977344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":84688896,"data_stored":84688896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:19.711 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:20.346 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:20.346 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:20.385 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-09-16T04:51:18.764713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":159655616,"num_objects":19495,"num_object_clones":0,"num_object_copies":77980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19495,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181992,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7236,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160619,"ondisk_log_size":160619,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2797372,"kb_used_data":313852,"kb_used_omap":14,"kb_used_meta":2483441,"kb_avail":374689988,"statfs":{"total":386547056640,"available":383682547712,"internally_reserved":0,"allocated":321384448,"data_stored":320590412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2543044145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,8,15,9,17,16,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19685376,"num_objects":-2403,"num_object_clones":0,"num_object_copies":-9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2403,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":185,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004898"},"pg_stats":[{"pgid":"1.f","version":"37'11402","reported_seq":11523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:15.266706+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:15.266706+0000","last_peered":"2024-09-16T04:51:15.266706+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:15.266706+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10256384,"num_objects":1252,"num_object_clones":0,"num_object_copies":5008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11402,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-934}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11451","reported_seq":11544,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:14.616741+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:14.616741+0000","last_peered":"2024-09-16T04:51:14.616741+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:14.616741+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10016012,"num_objects":1223,"num_object_clones":0,"num_object_copies":4892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11449,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11466","reported_seq":11569,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:18.505343+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:18.505343+0000","last_peered":"2024-09-16T04:51:18.505343+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:18.505343+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9861790,"num_objects":1205,"num_object_clones":0,"num_object_copies":4820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11465,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1014}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11438","reported_seq":15988,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:17.655930+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:17.655930+0000","last_peered":"2024-09-16T04:51:17.655930+0000","last_clean":"2024-09-16T04:51:17.655930+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:17.655930+0000","last_undegraded":"2024-09-16T04:51:17.655930+0000","last_fullsized":"2024-09-16T04:51:17.655930+0000","mapping_epoch":36,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:21.351408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10089740,"num_objects":1232,"num_object_clones":0,"num_object_copies":4928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11436,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11440","reported_seq":11578,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:15.413404+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:15.413404+0000","last_peered":"2024-09-16T04:51:15.413404+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:15.413404+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10654988,"num_objects":1301,"num_object_clones":0,"num_object_copies":5204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11438,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-947}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11121","reported_seq":11236,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:15.403372+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:15.403372+0000","last_peered":"2024-09-16T04:51:15.403372+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:15.403372+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9756672,"num_objects":1191,"num_object_clones":0,"num_object_copies":4764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11121,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11412","reported_seq":11515,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:14.713284+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:14.713284+0000","last_peered":"2024-09-16T04:51:14.713284+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:14.713284+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9729292,"num_objects":1188,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11410,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1025}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11521","reported_seq":16088,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:17.387799+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:17.387799+0000","last_peered":"2024-09-16T04:51:17.387799+0000","last_clean":"2024-09-16T04:51:17.387799+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:17.387799+0000","last_undegraded":"2024-09-16T04:51:17.387799+0000","last_fullsized":"2024-09-16T04:51:17.387799+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:21.351384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10351884,"num_objects":1264,"num_object_clones":0,"num_object_copies":5056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11519,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11514","reported_seq":16277,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:15.336566+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:15.336566+0000","last_peered":"2024-09-16T04:51:15.336566+0000","last_clean":"2024-09-16T04:51:15.336566+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:15.336566+0000","last_undegraded":"2024-09-16T04:51:15.336566+0000","last_fullsized":"2024-09-16T04:51:15.336566+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:15.571637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10168856,"num_objects":1242,"num_object_clones":0,"num_object_copies":4968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11511,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11538","reported_seq":11648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:18.605300+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:18.605300+0000","last_peered":"2024-09-16T04:51:18.605300+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:18.605300+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9991436,"num_objects":1220,"num_object_clones":0,"num_object_copies":4880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11536,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-969}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11292","reported_seq":11429,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:15.312090+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:15.312090+0000","last_peered":"2024-09-16T04:51:15.312090+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:15.312090+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9818002,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11289,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-956}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11155","reported_seq":11252,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:15.284776+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:15.284776+0000","last_peered":"2024-09-16T04:51:15.284776+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:15.284776+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9632390,"num_objects":1176,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11154,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-929}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11350","reported_seq":11443,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:18.518382+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:18.518382+0000","last_peered":"2024-09-16T04:51:18.518382+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:18.518382+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9689734,"num_objects":1183,"num_object_clones":0,"num_object_copies":4732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11349,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1038}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11519","reported_seq":11652,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:15.192939+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:15.192939+0000","last_peered":"2024-09-16T04:51:15.192939+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:15.192939+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10343692,"num_objects":1263,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11517,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-935}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11256","reported_seq":11645,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:18.674576+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:18.674576+0000","last_peered":"2024-09-16T04:51:18.674576+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:18.674576+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9770252,"num_objects":1193,"num_object_clones":0,"num_object_copies":4772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11254,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-792}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11144","reported_seq":11243,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:14.800411+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:14.800411+0000","last_peered":"2024-09-16T04:51:14.800411+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:14.800411+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9524492,"num_objects":1163,"num_object_clones":0,"num_object_copies":4652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11142,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1065}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":159655616,"num_objects":19495,"num_object_clones":0,"num_object_copies":77980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19495,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181992,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7236,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":319041536,"data_stored":319041536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160619,"ondisk_log_size":160619,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733280,"kb_used_data":79632,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93638560,"statfs":{"total":96636764160,"available":95885885440,"internally_reserved":0,"allocated":81543168,"data_stored":81344659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669314767},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743616,"kb_used_data":76592,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93628224,"statfs":{"total":96636764160,"available":95875301376,"internally_reserved":0,"allocated":78430208,"data_stored":78231699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,3,1,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587396,"kb_used_data":78132,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93784444,"statfs":{"total":96636764160,"available":96035270656,"internally_reserved":0,"allocated":80007168,"data_stored":79808659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521465551},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,0,10,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733080,"kb_used_data":79496,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93638760,"statfs":{"total":96636764160,"available":95886090240,"internally_reserved":0,"allocated":81403904,"data_stored":81205395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":669249230},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81002496,"data_stored":81002496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79421440,"data_stored":79421440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":78008320,"data_stored":78008320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":80609280,"data_stored":80609280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:20.386 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:21.020 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:21.020 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:21.040 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-09-16T04:51:20.765567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154650304,"num_objects":18884,"num_object_clones":0,"num_object_copies":75536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182603,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7236,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160830,"ondisk_log_size":160830,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781288,"kb_used_data":306088,"kb_used_omap":14,"kb_used_meta":2475121,"kb_avail":374706072,"statfs":{"total":386547056640,"available":383699017728,"internally_reserved":0,"allocated":313434112,"data_stored":312640076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2534524465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,10,22,9,14,17,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21504000,"num_objects":-2625,"num_object_clones":0,"num_object_copies":-10500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-83,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2625,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004856"},"pg_stats":[{"pgid":"1.f","version":"37'11478","reported_seq":11598,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.467355+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:20.467355+0000","last_peered":"2024-09-16T04:51:20.467355+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:20.467355+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9633792,"num_objects":1176,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11478,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1010}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11506","reported_seq":11599,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:19.715510+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:19.715510+0000","last_peered":"2024-09-16T04:51:19.715510+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:19.715510+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9565452,"num_objects":1168,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11504,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11466","reported_seq":11569,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:18.505343+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:18.505343+0000","last_peered":"2024-09-16T04:51:18.505343+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:18.505343+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9861790,"num_objects":1205,"num_object_clones":0,"num_object_copies":4820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11465,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1014}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11438","reported_seq":15988,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:17.655930+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:17.655930+0000","last_peered":"2024-09-16T04:51:17.655930+0000","last_clean":"2024-09-16T04:51:17.655930+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:17.655930+0000","last_undegraded":"2024-09-16T04:51:17.655930+0000","last_fullsized":"2024-09-16T04:51:17.655930+0000","mapping_epoch":36,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:21.351408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10089740,"num_objects":1232,"num_object_clones":0,"num_object_copies":4928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11436,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11516","reported_seq":11653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.473534+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:20.473534+0000","last_peered":"2024-09-16T04:51:20.473534+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:20.473534+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10032396,"num_objects":1225,"num_object_clones":0,"num_object_copies":4900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11514,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1023}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11180","reported_seq":11295,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.371600+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:20.371600+0000","last_peered":"2024-09-16T04:51:20.371600+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:20.371600+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9273344,"num_objects":1132,"num_object_clones":0,"num_object_copies":4528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11180,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1046}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11474","reported_seq":11577,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:19.951720+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:19.951720+0000","last_peered":"2024-09-16T04:51:19.951720+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:19.951720+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9221388,"num_objects":1126,"num_object_clones":0,"num_object_copies":4504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11472,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1087}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11521","reported_seq":16088,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:17.387799+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:17.387799+0000","last_peered":"2024-09-16T04:51:17.387799+0000","last_clean":"2024-09-16T04:51:17.387799+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:17.387799+0000","last_undegraded":"2024-09-16T04:51:17.387799+0000","last_fullsized":"2024-09-16T04:51:17.387799+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:21.351384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10351884,"num_objects":1264,"num_object_clones":0,"num_object_copies":5056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11519,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11570","reported_seq":16333,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:20.024853+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:20.024853+0000","last_peered":"2024-09-16T04:51:20.024853+0000","last_clean":"2024-09-16T04:51:20.024853+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:20.024853+0000","last_undegraded":"2024-09-16T04:51:20.024853+0000","last_fullsized":"2024-09-16T04:51:20.024853+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:20.665152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9710104,"num_objects":1186,"num_object_clones":0,"num_object_copies":4744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11567,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11538","reported_seq":11648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:18.605300+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:18.605300+0000","last_peered":"2024-09-16T04:51:18.605300+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:18.605300+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9991436,"num_objects":1220,"num_object_clones":0,"num_object_copies":4880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11536,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-969}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11339","reported_seq":11476,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.407915+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:20.407915+0000","last_peered":"2024-09-16T04:51:20.407915+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:20.407915+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9432978,"num_objects":1152,"num_object_clones":0,"num_object_copies":4608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11336,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1003}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11223","reported_seq":11320,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.466450+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:20.466450+0000","last_peered":"2024-09-16T04:51:20.466450+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:20.466450+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075334,"num_objects":1108,"num_object_clones":0,"num_object_copies":4432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11222,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-997}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11350","reported_seq":11443,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:18.518382+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:18.518382+0000","last_peered":"2024-09-16T04:51:18.518382+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:18.518382+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9689734,"num_objects":1183,"num_object_clones":0,"num_object_copies":4732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11349,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1038}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11583","reported_seq":11719,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.327849+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:20.327849+0000","last_peered":"2024-09-16T04:51:20.327849+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:20.327849+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9819404,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11581,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-999}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11256","reported_seq":11645,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:18.674576+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:18.674576+0000","last_peered":"2024-09-16T04:51:18.674576+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:18.674576+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9770252,"num_objects":1193,"num_object_clones":0,"num_object_copies":4772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11254,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-792}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11192","reported_seq":11291,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:19.745911+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:19.745911+0000","last_peered":"2024-09-16T04:51:19.745911+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:19.745911+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9131276,"num_objects":1115,"num_object_clones":0,"num_object_copies":4460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11190,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154650304,"num_objects":18884,"num_object_clones":0,"num_object_copies":75536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182603,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7236,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":310968320,"data_stored":310968320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160830,"ondisk_log_size":160830,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733548,"kb_used_data":75740,"kb_used_omap":4,"kb_used_meta":657787,"kb_avail":93638292,"statfs":{"total":96636764160,"available":95885611008,"internally_reserved":0,"allocated":77557760,"data_stored":77359251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":673574607},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,6,4,3,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743616,"kb_used_data":76592,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93628224,"statfs":{"total":96636764160,"available":95875301376,"internally_reserved":0,"allocated":78430208,"data_stored":78231699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,3,1,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587396,"kb_used_data":78132,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93784444,"statfs":{"total":96636764160,"available":96035270656,"internally_reserved":0,"allocated":80007168,"data_stored":79808659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521465551},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,0,10,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716728,"kb_used_data":75624,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93655112,"statfs":{"total":96636764160,"available":95902834688,"internally_reserved":0,"allocated":77438976,"data_stored":77240467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":656469710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,7,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":76926976,"data_stored":76926976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79421440,"data_stored":79421440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":78008320,"data_stored":78008320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":76611584,"data_stored":76611584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:21.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:51:21.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11478", 'reported_seq': 11598, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:20.467355+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:51:20.467355+0000', 'last_peered': '2024-09-16T04:51:20.467355+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:51:20.467355+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9633792, 'num_objects': 1176, 'num_object_clones': 0, 'num_object_copies': 4704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1176, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11478, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1010}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:21.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:51:21.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11506", 'reported_seq': 11599, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:19.715510+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:51:19.715510+0000', 'last_peered': '2024-09-16T04:51:19.715510+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:51:19.715510+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9565452, 'num_objects': 1168, 'num_object_clones': 0, 'num_object_copies': 4672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11504, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1035}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:21.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:51:21.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11466", 'reported_seq': 11569, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:18.505343+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:51:18.505343+0000', 'last_peered': '2024-09-16T04:51:18.505343+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:51:18.505343+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9861790, 'num_objects': 1205, 'num_object_clones': 0, 'num_object_copies': 4820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1205, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11465, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1014}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:21.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:51:21.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11516", 'reported_seq': 11653, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:20.473534+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:51:20.473534+0000', 'last_peered': '2024-09-16T04:51:20.473534+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:51:20.473534+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10032396, 'num_objects': 1225, 'num_object_clones': 0, 'num_object_copies': 4900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1225, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11514, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1023}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:21.042 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:51:21.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11180", 'reported_seq': 11295, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:20.371600+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:51:20.371600+0000', 'last_peered': '2024-09-16T04:51:20.371600+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:51:20.371600+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9273344, 'num_objects': 1132, 'num_object_clones': 0, 'num_object_copies': 4528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1132, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11180, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1046}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:21.042 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:51:21.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11474", 'reported_seq': 11577, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:19.951720+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:51:19.951720+0000', 'last_peered': '2024-09-16T04:51:19.951720+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:51:19.951720+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9221388, 'num_objects': 1126, 'num_object_clones': 0, 'num_object_copies': 4504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11472, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1087}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:21.042 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:51:21.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11538", 'reported_seq': 11648, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:18.605300+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:51:18.605300+0000', 'last_peered': '2024-09-16T04:51:18.605300+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:51:18.605300+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9991436, 'num_objects': 1220, 'num_object_clones': 0, 'num_object_copies': 4880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11536, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -969}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:21.042 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:51:21.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11339", 'reported_seq': 11476, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:20.407915+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:51:20.407915+0000', 'last_peered': '2024-09-16T04:51:20.407915+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:51:20.407915+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9432978, 'num_objects': 1152, 'num_object_clones': 0, 'num_object_copies': 4608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11336, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1003}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:21.043 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:51:21.043 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11223", 'reported_seq': 11320, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:20.466450+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:51:20.466450+0000', 'last_peered': '2024-09-16T04:51:20.466450+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:51:20.466450+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9075334, 'num_objects': 1108, 'num_object_clones': 0, 'num_object_copies': 4432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11222, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -997}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:21.043 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:51:21.043 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11350", 'reported_seq': 11443, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:18.518382+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:51:18.518382+0000', 'last_peered': '2024-09-16T04:51:18.518382+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:51:18.518382+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9689734, 'num_objects': 1183, 'num_object_clones': 0, 'num_object_copies': 4732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11349, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1038}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:21.043 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:51:21.043 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11583", 'reported_seq': 11719, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:20.327849+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:51:20.327849+0000', 'last_peered': '2024-09-16T04:51:20.327849+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:51:20.327849+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9819404, 'num_objects': 1199, 'num_object_clones': 0, 'num_object_copies': 4796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11581, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -999}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:21.043 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:51:21.043 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11256", 'reported_seq': 11645, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:18.674576+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:51:18.674576+0000', 'last_peered': '2024-09-16T04:51:18.674576+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:51:18.674576+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9770252, 'num_objects': 1193, 'num_object_clones': 0, 'num_object_copies': 4772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11254, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 151, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -792}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:21.043 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:51:21.043 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11192", 'reported_seq': 11291, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:19.745911+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:51:19.745911+0000', 'last_peered': '2024-09-16T04:51:19.745911+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:51:19.745911+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9131276, 'num_objects': 1115, 'num_object_clones': 0, 'num_object_copies': 4460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1115, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11190, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1113}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:21.044 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:51:21.744 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:21.744 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28346/75536 objects degraded (37.526%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":694,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18884,"data_bytes":154650304,"bytes_used":2848038912,"bytes_avail":383699017728,"bytes_total":386547056640,"degraded_objects":28346,"degraded_total":75536,"degraded_ratio":0.3752647744122008,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":218},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:51:21.772 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:51:21.772 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:22.352 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:22.352 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:22.388 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-09-16T04:51:20.765567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154650304,"num_objects":18884,"num_object_clones":0,"num_object_copies":75536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182603,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7236,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160830,"ondisk_log_size":160830,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781288,"kb_used_data":306088,"kb_used_omap":14,"kb_used_meta":2475121,"kb_avail":374706072,"statfs":{"total":386547056640,"available":383699017728,"internally_reserved":0,"allocated":313434112,"data_stored":312640076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2534524465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,10,22,9,14,17,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21504000,"num_objects":-2625,"num_object_clones":0,"num_object_copies":-10500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-83,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2625,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004856"},"pg_stats":[{"pgid":"1.f","version":"37'11478","reported_seq":11598,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.467355+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:20.467355+0000","last_peered":"2024-09-16T04:51:20.467355+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:20.467355+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9633792,"num_objects":1176,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11478,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1010}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11506","reported_seq":11599,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:19.715510+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:19.715510+0000","last_peered":"2024-09-16T04:51:19.715510+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:19.715510+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9565452,"num_objects":1168,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11504,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11466","reported_seq":11569,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:18.505343+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:18.505343+0000","last_peered":"2024-09-16T04:51:18.505343+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:18.505343+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9861790,"num_objects":1205,"num_object_clones":0,"num_object_copies":4820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11465,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1014}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11438","reported_seq":15988,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:17.655930+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:17.655930+0000","last_peered":"2024-09-16T04:51:17.655930+0000","last_clean":"2024-09-16T04:51:17.655930+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:17.655930+0000","last_undegraded":"2024-09-16T04:51:17.655930+0000","last_fullsized":"2024-09-16T04:51:17.655930+0000","mapping_epoch":36,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:21.351408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10089740,"num_objects":1232,"num_object_clones":0,"num_object_copies":4928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11436,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11516","reported_seq":11653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.473534+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:20.473534+0000","last_peered":"2024-09-16T04:51:20.473534+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:20.473534+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10032396,"num_objects":1225,"num_object_clones":0,"num_object_copies":4900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11514,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1023}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11180","reported_seq":11295,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.371600+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:20.371600+0000","last_peered":"2024-09-16T04:51:20.371600+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:20.371600+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9273344,"num_objects":1132,"num_object_clones":0,"num_object_copies":4528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11180,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1046}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11474","reported_seq":11577,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:19.951720+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:19.951720+0000","last_peered":"2024-09-16T04:51:19.951720+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:19.951720+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9221388,"num_objects":1126,"num_object_clones":0,"num_object_copies":4504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11472,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1087}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11521","reported_seq":16088,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:17.387799+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:17.387799+0000","last_peered":"2024-09-16T04:51:17.387799+0000","last_clean":"2024-09-16T04:51:17.387799+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:17.387799+0000","last_undegraded":"2024-09-16T04:51:17.387799+0000","last_fullsized":"2024-09-16T04:51:17.387799+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:21.351384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10351884,"num_objects":1264,"num_object_clones":0,"num_object_copies":5056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11519,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11570","reported_seq":16333,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:20.024853+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:20.024853+0000","last_peered":"2024-09-16T04:51:20.024853+0000","last_clean":"2024-09-16T04:51:20.024853+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:20.024853+0000","last_undegraded":"2024-09-16T04:51:20.024853+0000","last_fullsized":"2024-09-16T04:51:20.024853+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:20.665152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9710104,"num_objects":1186,"num_object_clones":0,"num_object_copies":4744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11567,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11538","reported_seq":11648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:18.605300+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:18.605300+0000","last_peered":"2024-09-16T04:51:18.605300+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:18.605300+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9991436,"num_objects":1220,"num_object_clones":0,"num_object_copies":4880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11536,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-969}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11339","reported_seq":11476,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.407915+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:20.407915+0000","last_peered":"2024-09-16T04:51:20.407915+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:20.407915+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9432978,"num_objects":1152,"num_object_clones":0,"num_object_copies":4608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11336,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1003}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11223","reported_seq":11320,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.466450+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:20.466450+0000","last_peered":"2024-09-16T04:51:20.466450+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:20.466450+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075334,"num_objects":1108,"num_object_clones":0,"num_object_copies":4432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11222,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-997}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11350","reported_seq":11443,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:18.518382+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:18.518382+0000","last_peered":"2024-09-16T04:51:18.518382+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:18.518382+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9689734,"num_objects":1183,"num_object_clones":0,"num_object_copies":4732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11349,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1038}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11583","reported_seq":11719,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.327849+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:20.327849+0000","last_peered":"2024-09-16T04:51:20.327849+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:20.327849+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9819404,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11581,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-999}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11256","reported_seq":11645,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:18.674576+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:18.674576+0000","last_peered":"2024-09-16T04:51:18.674576+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:18.674576+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9770252,"num_objects":1193,"num_object_clones":0,"num_object_copies":4772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11254,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-792}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11192","reported_seq":11291,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:19.745911+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:19.745911+0000","last_peered":"2024-09-16T04:51:19.745911+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:19.745911+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9131276,"num_objects":1115,"num_object_clones":0,"num_object_copies":4460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11190,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154650304,"num_objects":18884,"num_object_clones":0,"num_object_copies":75536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182603,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7236,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":310968320,"data_stored":310968320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160830,"ondisk_log_size":160830,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733548,"kb_used_data":75740,"kb_used_omap":4,"kb_used_meta":657787,"kb_avail":93638292,"statfs":{"total":96636764160,"available":95885611008,"internally_reserved":0,"allocated":77557760,"data_stored":77359251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":673574607},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,6,4,3,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743616,"kb_used_data":76592,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93628224,"statfs":{"total":96636764160,"available":95875301376,"internally_reserved":0,"allocated":78430208,"data_stored":78231699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,3,1,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587396,"kb_used_data":78132,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93784444,"statfs":{"total":96636764160,"available":96035270656,"internally_reserved":0,"allocated":80007168,"data_stored":79808659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521465551},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,0,10,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716728,"kb_used_data":75624,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93655112,"statfs":{"total":96636764160,"available":95902834688,"internally_reserved":0,"allocated":77438976,"data_stored":77240467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":656469710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,7,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":76926976,"data_stored":76926976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79421440,"data_stored":79421440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":78008320,"data_stored":78008320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":76611584,"data_stored":76611584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:25.390 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:25.961 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:25.962 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:25.988 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-09-16T04:51:24.767217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":151627456,"num_objects":18515,"num_object_clones":0,"num_object_copies":74060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18515,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182972,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7283,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160699,"ondisk_log_size":160699,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765364,"kb_used_data":298472,"kb_used_omap":14,"kb_used_meta":2466801,"kb_avail":374721996,"statfs":{"total":386547056640,"available":383715323904,"internally_reserved":0,"allocated":305635328,"data_stored":304841292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2526004785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,7,17,11,13,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18128896,"num_objects":-2213,"num_object_clones":0,"num_object_copies":-8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2213,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004792"},"pg_stats":[{"pgid":"1.f","version":"37'11478","reported_seq":11598,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.467355+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:20.467355+0000","last_peered":"2024-09-16T04:51:20.467355+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:20.467355+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9633792,"num_objects":1176,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11478,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1010}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11506","reported_seq":11599,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:19.715510+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:19.715510+0000","last_peered":"2024-09-16T04:51:19.715510+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:19.715510+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9565452,"num_objects":1168,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11504,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11530","reported_seq":11633,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:23.621472+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:23.621472+0000","last_peered":"2024-09-16T04:51:23.621472+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:23.621472+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9337502,"num_objects":1141,"num_object_clones":0,"num_object_copies":4564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11529,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1078}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11504","reported_seq":16054,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:22.587646+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:22.587646+0000","last_peered":"2024-09-16T04:51:22.587646+0000","last_clean":"2024-09-16T04:51:22.587646+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:22.587646+0000","last_undegraded":"2024-09-16T04:51:22.587646+0000","last_fullsized":"2024-09-16T04:51:22.587646+0000","mapping_epoch":36,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:26.399766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9549068,"num_objects":1166,"num_object_clones":0,"num_object_copies":4664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11502,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11516","reported_seq":11653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.473534+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:20.473534+0000","last_peered":"2024-09-16T04:51:20.473534+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:20.473534+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10032396,"num_objects":1225,"num_object_clones":0,"num_object_copies":4900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11514,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1023}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11180","reported_seq":11295,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.371600+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:20.371600+0000","last_peered":"2024-09-16T04:51:20.371600+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:20.371600+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9273344,"num_objects":1132,"num_object_clones":0,"num_object_copies":4528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11180,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1046}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11474","reported_seq":11577,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:19.951720+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:19.951720+0000","last_peered":"2024-09-16T04:51:19.951720+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:19.951720+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9221388,"num_objects":1126,"num_object_clones":0,"num_object_copies":4504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11472,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1087}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11586","reported_seq":16153,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:22.647357+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:22.647357+0000","last_peered":"2024-09-16T04:51:22.647357+0000","last_clean":"2024-09-16T04:51:22.647357+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:22.647357+0000","last_undegraded":"2024-09-16T04:51:22.647357+0000","last_fullsized":"2024-09-16T04:51:22.647357+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:26.372805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9819404,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11570","reported_seq":16333,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:20.024853+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:20.024853+0000","last_peered":"2024-09-16T04:51:20.024853+0000","last_clean":"2024-09-16T04:51:20.024853+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:20.024853+0000","last_undegraded":"2024-09-16T04:51:20.024853+0000","last_fullsized":"2024-09-16T04:51:20.024853+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:20.665152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9710104,"num_objects":1186,"num_object_clones":0,"num_object_copies":4744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11567,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11604","reported_seq":11716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:23.726925+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:23.726925+0000","last_peered":"2024-09-16T04:51:23.726925+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:23.726925+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9450764,"num_objects":1154,"num_object_clones":0,"num_object_copies":4616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11602,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1035}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11339","reported_seq":11476,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.407915+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:20.407915+0000","last_peered":"2024-09-16T04:51:20.407915+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:20.407915+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9432978,"num_objects":1152,"num_object_clones":0,"num_object_copies":4608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11336,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1003}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11223","reported_seq":11320,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.466450+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:20.466450+0000","last_peered":"2024-09-16T04:51:20.466450+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:20.466450+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075334,"num_objects":1108,"num_object_clones":0,"num_object_copies":4432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11222,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-997}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11405","reported_seq":11498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:23.718082+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:23.718082+0000","last_peered":"2024-09-16T04:51:23.718082+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:23.718082+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9239174,"num_objects":1128,"num_object_clones":0,"num_object_copies":4512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11404,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1093}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11583","reported_seq":11719,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.327849+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:20.327849+0000","last_peered":"2024-09-16T04:51:20.327849+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:20.327849+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9819404,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11581,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-999}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11309","reported_seq":11793,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:23.692956+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:23.692956+0000","last_peered":"2024-09-16T04:51:23.692956+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:23.692956+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9336076,"num_objects":1140,"num_object_clones":0,"num_object_copies":4560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11307,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-798}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11192","reported_seq":11291,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:19.745911+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:19.745911+0000","last_peered":"2024-09-16T04:51:19.745911+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:19.745911+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9131276,"num_objects":1115,"num_object_clones":0,"num_object_copies":4460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11190,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":151627456,"num_objects":18515,"num_object_clones":0,"num_object_copies":74060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18515,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182972,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7283,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":303157248,"data_stored":303157248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160699,"ondisk_log_size":160699,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733548,"kb_used_data":75740,"kb_used_omap":4,"kb_used_meta":657787,"kb_avail":93638292,"statfs":{"total":96636764160,"available":95885611008,"internally_reserved":0,"allocated":77557760,"data_stored":77359251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":673574607},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,6,4,3,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744016,"kb_used_data":72832,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93627824,"statfs":{"total":96636764160,"available":95874891776,"internally_reserved":0,"allocated":74579968,"data_stored":74381459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571072,"kb_used_data":74276,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93800768,"statfs":{"total":96636764160,"available":96051986432,"internally_reserved":0,"allocated":76058624,"data_stored":75860115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508686031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,7,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716728,"kb_used_data":75624,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93655112,"statfs":{"total":96636764160,"available":95902834688,"internally_reserved":0,"allocated":77438976,"data_stored":77240467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":656469710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,7,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":76926976,"data_stored":76926976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75505664,"data_stored":75505664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":74113024,"data_stored":74113024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":76611584,"data_stored":76611584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:25.990 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:26.486 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:26.487 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:26.513 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-09-16T04:51:24.767217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":151627456,"num_objects":18515,"num_object_clones":0,"num_object_copies":74060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18515,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182972,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7283,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160699,"ondisk_log_size":160699,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765364,"kb_used_data":298472,"kb_used_omap":14,"kb_used_meta":2466801,"kb_avail":374721996,"statfs":{"total":386547056640,"available":383715323904,"internally_reserved":0,"allocated":305635328,"data_stored":304841292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2526004785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,7,17,11,13,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18128896,"num_objects":-2213,"num_object_clones":0,"num_object_copies":-8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2213,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004792"},"pg_stats":[{"pgid":"1.f","version":"37'11478","reported_seq":11598,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.467355+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:20.467355+0000","last_peered":"2024-09-16T04:51:20.467355+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:20.467355+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9633792,"num_objects":1176,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11478,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1010}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11506","reported_seq":11599,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:19.715510+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:19.715510+0000","last_peered":"2024-09-16T04:51:19.715510+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:19.715510+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9565452,"num_objects":1168,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11504,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1035}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11530","reported_seq":11633,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:23.621472+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:23.621472+0000","last_peered":"2024-09-16T04:51:23.621472+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:23.621472+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9337502,"num_objects":1141,"num_object_clones":0,"num_object_copies":4564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11529,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1078}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11504","reported_seq":16054,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:22.587646+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:22.587646+0000","last_peered":"2024-09-16T04:51:22.587646+0000","last_clean":"2024-09-16T04:51:22.587646+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:22.587646+0000","last_undegraded":"2024-09-16T04:51:22.587646+0000","last_fullsized":"2024-09-16T04:51:22.587646+0000","mapping_epoch":36,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:26.399766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9549068,"num_objects":1166,"num_object_clones":0,"num_object_copies":4664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11502,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11516","reported_seq":11653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.473534+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:20.473534+0000","last_peered":"2024-09-16T04:51:20.473534+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:20.473534+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10032396,"num_objects":1225,"num_object_clones":0,"num_object_copies":4900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11514,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1023}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11180","reported_seq":11295,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.371600+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:20.371600+0000","last_peered":"2024-09-16T04:51:20.371600+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:20.371600+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9273344,"num_objects":1132,"num_object_clones":0,"num_object_copies":4528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11180,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1046}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11474","reported_seq":11577,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:19.951720+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:19.951720+0000","last_peered":"2024-09-16T04:51:19.951720+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:19.951720+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9221388,"num_objects":1126,"num_object_clones":0,"num_object_copies":4504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11472,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1087}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11586","reported_seq":16153,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:22.647357+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:22.647357+0000","last_peered":"2024-09-16T04:51:22.647357+0000","last_clean":"2024-09-16T04:51:22.647357+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:22.647357+0000","last_undegraded":"2024-09-16T04:51:22.647357+0000","last_fullsized":"2024-09-16T04:51:22.647357+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:26.372805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9819404,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11570","reported_seq":16333,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:20.024853+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:20.024853+0000","last_peered":"2024-09-16T04:51:20.024853+0000","last_clean":"2024-09-16T04:51:20.024853+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:20.024853+0000","last_undegraded":"2024-09-16T04:51:20.024853+0000","last_fullsized":"2024-09-16T04:51:20.024853+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:20.665152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9710104,"num_objects":1186,"num_object_clones":0,"num_object_copies":4744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11567,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11604","reported_seq":11716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:23.726925+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:23.726925+0000","last_peered":"2024-09-16T04:51:23.726925+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:23.726925+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9450764,"num_objects":1154,"num_object_clones":0,"num_object_copies":4616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11602,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1035}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11339","reported_seq":11476,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.407915+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:20.407915+0000","last_peered":"2024-09-16T04:51:20.407915+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:20.407915+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9432978,"num_objects":1152,"num_object_clones":0,"num_object_copies":4608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11336,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1003}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11223","reported_seq":11320,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.466450+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:20.466450+0000","last_peered":"2024-09-16T04:51:20.466450+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:20.466450+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075334,"num_objects":1108,"num_object_clones":0,"num_object_copies":4432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11222,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-997}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11405","reported_seq":11498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:23.718082+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:23.718082+0000","last_peered":"2024-09-16T04:51:23.718082+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:23.718082+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9239174,"num_objects":1128,"num_object_clones":0,"num_object_copies":4512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11404,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1093}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11583","reported_seq":11719,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:20.327849+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:20.327849+0000","last_peered":"2024-09-16T04:51:20.327849+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:20.327849+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9819404,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11581,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-999}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11309","reported_seq":11793,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:23.692956+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:23.692956+0000","last_peered":"2024-09-16T04:51:23.692956+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:23.692956+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9336076,"num_objects":1140,"num_object_clones":0,"num_object_copies":4560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11307,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-798}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11192","reported_seq":11291,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:19.745911+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:19.745911+0000","last_peered":"2024-09-16T04:51:19.745911+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:19.745911+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9131276,"num_objects":1115,"num_object_clones":0,"num_object_copies":4460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11190,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":151627456,"num_objects":18515,"num_object_clones":0,"num_object_copies":74060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18515,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182972,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7283,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":303157248,"data_stored":303157248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160699,"ondisk_log_size":160699,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733548,"kb_used_data":75740,"kb_used_omap":4,"kb_used_meta":657787,"kb_avail":93638292,"statfs":{"total":96636764160,"available":95885611008,"internally_reserved":0,"allocated":77557760,"data_stored":77359251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":673574607},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,6,4,3,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744016,"kb_used_data":72832,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93627824,"statfs":{"total":96636764160,"available":95874891776,"internally_reserved":0,"allocated":74579968,"data_stored":74381459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571072,"kb_used_data":74276,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93800768,"statfs":{"total":96636764160,"available":96051986432,"internally_reserved":0,"allocated":76058624,"data_stored":75860115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508686031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,7,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716728,"kb_used_data":75624,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93655112,"statfs":{"total":96636764160,"available":95902834688,"internally_reserved":0,"allocated":77438976,"data_stored":77240467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":656469710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,7,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":76926976,"data_stored":76926976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75505664,"data_stored":75505664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":74113024,"data_stored":74113024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":76611584,"data_stored":76611584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:26.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:51:26.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11478", 'reported_seq': 11598, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:20.467355+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:51:20.467355+0000', 'last_peered': '2024-09-16T04:51:20.467355+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:51:20.467355+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9633792, 'num_objects': 1176, 'num_object_clones': 0, 'num_object_copies': 4704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1176, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11478, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1010}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:26.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:51:26.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11506", 'reported_seq': 11599, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:19.715510+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:51:19.715510+0000', 'last_peered': '2024-09-16T04:51:19.715510+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:51:19.715510+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9565452, 'num_objects': 1168, 'num_object_clones': 0, 'num_object_copies': 4672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11504, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1035}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:26.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:51:26.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11530", 'reported_seq': 11633, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:23.621472+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:51:23.621472+0000', 'last_peered': '2024-09-16T04:51:23.621472+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:51:23.621472+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9337502, 'num_objects': 1141, 'num_object_clones': 0, 'num_object_copies': 4564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1141, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11529, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1078}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:26.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:51:26.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11516", 'reported_seq': 11653, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:20.473534+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:51:20.473534+0000', 'last_peered': '2024-09-16T04:51:20.473534+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:51:20.473534+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10032396, 'num_objects': 1225, 'num_object_clones': 0, 'num_object_copies': 4900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1225, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11514, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1023}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:26.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:51:26.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11180", 'reported_seq': 11295, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:20.371600+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:51:20.371600+0000', 'last_peered': '2024-09-16T04:51:20.371600+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:51:20.371600+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9273344, 'num_objects': 1132, 'num_object_clones': 0, 'num_object_copies': 4528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1132, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11180, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1046}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:26.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:51:26.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11474", 'reported_seq': 11577, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:19.951720+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:51:19.951720+0000', 'last_peered': '2024-09-16T04:51:19.951720+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:51:19.951720+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9221388, 'num_objects': 1126, 'num_object_clones': 0, 'num_object_copies': 4504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11472, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1087}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:26.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:51:26.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11604", 'reported_seq': 11716, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:23.726925+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:51:23.726925+0000', 'last_peered': '2024-09-16T04:51:23.726925+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:51:23.726925+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9450764, 'num_objects': 1154, 'num_object_clones': 0, 'num_object_copies': 4616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1154, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11602, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1035}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:26.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:51:26.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11339", 'reported_seq': 11476, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:20.407915+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:51:20.407915+0000', 'last_peered': '2024-09-16T04:51:20.407915+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:51:20.407915+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9432978, 'num_objects': 1152, 'num_object_clones': 0, 'num_object_copies': 4608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11336, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1003}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:26.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:51:26.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11223", 'reported_seq': 11320, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:20.466450+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:51:20.466450+0000', 'last_peered': '2024-09-16T04:51:20.466450+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:51:20.466450+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9075334, 'num_objects': 1108, 'num_object_clones': 0, 'num_object_copies': 4432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11222, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -997}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:26.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:51:26.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11405", 'reported_seq': 11498, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:23.718082+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:51:23.718082+0000', 'last_peered': '2024-09-16T04:51:23.718082+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:51:23.718082+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9239174, 'num_objects': 1128, 'num_object_clones': 0, 'num_object_copies': 4512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11404, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1093}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:26.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:51:26.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11583", 'reported_seq': 11719, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:20.327849+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:51:20.327849+0000', 'last_peered': '2024-09-16T04:51:20.327849+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:51:20.327849+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9819404, 'num_objects': 1199, 'num_object_clones': 0, 'num_object_copies': 4796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11581, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -999}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:26.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:51:26.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11309", 'reported_seq': 11793, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:23.692956+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:51:23.692956+0000', 'last_peered': '2024-09-16T04:51:23.692956+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:51:23.692956+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9336076, 'num_objects': 1140, 'num_object_clones': 0, 'num_object_copies': 4560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1938, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1140, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11307, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 198, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -798}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:26.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:51:26.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11192", 'reported_seq': 11291, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:19.745911+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:51:19.745911+0000', 'last_peered': '2024-09-16T04:51:19.745911+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:51:19.745911+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9131276, 'num_objects': 1115, 'num_object_clones': 0, 'num_object_copies': 4460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1115, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11190, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1113}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:26.517 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:51:27.283 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:27.283 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28299/71736 objects degraded (39.449%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":699,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":17934,"data_bytes":146867904,"bytes_used":2815557632,"bytes_avail":383731499008,"bytes_total":386547056640,"degraded_objects":28299,"degraded_total":71736,"degraded_ratio":0.39448812311809972,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":210},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:51:27.315 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:51:27.316 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:27.953 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:27.953 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:27.981 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-09-16T04:51:26.768116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":146867904,"num_objects":17934,"num_object_clones":0,"num_object_copies":71736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183553,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7283,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160580,"ondisk_log_size":160580,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749568,"kb_used_data":290988,"kb_used_omap":14,"kb_used_meta":2458481,"kb_avail":374737792,"statfs":{"total":386547056640,"available":383731499008,"internally_reserved":0,"allocated":297971712,"data_stored":297177676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2517485105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,6,10,10,13,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20725760,"num_objects":-2530,"num_object_clones":0,"num_object_copies":-10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2530,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004895"},"pg_stats":[{"pgid":"1.f","version":"37'11535","reported_seq":11656,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:25.291599+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:25.291599+0000","last_peered":"2024-09-16T04:51:25.291599+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:25.291599+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9166848,"num_objects":1119,"num_object_clones":0,"num_object_copies":4476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11535,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11572","reported_seq":11665,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:24.907734+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:24.907734+0000","last_peered":"2024-09-16T04:51:24.907734+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:24.907734+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9024780,"num_objects":1102,"num_object_clones":0,"num_object_copies":4408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11570,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1101}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11530","reported_seq":11633,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:23.621472+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:23.621472+0000","last_peered":"2024-09-16T04:51:23.621472+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:23.621472+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9337502,"num_objects":1141,"num_object_clones":0,"num_object_copies":4564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11529,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1078}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11504","reported_seq":16054,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:22.587646+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:22.587646+0000","last_peered":"2024-09-16T04:51:22.587646+0000","last_clean":"2024-09-16T04:51:22.587646+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:22.587646+0000","last_undegraded":"2024-09-16T04:51:22.587646+0000","last_fullsized":"2024-09-16T04:51:22.587646+0000","mapping_epoch":36,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:26.399766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9549068,"num_objects":1166,"num_object_clones":0,"num_object_copies":4664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11502,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11579","reported_seq":11717,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:25.404022+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:25.404022+0000","last_peered":"2024-09-16T04:51:25.404022+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:25.404022+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9516300,"num_objects":1162,"num_object_clones":0,"num_object_copies":4648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11577,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1086}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11246","reported_seq":11360,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:25.379130+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:25.379130+0000","last_peered":"2024-09-16T04:51:25.379130+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:25.379130+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8732672,"num_objects":1066,"num_object_clones":0,"num_object_copies":4264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11246,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11531","reported_seq":11634,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:24.998846+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:24.998846+0000","last_peered":"2024-09-16T04:51:24.998846+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:24.998846+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8754444,"num_objects":1069,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11529,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1144}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11586","reported_seq":16153,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:22.647357+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:22.647357+0000","last_peered":"2024-09-16T04:51:22.647357+0000","last_clean":"2024-09-16T04:51:22.647357+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:22.647357+0000","last_undegraded":"2024-09-16T04:51:22.647357+0000","last_fullsized":"2024-09-16T04:51:22.647357+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:26.372805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9819404,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11617","reported_seq":16380,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:25.370085+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:25.370085+0000","last_peered":"2024-09-16T04:51:25.370085+0000","last_clean":"2024-09-16T04:51:25.370085+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:25.370085+0000","last_undegraded":"2024-09-16T04:51:25.370085+0000","last_fullsized":"2024-09-16T04:51:25.370085+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:26.624641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9325080,"num_objects":1139,"num_object_clones":0,"num_object_copies":4556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11614,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11604","reported_seq":11716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:23.726925+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:23.726925+0000","last_peered":"2024-09-16T04:51:23.726925+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:23.726925+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9450764,"num_objects":1154,"num_object_clones":0,"num_object_copies":4616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11602,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1035}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11401","reported_seq":11539,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:25.285311+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:25.285311+0000","last_peered":"2024-09-16T04:51:25.285311+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:25.285311+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8925074,"num_objects":1090,"num_object_clones":0,"num_object_copies":4360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11398,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1065}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11278","reported_seq":11374,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:25.349267+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:25.349267+0000","last_peered":"2024-09-16T04:51:25.349267+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:25.349267+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8624774,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11277,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1052}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11405","reported_seq":11498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:23.718082+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:23.718082+0000","last_peered":"2024-09-16T04:51:23.718082+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:23.718082+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9239174,"num_objects":1128,"num_object_clones":0,"num_object_copies":4512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11404,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1093}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11644","reported_seq":11780,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:25.488094+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:25.488094+0000","last_peered":"2024-09-16T04:51:25.488094+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:25.488094+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9319692,"num_objects":1138,"num_object_clones":0,"num_object_copies":4552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11642,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1060}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11309","reported_seq":11793,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:23.692956+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:23.692956+0000","last_peered":"2024-09-16T04:51:23.692956+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:23.692956+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9336076,"num_objects":1140,"num_object_clones":0,"num_object_copies":4560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11307,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-798}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11239","reported_seq":11337,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:24.999954+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:24.999954+0000","last_peered":"2024-09-16T04:51:24.999954+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:24.999954+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8746252,"num_objects":1068,"num_object_clones":0,"num_object_copies":4272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11237,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1160}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":146867904,"num_objects":17934,"num_object_clones":0,"num_object_copies":71736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183553,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7283,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":295280640,"data_stored":295280640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160580,"ondisk_log_size":160580,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717340,"kb_used_data":72004,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93654500,"statfs":{"total":96636764160,"available":95902208000,"internally_reserved":0,"allocated":73732096,"data_stored":73533587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":660795087},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,6,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744016,"kb_used_data":72832,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93627824,"statfs":{"total":96636764160,"available":95874891776,"internally_reserved":0,"allocated":74579968,"data_stored":74381459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571072,"kb_used_data":74276,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93800768,"statfs":{"total":96636764160,"available":96051986432,"internally_reserved":0,"allocated":76058624,"data_stored":75860115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508686031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,7,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717140,"kb_used_data":71876,"kb_used_omap":4,"kb_used_meta":645243,"kb_avail":93654700,"statfs":{"total":96636764160,"available":95902412800,"internally_reserved":0,"allocated":73601024,"data_stored":73402515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":660729550},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73027584,"data_stored":73027584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75505664,"data_stored":75505664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":74113024,"data_stored":74113024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":72634368,"data_stored":72634368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:30.984 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:31.634 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:31.635 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:31.659 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-09-16T04:51:30.769432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":139224768,"num_objects":17001,"num_object_clones":0,"num_object_copies":68004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17001,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184486,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7341,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160913,"ondisk_log_size":160913,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769468,"kb_used_data":276216,"kb_used_omap":14,"kb_used_meta":2493169,"kb_avail":374717892,"statfs":{"total":386547056640,"available":383711121408,"internally_reserved":0,"allocated":282845184,"data_stored":282051148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2553005617},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,1,0,8,13,15,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20430848,"num_objects":-2494,"num_object_clones":0,"num_object_copies":-9976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2494,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004719"},"pg_stats":[{"pgid":"1.f","version":"37'11593","reported_seq":11714,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.495165+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:30.495165+0000","last_peered":"2024-09-16T04:51:30.495165+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:30.495165+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8691712,"num_objects":1061,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11637","reported_seq":11729,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.018068+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:30.018068+0000","last_peered":"2024-09-16T04:51:30.018068+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:30.018068+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8492300,"num_objects":1037,"num_object_clones":0,"num_object_copies":4148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1166}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11595","reported_seq":11698,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:28.687183+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:28.687183+0000","last_peered":"2024-09-16T04:51:28.687183+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:28.687183+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8805022,"num_objects":1076,"num_object_clones":0,"num_object_copies":4304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1076,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11594,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1143}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11558","reported_seq":16108,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:27.654179+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:27.654179+0000","last_peered":"2024-09-16T04:51:27.654179+0000","last_clean":"2024-09-16T04:51:27.654179+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:27.654179+0000","last_undegraded":"2024-09-16T04:51:27.654179+0000","last_fullsized":"2024-09-16T04:51:27.654179+0000","mapping_epoch":36,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:31.464846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9106700,"num_objects":1112,"num_object_clones":0,"num_object_copies":4448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11556,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11632","reported_seq":11770,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.114712+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:30.114712+0000","last_peered":"2024-09-16T04:51:30.114712+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:30.114712+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9082124,"num_objects":1109,"num_object_clones":0,"num_object_copies":4436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11630,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11303","reported_seq":11417,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.285610+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:30.285610+0000","last_peered":"2024-09-16T04:51:30.285610+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:30.285610+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8265728,"num_objects":1009,"num_object_clones":0,"num_object_copies":4036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11303,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11587","reported_seq":11689,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.174507+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:30.174507+0000","last_peered":"2024-09-16T04:51:30.174507+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:30.174507+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8295692,"num_objects":1013,"num_object_clones":0,"num_object_copies":4052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11585,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1200}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11638","reported_seq":16205,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:27.550271+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:27.550271+0000","last_peered":"2024-09-16T04:51:27.550271+0000","last_clean":"2024-09-16T04:51:27.550271+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:27.550271+0000","last_undegraded":"2024-09-16T04:51:27.550271+0000","last_fullsized":"2024-09-16T04:51:27.550271+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:31.464820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9393420,"num_objects":1147,"num_object_clones":0,"num_object_copies":4588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11636,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11690","reported_seq":16453,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:30.406130+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:30.406130+0000","last_peered":"2024-09-16T04:51:30.406130+0000","last_clean":"2024-09-16T04:51:30.406130+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:30.406130+0000","last_undegraded":"2024-09-16T04:51:30.406130+0000","last_fullsized":"2024-09-16T04:51:30.406130+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:31.632999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8727064,"num_objects":1066,"num_object_clones":0,"num_object_copies":4264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11687,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11661","reported_seq":11773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:28.647084+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:28.647084+0000","last_peered":"2024-09-16T04:51:28.647084+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:28.647084+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8983820,"num_objects":1097,"num_object_clones":0,"num_object_copies":4388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11659,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1092}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11460","reported_seq":11598,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.439513+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:30.439513+0000","last_peered":"2024-09-16T04:51:30.439513+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:30.439513+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8441746,"num_objects":1031,"num_object_clones":0,"num_object_copies":4124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11457,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11337","reported_seq":11433,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.454115+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:30.454115+0000","last_peered":"2024-09-16T04:51:30.454115+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:30.454115+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8141446,"num_objects":994,"num_object_clones":0,"num_object_copies":3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11336,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1111}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11460","reported_seq":11551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:28.620813+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:28.620813+0000","last_peered":"2024-09-16T04:51:28.620813+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:28.620813+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8788614,"num_objects":1073,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11459,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1148}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11712","reported_seq":11848,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.563826+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:30.563826+0000","last_peered":"2024-09-16T04:51:30.563826+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:30.563826+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8762636,"num_objects":1070,"num_object_clones":0,"num_object_copies":4280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11710,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1128}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11366","reported_seq":11965,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:28.791617+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:28.791617+0000","last_peered":"2024-09-16T04:51:28.791617+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:28.791617+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8869132,"num_objects":1083,"num_object_clones":0,"num_object_copies":4332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11364,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-797}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11284","reported_seq":11383,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:29.928659+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:29.928659+0000","last_peered":"2024-09-16T04:51:29.928659+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:29.928659+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8377612,"num_objects":1023,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11282,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1205}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":139224768,"num_objects":17001,"num_object_clones":0,"num_object_copies":68004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17001,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184486,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7341,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":279953408,"data_stored":279953408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160913,"ondisk_log_size":160913,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717808,"kb_used_data":68320,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93654032,"statfs":{"total":96636764160,"available":95901728768,"internally_reserved":0,"allocated":69959680,"data_stored":69761171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":665054927},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744420,"kb_used_data":69076,"kb_used_omap":1,"kb_used_meta":675326,"kb_avail":93627420,"statfs":{"total":96636764160,"available":95874478080,"internally_reserved":0,"allocated":70733824,"data_stored":70535315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":691534277},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,3,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589584,"kb_used_data":70592,"kb_used_omap":4,"kb_used_meta":518971,"kb_avail":93782256,"statfs":{"total":96636764160,"available":96033030144,"internally_reserved":0,"allocated":72286208,"data_stored":72087699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":531427023},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,2,3,6,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717656,"kb_used_data":68228,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93654184,"statfs":{"total":96636764160,"available":95901884416,"internally_reserved":0,"allocated":69865472,"data_stored":69666963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":664989390},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,4,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69173248,"data_stored":69173248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":71622656,"data_stored":71622656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":70299648,"data_stored":70299648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":68857856,"data_stored":68857856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:31.660 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:32.248 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:32.249 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:32.279 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-09-16T04:51:30.769432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":139224768,"num_objects":17001,"num_object_clones":0,"num_object_copies":68004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17001,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184486,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7341,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160913,"ondisk_log_size":160913,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769468,"kb_used_data":276216,"kb_used_omap":14,"kb_used_meta":2493169,"kb_avail":374717892,"statfs":{"total":386547056640,"available":383711121408,"internally_reserved":0,"allocated":282845184,"data_stored":282051148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2553005617},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,1,0,8,13,15,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20430848,"num_objects":-2494,"num_object_clones":0,"num_object_copies":-9976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2494,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004719"},"pg_stats":[{"pgid":"1.f","version":"37'11593","reported_seq":11714,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.495165+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:30.495165+0000","last_peered":"2024-09-16T04:51:30.495165+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:30.495165+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8691712,"num_objects":1061,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11637","reported_seq":11729,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.018068+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:30.018068+0000","last_peered":"2024-09-16T04:51:30.018068+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:30.018068+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8492300,"num_objects":1037,"num_object_clones":0,"num_object_copies":4148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1166}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11595","reported_seq":11698,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:28.687183+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:28.687183+0000","last_peered":"2024-09-16T04:51:28.687183+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:28.687183+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8805022,"num_objects":1076,"num_object_clones":0,"num_object_copies":4304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1076,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11594,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1143}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11558","reported_seq":16108,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:27.654179+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:27.654179+0000","last_peered":"2024-09-16T04:51:27.654179+0000","last_clean":"2024-09-16T04:51:27.654179+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:27.654179+0000","last_undegraded":"2024-09-16T04:51:27.654179+0000","last_fullsized":"2024-09-16T04:51:27.654179+0000","mapping_epoch":36,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:31.464846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9106700,"num_objects":1112,"num_object_clones":0,"num_object_copies":4448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11556,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11632","reported_seq":11770,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.114712+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:30.114712+0000","last_peered":"2024-09-16T04:51:30.114712+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:30.114712+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9082124,"num_objects":1109,"num_object_clones":0,"num_object_copies":4436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11630,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11303","reported_seq":11417,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.285610+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:30.285610+0000","last_peered":"2024-09-16T04:51:30.285610+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:30.285610+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8265728,"num_objects":1009,"num_object_clones":0,"num_object_copies":4036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11303,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11587","reported_seq":11689,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.174507+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:30.174507+0000","last_peered":"2024-09-16T04:51:30.174507+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:30.174507+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8295692,"num_objects":1013,"num_object_clones":0,"num_object_copies":4052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11585,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1200}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11638","reported_seq":16205,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:27.550271+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:27.550271+0000","last_peered":"2024-09-16T04:51:27.550271+0000","last_clean":"2024-09-16T04:51:27.550271+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:27.550271+0000","last_undegraded":"2024-09-16T04:51:27.550271+0000","last_fullsized":"2024-09-16T04:51:27.550271+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:31.464820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9393420,"num_objects":1147,"num_object_clones":0,"num_object_copies":4588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11636,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11690","reported_seq":16453,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:30.406130+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:30.406130+0000","last_peered":"2024-09-16T04:51:30.406130+0000","last_clean":"2024-09-16T04:51:30.406130+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:30.406130+0000","last_undegraded":"2024-09-16T04:51:30.406130+0000","last_fullsized":"2024-09-16T04:51:30.406130+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:31.632999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8727064,"num_objects":1066,"num_object_clones":0,"num_object_copies":4264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11687,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11661","reported_seq":11773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:28.647084+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:28.647084+0000","last_peered":"2024-09-16T04:51:28.647084+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:28.647084+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8983820,"num_objects":1097,"num_object_clones":0,"num_object_copies":4388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11659,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1092}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11460","reported_seq":11598,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.439513+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:30.439513+0000","last_peered":"2024-09-16T04:51:30.439513+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:30.439513+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8441746,"num_objects":1031,"num_object_clones":0,"num_object_copies":4124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11457,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11337","reported_seq":11433,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.454115+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:30.454115+0000","last_peered":"2024-09-16T04:51:30.454115+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:30.454115+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8141446,"num_objects":994,"num_object_clones":0,"num_object_copies":3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11336,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1111}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11460","reported_seq":11551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:28.620813+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:28.620813+0000","last_peered":"2024-09-16T04:51:28.620813+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:28.620813+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8788614,"num_objects":1073,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11459,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1148}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11712","reported_seq":11848,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.563826+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:30.563826+0000","last_peered":"2024-09-16T04:51:30.563826+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:30.563826+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8762636,"num_objects":1070,"num_object_clones":0,"num_object_copies":4280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11710,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1128}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11366","reported_seq":11965,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:28.791617+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:28.791617+0000","last_peered":"2024-09-16T04:51:28.791617+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:28.791617+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8869132,"num_objects":1083,"num_object_clones":0,"num_object_copies":4332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11364,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-797}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11284","reported_seq":11383,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:29.928659+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:29.928659+0000","last_peered":"2024-09-16T04:51:29.928659+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:29.928659+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8377612,"num_objects":1023,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11282,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1205}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":139224768,"num_objects":17001,"num_object_clones":0,"num_object_copies":68004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17001,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184486,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7341,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":279953408,"data_stored":279953408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160913,"ondisk_log_size":160913,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717808,"kb_used_data":68320,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93654032,"statfs":{"total":96636764160,"available":95901728768,"internally_reserved":0,"allocated":69959680,"data_stored":69761171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":665054927},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744420,"kb_used_data":69076,"kb_used_omap":1,"kb_used_meta":675326,"kb_avail":93627420,"statfs":{"total":96636764160,"available":95874478080,"internally_reserved":0,"allocated":70733824,"data_stored":70535315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":691534277},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,3,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589584,"kb_used_data":70592,"kb_used_omap":4,"kb_used_meta":518971,"kb_avail":93782256,"statfs":{"total":96636764160,"available":96033030144,"internally_reserved":0,"allocated":72286208,"data_stored":72087699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":531427023},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,2,3,6,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717656,"kb_used_data":68228,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93654184,"statfs":{"total":96636764160,"available":95901884416,"internally_reserved":0,"allocated":69865472,"data_stored":69666963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":664989390},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,4,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69173248,"data_stored":69173248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":71622656,"data_stored":71622656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":70299648,"data_stored":70299648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":68857856,"data_stored":68857856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:32.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:51:32.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11593", 'reported_seq': 11714, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:30.495165+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:51:30.495165+0000', 'last_peered': '2024-09-16T04:51:30.495165+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:51:30.495165+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8691712, 'num_objects': 1061, 'num_object_clones': 0, 'num_object_copies': 4244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11593, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1125}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:32.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:51:32.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11637", 'reported_seq': 11729, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:30.018068+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:51:30.018068+0000', 'last_peered': '2024-09-16T04:51:30.018068+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:51:30.018068+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8492300, 'num_objects': 1037, 'num_object_clones': 0, 'num_object_copies': 4148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11635, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1166}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:32.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:51:32.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11595", 'reported_seq': 11698, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:28.687183+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:51:28.687183+0000', 'last_peered': '2024-09-16T04:51:28.687183+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:51:28.687183+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8805022, 'num_objects': 1076, 'num_object_clones': 0, 'num_object_copies': 4304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1076, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11594, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1143}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:32.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:51:32.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11632", 'reported_seq': 11770, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:30.114712+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:51:30.114712+0000', 'last_peered': '2024-09-16T04:51:30.114712+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:51:30.114712+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9082124, 'num_objects': 1109, 'num_object_clones': 0, 'num_object_copies': 4436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1109, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11630, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1139}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:32.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:51:32.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11303", 'reported_seq': 11417, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:30.285610+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:51:30.285610+0000', 'last_peered': '2024-09-16T04:51:30.285610+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:51:30.285610+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8265728, 'num_objects': 1009, 'num_object_clones': 0, 'num_object_copies': 4036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1009, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11303, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1169}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:32.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:51:32.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11587", 'reported_seq': 11689, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:30.174507+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:51:30.174507+0000', 'last_peered': '2024-09-16T04:51:30.174507+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:51:30.174507+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8295692, 'num_objects': 1013, 'num_object_clones': 0, 'num_object_copies': 4052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11585, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1200}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:32.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:51:32.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11661", 'reported_seq': 11773, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:28.647084+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:51:28.647084+0000', 'last_peered': '2024-09-16T04:51:28.647084+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:51:28.647084+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8983820, 'num_objects': 1097, 'num_object_clones': 0, 'num_object_copies': 4388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1097, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11659, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1092}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:32.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:51:32.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11460", 'reported_seq': 11598, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:30.439513+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:51:30.439513+0000', 'last_peered': '2024-09-16T04:51:30.439513+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:51:30.439513+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8441746, 'num_objects': 1031, 'num_object_clones': 0, 'num_object_copies': 4124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11457, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1124}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:32.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:51:32.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11337", 'reported_seq': 11433, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:30.454115+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:51:30.454115+0000', 'last_peered': '2024-09-16T04:51:30.454115+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:51:30.454115+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8141446, 'num_objects': 994, 'num_object_clones': 0, 'num_object_copies': 3976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11336, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1111}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:32.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:51:32.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11460", 'reported_seq': 11551, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:28.620813+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:51:28.620813+0000', 'last_peered': '2024-09-16T04:51:28.620813+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:51:28.620813+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8788614, 'num_objects': 1073, 'num_object_clones': 0, 'num_object_copies': 4292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1073, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11459, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1148}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:32.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:51:32.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11712", 'reported_seq': 11848, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:30.563826+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:51:30.563826+0000', 'last_peered': '2024-09-16T04:51:30.563826+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:51:30.563826+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8762636, 'num_objects': 1070, 'num_object_clones': 0, 'num_object_copies': 4280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1070, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11710, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1128}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:32.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:51:32.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11366", 'reported_seq': 11965, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:28.791617+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:51:28.791617+0000', 'last_peered': '2024-09-16T04:51:28.791617+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:51:28.791617+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8869132, 'num_objects': 1083, 'num_object_clones': 0, 'num_object_copies': 4332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1880, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11364, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 256, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -797}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:32.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:51:32.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11284", 'reported_seq': 11383, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:29.928659+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:51:29.928659+0000', 'last_peered': '2024-09-16T04:51:29.928659+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:51:29.928659+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8377612, 'num_objects': 1023, 'num_object_clones': 0, 'num_object_copies': 4092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1023, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11282, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1205}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:32.283 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:51:32.980 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:32.980 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28241/68004 objects degraded (41.528%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":705,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":17001,"data_bytes":139224768,"bytes_used":2835935232,"bytes_avail":383711121408,"bytes_total":386547056640,"degraded_objects":28241,"degraded_total":68004,"degraded_ratio":0.41528439503558606,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":207},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:51:33.010 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:51:33.010 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:33.583 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:33.583 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:33.605 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-09-16T04:51:32.770259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":139224768,"num_objects":17001,"num_object_clones":0,"num_object_copies":68004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17001,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184486,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7341,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160913,"ondisk_log_size":160913,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769468,"kb_used_data":276216,"kb_used_omap":14,"kb_used_meta":2493169,"kb_avail":374717892,"statfs":{"total":386547056640,"available":383711121408,"internally_reserved":0,"allocated":282845184,"data_stored":282051148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2553005617},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,1,0,8,13,15,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15425536,"num_objects":-1883,"num_object_clones":0,"num_object_copies":-7532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1883,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004691"},"pg_stats":[{"pgid":"1.f","version":"37'11593","reported_seq":11714,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.495165+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:30.495165+0000","last_peered":"2024-09-16T04:51:30.495165+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:30.495165+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8691712,"num_objects":1061,"num_object_clones":0,"num_object_copies":4244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11637","reported_seq":11729,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.018068+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:30.018068+0000","last_peered":"2024-09-16T04:51:30.018068+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:30.018068+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8492300,"num_objects":1037,"num_object_clones":0,"num_object_copies":4148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1166}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11595","reported_seq":11698,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:28.687183+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:28.687183+0000","last_peered":"2024-09-16T04:51:28.687183+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:28.687183+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8805022,"num_objects":1076,"num_object_clones":0,"num_object_copies":4304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1076,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11594,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1143}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11558","reported_seq":16108,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:27.654179+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:27.654179+0000","last_peered":"2024-09-16T04:51:27.654179+0000","last_clean":"2024-09-16T04:51:27.654179+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:27.654179+0000","last_undegraded":"2024-09-16T04:51:27.654179+0000","last_fullsized":"2024-09-16T04:51:27.654179+0000","mapping_epoch":36,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:31.464846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9106700,"num_objects":1112,"num_object_clones":0,"num_object_copies":4448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11556,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11632","reported_seq":11770,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.114712+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:30.114712+0000","last_peered":"2024-09-16T04:51:30.114712+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:30.114712+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9082124,"num_objects":1109,"num_object_clones":0,"num_object_copies":4436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11630,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11303","reported_seq":11417,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.285610+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:30.285610+0000","last_peered":"2024-09-16T04:51:30.285610+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:30.285610+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8265728,"num_objects":1009,"num_object_clones":0,"num_object_copies":4036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11303,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11587","reported_seq":11689,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.174507+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:30.174507+0000","last_peered":"2024-09-16T04:51:30.174507+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:30.174507+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8295692,"num_objects":1013,"num_object_clones":0,"num_object_copies":4052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11585,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1200}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11638","reported_seq":16205,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:27.550271+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:27.550271+0000","last_peered":"2024-09-16T04:51:27.550271+0000","last_clean":"2024-09-16T04:51:27.550271+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:27.550271+0000","last_undegraded":"2024-09-16T04:51:27.550271+0000","last_fullsized":"2024-09-16T04:51:27.550271+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:31.464820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9393420,"num_objects":1147,"num_object_clones":0,"num_object_copies":4588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11636,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11690","reported_seq":16453,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:30.406130+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:30.406130+0000","last_peered":"2024-09-16T04:51:30.406130+0000","last_clean":"2024-09-16T04:51:30.406130+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:30.406130+0000","last_undegraded":"2024-09-16T04:51:30.406130+0000","last_fullsized":"2024-09-16T04:51:30.406130+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:31.632999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8727064,"num_objects":1066,"num_object_clones":0,"num_object_copies":4264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11687,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11661","reported_seq":11773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:28.647084+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:28.647084+0000","last_peered":"2024-09-16T04:51:28.647084+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:28.647084+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8983820,"num_objects":1097,"num_object_clones":0,"num_object_copies":4388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11659,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1092}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11460","reported_seq":11598,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.439513+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:30.439513+0000","last_peered":"2024-09-16T04:51:30.439513+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:30.439513+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8441746,"num_objects":1031,"num_object_clones":0,"num_object_copies":4124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11457,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11337","reported_seq":11433,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.454115+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:30.454115+0000","last_peered":"2024-09-16T04:51:30.454115+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:30.454115+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8141446,"num_objects":994,"num_object_clones":0,"num_object_copies":3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11336,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1111}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11460","reported_seq":11551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:28.620813+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:28.620813+0000","last_peered":"2024-09-16T04:51:28.620813+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:28.620813+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8788614,"num_objects":1073,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11459,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1148}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11712","reported_seq":11848,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:30.563826+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:30.563826+0000","last_peered":"2024-09-16T04:51:30.563826+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:30.563826+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8762636,"num_objects":1070,"num_object_clones":0,"num_object_copies":4280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11710,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1128}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11366","reported_seq":11965,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:28.791617+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:28.791617+0000","last_peered":"2024-09-16T04:51:28.791617+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:28.791617+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8869132,"num_objects":1083,"num_object_clones":0,"num_object_copies":4332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11364,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-797}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11284","reported_seq":11383,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:29.928659+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:29.928659+0000","last_peered":"2024-09-16T04:51:29.928659+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:29.928659+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8377612,"num_objects":1023,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11282,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1205}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":139224768,"num_objects":17001,"num_object_clones":0,"num_object_copies":68004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17001,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184486,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7341,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":279953408,"data_stored":279953408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160913,"ondisk_log_size":160913,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717808,"kb_used_data":68320,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93654032,"statfs":{"total":96636764160,"available":95901728768,"internally_reserved":0,"allocated":69959680,"data_stored":69761171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":665054927},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744420,"kb_used_data":69076,"kb_used_omap":1,"kb_used_meta":675326,"kb_avail":93627420,"statfs":{"total":96636764160,"available":95874478080,"internally_reserved":0,"allocated":70733824,"data_stored":70535315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":691534277},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,3,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589584,"kb_used_data":70592,"kb_used_omap":4,"kb_used_meta":518971,"kb_avail":93782256,"statfs":{"total":96636764160,"available":96033030144,"internally_reserved":0,"allocated":72286208,"data_stored":72087699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":531427023},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,2,3,6,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717656,"kb_used_data":68228,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93654184,"statfs":{"total":96636764160,"available":95901884416,"internally_reserved":0,"allocated":69865472,"data_stored":69666963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":664989390},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,4,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69173248,"data_stored":69173248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":71622656,"data_stored":71622656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":70299648,"data_stored":70299648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":68857856,"data_stored":68857856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:36.608 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:37.167 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:37.168 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:37.189 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-09-16T04:51:36.771372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131221184,"num_objects":16024,"num_object_clones":0,"num_object_copies":64096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16024,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185463,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7385,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745756,"kb_used_data":260792,"kb_used_omap":14,"kb_used_meta":2484849,"kb_avail":374741604,"statfs":{"total":386547056640,"available":383735402496,"internally_reserved":0,"allocated":267051008,"data_stored":266256972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2544485937},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,6,10,13,10,10,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20406272,"num_objects":-2491,"num_object_clones":0,"num_object_copies":-9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2491,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004154"},"pg_stats":[{"pgid":"1.f","version":"37'11650","reported_seq":11771,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.444490+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:35.444490+0000","last_peered":"2024-09-16T04:51:35.444490+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:35.444490+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8224768,"num_objects":1004,"num_object_clones":0,"num_object_copies":4016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11650,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1182}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11708","reported_seq":11799,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.210258+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:35.210258+0000","last_peered":"2024-09-16T04:51:35.210258+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:35.210258+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7910668,"num_objects":966,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11706,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1237}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11652","reported_seq":11754,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:33.806495+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:33.806495+0000","last_peered":"2024-09-16T04:51:33.806495+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:33.806495+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8338078,"num_objects":1019,"num_object_clones":0,"num_object_copies":4076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1019,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11651,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11624","reported_seq":16174,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:32.898656+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:32.898656+0000","last_peered":"2024-09-16T04:51:32.898656+0000","last_clean":"2024-09-16T04:51:32.898656+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:32.898656+0000","last_undegraded":"2024-09-16T04:51:32.898656+0000","last_fullsized":"2024-09-16T04:51:32.898656+0000","mapping_epoch":36,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:36.491406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8566028,"num_objects":1046,"num_object_clones":0,"num_object_copies":4184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11622,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11700","reported_seq":11838,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.596195+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:35.596195+0000","last_peered":"2024-09-16T04:51:35.596195+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:35.596195+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8525068,"num_objects":1041,"num_object_clones":0,"num_object_copies":4164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11698,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11348","reported_seq":11463,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.516984+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:35.516984+0000","last_peered":"2024-09-16T04:51:35.516984+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:35.516984+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7897088,"num_objects":964,"num_object_clones":0,"num_object_copies":3856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11348,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11648","reported_seq":11750,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.101202+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:35.101202+0000","last_peered":"2024-09-16T04:51:35.101202+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:35.101202+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7795980,"num_objects":952,"num_object_clones":0,"num_object_copies":3808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11646,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1261}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11709","reported_seq":16276,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:32.990149+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:32.990149+0000","last_peered":"2024-09-16T04:51:32.990149+0000","last_clean":"2024-09-16T04:51:32.990149+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:32.990149+0000","last_undegraded":"2024-09-16T04:51:32.990149+0000","last_fullsized":"2024-09-16T04:51:32.990149+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:36.491368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8811788,"num_objects":1076,"num_object_clones":0,"num_object_copies":4304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11707,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11736","reported_seq":16499,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:35.478575+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:35.478575+0000","last_peered":"2024-09-16T04:51:35.478575+0000","last_clean":"2024-09-16T04:51:35.478575+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:35.478575+0000","last_undegraded":"2024-09-16T04:51:35.478575+0000","last_fullsized":"2024-09-16T04:51:35.478575+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:36.666814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8350232,"num_objects":1020,"num_object_clones":0,"num_object_copies":4080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11722","reported_seq":11834,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:33.873802+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:33.873802+0000","last_peered":"2024-09-16T04:51:33.873802+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:33.873802+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8484108,"num_objects":1036,"num_object_clones":0,"num_object_copies":4144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1153}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11513","reported_seq":11650,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.520122+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:35.520122+0000","last_peered":"2024-09-16T04:51:35.520122+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:35.520122+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8007570,"num_objects":978,"num_object_clones":0,"num_object_copies":3912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11510,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11391","reported_seq":11488,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.411025+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:35.411025+0000","last_peered":"2024-09-16T04:51:35.411025+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:35.411025+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7699078,"num_objects":940,"num_object_clones":0,"num_object_copies":3760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11390,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1165}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11529","reported_seq":11622,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:33.754898+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:33.754898+0000","last_peered":"2024-09-16T04:51:33.754898+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:33.754898+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8223366,"num_objects":1004,"num_object_clones":0,"num_object_copies":4016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11528,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1217}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11772","reported_seq":11909,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.597489+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:35.597489+0000","last_peered":"2024-09-16T04:51:35.597489+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:35.597489+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8271116,"num_objects":1010,"num_object_clones":0,"num_object_copies":4040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11770,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1188}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11438","reported_seq":12122,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:33.859668+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:33.859668+0000","last_peered":"2024-09-16T04:51:33.859668+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:33.859668+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8279308,"num_objects":1011,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11436,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":300,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11350","reported_seq":11449,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.237837+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:35.237837+0000","last_peered":"2024-09-16T04:51:35.237837+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:35.237837+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7836940,"num_objects":957,"num_object_clones":0,"num_object_copies":3828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11348,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1271}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":131221184,"num_objects":16024,"num_object_clones":0,"num_object_copies":64096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16024,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185463,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7385,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":264019968,"data_stored":264019968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713964,"kb_used_data":64476,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93657876,"statfs":{"total":96636764160,"available":95905665024,"internally_reserved":0,"allocated":66023424,"data_stored":65824915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":665054927},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,3,7,1,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723952,"kb_used_data":65236,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93647888,"statfs":{"total":96636764160,"available":95895437312,"internally_reserved":0,"allocated":66801664,"data_stored":66603155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589804,"kb_used_data":66628,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93782036,"statfs":{"total":96636764160,"available":96032804864,"internally_reserved":0,"allocated":68227072,"data_stored":68028563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":535686863},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,6,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718036,"kb_used_data":64452,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93653804,"statfs":{"total":96636764160,"available":95901495296,"internally_reserved":0,"allocated":65998848,"data_stored":65800339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":669249230},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,2,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65208320,"data_stored":65208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67588096,"data_stored":67588096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":66334720,"data_stored":66334720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":64888832,"data_stored":64888832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:37.190 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:37.821 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:37.822 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:37.847 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-09-16T04:51:36.771372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131221184,"num_objects":16024,"num_object_clones":0,"num_object_copies":64096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16024,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185463,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7385,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745756,"kb_used_data":260792,"kb_used_omap":14,"kb_used_meta":2484849,"kb_avail":374741604,"statfs":{"total":386547056640,"available":383735402496,"internally_reserved":0,"allocated":267051008,"data_stored":266256972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2544485937},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,6,10,13,10,10,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20406272,"num_objects":-2491,"num_object_clones":0,"num_object_copies":-9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2491,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004154"},"pg_stats":[{"pgid":"1.f","version":"37'11650","reported_seq":11771,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.444490+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:35.444490+0000","last_peered":"2024-09-16T04:51:35.444490+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:35.444490+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8224768,"num_objects":1004,"num_object_clones":0,"num_object_copies":4016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11650,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1182}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11708","reported_seq":11799,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.210258+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:35.210258+0000","last_peered":"2024-09-16T04:51:35.210258+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:35.210258+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7910668,"num_objects":966,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11706,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1237}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11652","reported_seq":11754,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:33.806495+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:33.806495+0000","last_peered":"2024-09-16T04:51:33.806495+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:33.806495+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8338078,"num_objects":1019,"num_object_clones":0,"num_object_copies":4076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1019,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11651,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11624","reported_seq":16174,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:32.898656+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:32.898656+0000","last_peered":"2024-09-16T04:51:32.898656+0000","last_clean":"2024-09-16T04:51:32.898656+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:32.898656+0000","last_undegraded":"2024-09-16T04:51:32.898656+0000","last_fullsized":"2024-09-16T04:51:32.898656+0000","mapping_epoch":36,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:36.491406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8566028,"num_objects":1046,"num_object_clones":0,"num_object_copies":4184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11622,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11700","reported_seq":11838,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.596195+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:35.596195+0000","last_peered":"2024-09-16T04:51:35.596195+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:35.596195+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8525068,"num_objects":1041,"num_object_clones":0,"num_object_copies":4164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11698,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11348","reported_seq":11463,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.516984+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:35.516984+0000","last_peered":"2024-09-16T04:51:35.516984+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:35.516984+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7897088,"num_objects":964,"num_object_clones":0,"num_object_copies":3856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11348,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11648","reported_seq":11750,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.101202+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:35.101202+0000","last_peered":"2024-09-16T04:51:35.101202+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:35.101202+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7795980,"num_objects":952,"num_object_clones":0,"num_object_copies":3808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11646,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1261}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11709","reported_seq":16276,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:32.990149+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:32.990149+0000","last_peered":"2024-09-16T04:51:32.990149+0000","last_clean":"2024-09-16T04:51:32.990149+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:32.990149+0000","last_undegraded":"2024-09-16T04:51:32.990149+0000","last_fullsized":"2024-09-16T04:51:32.990149+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:36.491368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8811788,"num_objects":1076,"num_object_clones":0,"num_object_copies":4304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11707,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11736","reported_seq":16499,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:35.478575+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:35.478575+0000","last_peered":"2024-09-16T04:51:35.478575+0000","last_clean":"2024-09-16T04:51:35.478575+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:35.478575+0000","last_undegraded":"2024-09-16T04:51:35.478575+0000","last_fullsized":"2024-09-16T04:51:35.478575+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:36.666814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8350232,"num_objects":1020,"num_object_clones":0,"num_object_copies":4080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11722","reported_seq":11834,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:33.873802+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:33.873802+0000","last_peered":"2024-09-16T04:51:33.873802+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:33.873802+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8484108,"num_objects":1036,"num_object_clones":0,"num_object_copies":4144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1153}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11513","reported_seq":11650,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.520122+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:35.520122+0000","last_peered":"2024-09-16T04:51:35.520122+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:35.520122+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8007570,"num_objects":978,"num_object_clones":0,"num_object_copies":3912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11510,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11391","reported_seq":11488,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.411025+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:35.411025+0000","last_peered":"2024-09-16T04:51:35.411025+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:35.411025+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7699078,"num_objects":940,"num_object_clones":0,"num_object_copies":3760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11390,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1165}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11529","reported_seq":11622,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:33.754898+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:33.754898+0000","last_peered":"2024-09-16T04:51:33.754898+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:33.754898+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8223366,"num_objects":1004,"num_object_clones":0,"num_object_copies":4016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11528,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1217}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11772","reported_seq":11909,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.597489+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:35.597489+0000","last_peered":"2024-09-16T04:51:35.597489+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:35.597489+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8271116,"num_objects":1010,"num_object_clones":0,"num_object_copies":4040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11770,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1188}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11438","reported_seq":12122,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:33.859668+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:33.859668+0000","last_peered":"2024-09-16T04:51:33.859668+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:33.859668+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8279308,"num_objects":1011,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11436,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":300,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11350","reported_seq":11449,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.237837+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:35.237837+0000","last_peered":"2024-09-16T04:51:35.237837+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:35.237837+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7836940,"num_objects":957,"num_object_clones":0,"num_object_copies":3828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11348,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1271}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":131221184,"num_objects":16024,"num_object_clones":0,"num_object_copies":64096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16024,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185463,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7385,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":264019968,"data_stored":264019968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713964,"kb_used_data":64476,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93657876,"statfs":{"total":96636764160,"available":95905665024,"internally_reserved":0,"allocated":66023424,"data_stored":65824915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":665054927},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,3,7,1,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723952,"kb_used_data":65236,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93647888,"statfs":{"total":96636764160,"available":95895437312,"internally_reserved":0,"allocated":66801664,"data_stored":66603155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589804,"kb_used_data":66628,"kb_used_omap":4,"kb_used_meta":523131,"kb_avail":93782036,"statfs":{"total":96636764160,"available":96032804864,"internally_reserved":0,"allocated":68227072,"data_stored":68028563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":535686863},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,6,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718036,"kb_used_data":64452,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93653804,"statfs":{"total":96636764160,"available":95901495296,"internally_reserved":0,"allocated":65998848,"data_stored":65800339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":669249230},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,2,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65208320,"data_stored":65208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67588096,"data_stored":67588096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":66334720,"data_stored":66334720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":64888832,"data_stored":64888832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:37.848 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:51:37.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11650", 'reported_seq': 11771, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:35.444490+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:51:35.444490+0000', 'last_peered': '2024-09-16T04:51:35.444490+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:51:35.444490+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8224768, 'num_objects': 1004, 'num_object_clones': 0, 'num_object_copies': 4016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1004, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11650, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1182}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:37.848 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:51:37.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11708", 'reported_seq': 11799, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:35.210258+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:51:35.210258+0000', 'last_peered': '2024-09-16T04:51:35.210258+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:51:35.210258+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7910668, 'num_objects': 966, 'num_object_clones': 0, 'num_object_copies': 3864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 966, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11706, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:37.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:51:37.849 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11652", 'reported_seq': 11754, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:33.806495+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:51:33.806495+0000', 'last_peered': '2024-09-16T04:51:33.806495+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:51:33.806495+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8338078, 'num_objects': 1019, 'num_object_clones': 0, 'num_object_copies': 4076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1019, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11651, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1200}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:37.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:51:37.849 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11700", 'reported_seq': 11838, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:35.596195+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:51:35.596195+0000', 'last_peered': '2024-09-16T04:51:35.596195+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:51:35.596195+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8525068, 'num_objects': 1041, 'num_object_clones': 0, 'num_object_copies': 4164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11698, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1207}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:37.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:51:37.849 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11348", 'reported_seq': 11463, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:35.516984+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:51:35.516984+0000', 'last_peered': '2024-09-16T04:51:35.516984+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:51:35.516984+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7897088, 'num_objects': 964, 'num_object_clones': 0, 'num_object_copies': 3856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 964, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11348, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1214}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:37.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:51:37.849 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11648", 'reported_seq': 11750, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:35.101202+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:51:35.101202+0000', 'last_peered': '2024-09-16T04:51:35.101202+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:51:35.101202+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7795980, 'num_objects': 952, 'num_object_clones': 0, 'num_object_copies': 3808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 952, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11646, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1261}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:37.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:51:37.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11722", 'reported_seq': 11834, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:33.873802+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:51:33.873802+0000', 'last_peered': '2024-09-16T04:51:33.873802+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:51:33.873802+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8484108, 'num_objects': 1036, 'num_object_clones': 0, 'num_object_copies': 4144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1036, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11720, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1153}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:37.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:51:37.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11513", 'reported_seq': 11650, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:35.520122+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:51:35.520122+0000', 'last_peered': '2024-09-16T04:51:35.520122+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:51:35.520122+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8007570, 'num_objects': 978, 'num_object_clones': 0, 'num_object_copies': 3912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11510, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1177}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:37.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:51:37.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11391", 'reported_seq': 11488, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:35.411025+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:51:35.411025+0000', 'last_peered': '2024-09-16T04:51:35.411025+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:51:35.411025+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7699078, 'num_objects': 940, 'num_object_clones': 0, 'num_object_copies': 3760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 940, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11390, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1165}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:37.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:51:37.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11529", 'reported_seq': 11622, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:33.754898+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:51:33.754898+0000', 'last_peered': '2024-09-16T04:51:33.754898+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:51:33.754898+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8223366, 'num_objects': 1004, 'num_object_clones': 0, 'num_object_copies': 4016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1004, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11528, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:37.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:51:37.850 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11772", 'reported_seq': 11909, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:35.597489+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:51:35.597489+0000', 'last_peered': '2024-09-16T04:51:35.597489+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:51:35.597489+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8271116, 'num_objects': 1010, 'num_object_clones': 0, 'num_object_copies': 4040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11770, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1188}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:37.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:51:37.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11438", 'reported_seq': 12122, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:33.859668+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:51:33.859668+0000', 'last_peered': '2024-09-16T04:51:33.859668+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:51:33.859668+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8279308, 'num_objects': 1011, 'num_object_clones': 0, 'num_object_copies': 4044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1836, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11436, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 300, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -825}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:37.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:51:37.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11350", 'reported_seq': 11449, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:35.237837+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:51:35.237837+0000', 'last_peered': '2024-09-16T04:51:35.237837+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:51:35.237837+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7836940, 'num_objects': 957, 'num_object_clones': 0, 'num_object_copies': 3828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11348, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1271}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:37.851 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:51:38.579 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:38.579 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28197/64096 objects degraded (43.992%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":710,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16024,"data_bytes":131221184,"bytes_used":2811654144,"bytes_avail":383735402496,"bytes_total":386547056640,"degraded_objects":28197,"degraded_total":64096,"degraded_ratio":0.43991824762855719,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":207},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:51:38.602 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:51:38.602 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:39.201 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:39.202 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:39.224 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-16T04:51:38.771913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":130270912,"num_objects":15908,"num_object_clones":0,"num_object_copies":63632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15908,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185579,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7385,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160806,"ondisk_log_size":160806,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2764076,"kb_used_data":256912,"kb_used_omap":14,"kb_used_meta":2507057,"kb_avail":374723284,"statfs":{"total":386547056640,"available":383716642816,"internally_reserved":0,"allocated":263077888,"data_stored":262283852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2567226929},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,6,8,18,9,13,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16596992,"num_objects":-2026,"num_object_clones":0,"num_object_copies":-8104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003797"},"pg_stats":[{"pgid":"1.f","version":"37'11650","reported_seq":11771,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.444490+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:35.444490+0000","last_peered":"2024-09-16T04:51:35.444490+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:35.444490+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8224768,"num_objects":1004,"num_object_clones":0,"num_object_copies":4016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11650,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1182}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11708","reported_seq":11799,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.210258+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:35.210258+0000","last_peered":"2024-09-16T04:51:35.210258+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:35.210258+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7910668,"num_objects":966,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11706,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1237}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11652","reported_seq":11754,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:33.806495+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:33.806495+0000","last_peered":"2024-09-16T04:51:33.806495+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:33.806495+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8338078,"num_objects":1019,"num_object_clones":0,"num_object_copies":4076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1019,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11651,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11680","reported_seq":16230,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:37.950783+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:37.950783+0000","last_peered":"2024-09-16T04:51:37.950783+0000","last_clean":"2024-09-16T04:51:37.950783+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:37.950783+0000","last_undegraded":"2024-09-16T04:51:37.950783+0000","last_fullsized":"2024-09-16T04:51:37.950783+0000","mapping_epoch":36,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:41.549520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8107276,"num_objects":990,"num_object_clones":0,"num_object_copies":3960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11700","reported_seq":11838,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.596195+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:35.596195+0000","last_peered":"2024-09-16T04:51:35.596195+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:35.596195+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8525068,"num_objects":1041,"num_object_clones":0,"num_object_copies":4164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11698,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11348","reported_seq":11463,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.516984+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:35.516984+0000","last_peered":"2024-09-16T04:51:35.516984+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:35.516984+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7897088,"num_objects":964,"num_object_clones":0,"num_object_copies":3856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11348,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11648","reported_seq":11750,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.101202+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:35.101202+0000","last_peered":"2024-09-16T04:51:35.101202+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:35.101202+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7795980,"num_objects":952,"num_object_clones":0,"num_object_copies":3808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11646,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1261}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11769","reported_seq":16333,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:38.066705+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:38.066705+0000","last_peered":"2024-09-16T04:51:38.066705+0000","last_clean":"2024-09-16T04:51:38.066705+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:38.066705+0000","last_undegraded":"2024-09-16T04:51:38.066705+0000","last_fullsized":"2024-09-16T04:51:38.066705+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:41.549495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8320268,"num_objects":1016,"num_object_clones":0,"num_object_copies":4064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11767,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11736","reported_seq":16499,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:35.478575+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:35.478575+0000","last_peered":"2024-09-16T04:51:35.478575+0000","last_clean":"2024-09-16T04:51:35.478575+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:35.478575+0000","last_undegraded":"2024-09-16T04:51:35.478575+0000","last_fullsized":"2024-09-16T04:51:35.478575+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:36.666814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8350232,"num_objects":1020,"num_object_clones":0,"num_object_copies":4080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11722","reported_seq":11834,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:33.873802+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:33.873802+0000","last_peered":"2024-09-16T04:51:33.873802+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:33.873802+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8484108,"num_objects":1036,"num_object_clones":0,"num_object_copies":4144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1153}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11513","reported_seq":11650,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.520122+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:35.520122+0000","last_peered":"2024-09-16T04:51:35.520122+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:35.520122+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8007570,"num_objects":978,"num_object_clones":0,"num_object_copies":3912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11510,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11391","reported_seq":11488,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.411025+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:35.411025+0000","last_peered":"2024-09-16T04:51:35.411025+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:35.411025+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7699078,"num_objects":940,"num_object_clones":0,"num_object_copies":3760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11390,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1165}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11529","reported_seq":11622,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:33.754898+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:33.754898+0000","last_peered":"2024-09-16T04:51:33.754898+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:33.754898+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8223366,"num_objects":1004,"num_object_clones":0,"num_object_copies":4016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11528,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1217}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11772","reported_seq":11909,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.597489+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:35.597489+0000","last_peered":"2024-09-16T04:51:35.597489+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:35.597489+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8271116,"num_objects":1010,"num_object_clones":0,"num_object_copies":4040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11770,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1188}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11438","reported_seq":12122,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:33.859668+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:33.859668+0000","last_peered":"2024-09-16T04:51:33.859668+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:33.859668+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8279308,"num_objects":1011,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11436,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":300,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11350","reported_seq":11449,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:35.237837+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:35.237837+0000","last_peered":"2024-09-16T04:51:35.237837+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:35.237837+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7836940,"num_objects":957,"num_object_clones":0,"num_object_copies":3828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11348,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1271}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":130270912,"num_objects":15908,"num_object_clones":0,"num_object_copies":63632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15908,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185579,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7385,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":259993600,"data_stored":259993600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160806,"ondisk_log_size":160806,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713964,"kb_used_data":64476,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93657876,"statfs":{"total":96636764160,"available":95905665024,"internally_reserved":0,"allocated":66023424,"data_stored":65824915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":665054927},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,3,7,1,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723952,"kb_used_data":65236,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93647888,"statfs":{"total":96636764160,"available":95895437312,"internally_reserved":0,"allocated":66801664,"data_stored":66603155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608124,"kb_used_data":62748,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93763716,"statfs":{"total":96636764160,"available":96014045184,"internally_reserved":0,"allocated":64253952,"data_stored":64055443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558427855},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,7,5,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718036,"kb_used_data":64452,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93653804,"statfs":{"total":96636764160,"available":95901495296,"internally_reserved":0,"allocated":65998848,"data_stored":65800339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":669249230},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,2,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65208320,"data_stored":65208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63561728,"data_stored":63561728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":66334720,"data_stored":66334720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":64888832,"data_stored":64888832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:42.226 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:42.837 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:42.837 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:42.857 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-09-16T04:51:42.773364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":123373248,"num_objects":15066,"num_object_clones":0,"num_object_copies":60264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186421,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7440,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161048,"ondisk_log_size":161048,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2748148,"kb_used_data":245156,"kb_used_omap":14,"kb_used_meta":2502897,"kb_avail":374739212,"statfs":{"total":386547056640,"available":383732953088,"internally_reserved":0,"allocated":251039744,"data_stored":250245708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2562967089},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,9,6,15,9,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15851520,"num_objects":-1935,"num_object_clones":0,"num_object_copies":-7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1935,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003931"},"pg_stats":[{"pgid":"1.f","version":"37'11725","reported_seq":11846,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.611181+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:40.611181+0000","last_peered":"2024-09-16T04:51:40.611181+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:40.611181+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7610368,"num_objects":929,"num_object_clones":0,"num_object_copies":3716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11768","reported_seq":11860,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.276821+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:40.276821+0000","last_peered":"2024-09-16T04:51:40.276821+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:40.276821+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7419148,"num_objects":906,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11766,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1297}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11717","reported_seq":11821,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:38.613866+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:38.613866+0000","last_peered":"2024-09-16T04:51:38.613866+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:38.613866+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7805598,"num_objects":954,"num_object_clones":0,"num_object_copies":3816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11716,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1265}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11680","reported_seq":16230,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:37.950783+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:37.950783+0000","last_peered":"2024-09-16T04:51:37.950783+0000","last_clean":"2024-09-16T04:51:37.950783+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:37.950783+0000","last_undegraded":"2024-09-16T04:51:37.950783+0000","last_fullsized":"2024-09-16T04:51:37.950783+0000","mapping_epoch":36,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:41.549520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8107276,"num_objects":990,"num_object_clones":0,"num_object_copies":3960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11768","reported_seq":11904,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.639488+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:40.639488+0000","last_peered":"2024-09-16T04:51:40.639488+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:40.639488+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7968012,"num_objects":973,"num_object_clones":0,"num_object_copies":3892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11766,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11411","reported_seq":11526,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.641666+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:40.641666+0000","last_peered":"2024-09-16T04:51:40.641666+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:40.641666+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7380992,"num_objects":901,"num_object_clones":0,"num_object_copies":3604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11411,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11687","reported_seq":11790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.062888+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:40.062888+0000","last_peered":"2024-09-16T04:51:40.062888+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:40.062888+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7476492,"num_objects":913,"num_object_clones":0,"num_object_copies":3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11685,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1300}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11769","reported_seq":16333,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:38.066705+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:38.066705+0000","last_peered":"2024-09-16T04:51:38.066705+0000","last_clean":"2024-09-16T04:51:38.066705+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:38.066705+0000","last_undegraded":"2024-09-16T04:51:38.066705+0000","last_fullsized":"2024-09-16T04:51:38.066705+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:41.549495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8320268,"num_objects":1016,"num_object_clones":0,"num_object_copies":4064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11767,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11798","reported_seq":16561,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:40.409668+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:40.409668+0000","last_peered":"2024-09-16T04:51:40.409668+0000","last_clean":"2024-09-16T04:51:40.409668+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:40.409668+0000","last_undegraded":"2024-09-16T04:51:40.409668+0000","last_fullsized":"2024-09-16T04:51:40.409668+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:41.713767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7842328,"num_objects":958,"num_object_clones":0,"num_object_copies":3832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11795,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11779","reported_seq":11891,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:38.743800+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:38.743800+0000","last_peered":"2024-09-16T04:51:38.743800+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:38.743800+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8017164,"num_objects":979,"num_object_clones":0,"num_object_copies":3916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11578","reported_seq":11715,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.712121+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:40.712121+0000","last_peered":"2024-09-16T04:51:40.712121+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:40.712121+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7475090,"num_objects":913,"num_object_clones":0,"num_object_copies":3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11575,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11433","reported_seq":11530,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.306334+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:40.306334+0000","last_peered":"2024-09-16T04:51:40.306334+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:40.306334+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7355014,"num_objects":898,"num_object_clones":0,"num_object_copies":3592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11432,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1207}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11600","reported_seq":11693,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:38.925873+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:38.925873+0000","last_peered":"2024-09-16T04:51:38.925873+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:38.925873+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7641734,"num_objects":933,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11599,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1288}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11838","reported_seq":11974,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.635485+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:40.635485+0000","last_peered":"2024-09-16T04:51:40.635485+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:40.635485+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7730444,"num_objects":944,"num_object_clones":0,"num_object_copies":3776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11836,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1254}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11498","reported_seq":12296,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:38.896187+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:38.896187+0000","last_peered":"2024-09-16T04:51:38.896187+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:38.896187+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7787788,"num_objects":951,"num_object_clones":0,"num_object_copies":3804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1781,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11496,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":355,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11399","reported_seq":11498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.242981+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:40.242981+0000","last_peered":"2024-09-16T04:51:40.242981+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:40.242981+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7435532,"num_objects":908,"num_object_clones":0,"num_object_copies":3632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11397,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":123373248,"num_objects":15066,"num_object_clones":0,"num_object_copies":60264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186421,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7440,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":248041472,"data_stored":248041472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161048,"ondisk_log_size":161048,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714168,"kb_used_data":60520,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93657672,"statfs":{"total":96636764160,"available":95905456128,"internally_reserved":0,"allocated":61972480,"data_stored":61773971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669314767},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,7,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724212,"kb_used_data":61348,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93647628,"statfs":{"total":96636764160,"available":95895171072,"internally_reserved":0,"allocated":62820352,"data_stored":62621843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,1,0,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608124,"kb_used_data":62748,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93763716,"statfs":{"total":96636764160,"available":96014045184,"internally_reserved":0,"allocated":64253952,"data_stored":64055443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558427855},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,7,5,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701644,"kb_used_data":60540,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93670196,"statfs":{"total":96636764160,"available":95918280704,"internally_reserved":0,"allocated":61992960,"data_stored":61794451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":656469710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61231104,"data_stored":61231104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63561728,"data_stored":63561728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":62345216,"data_stored":62345216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":60903424,"data_stored":60903424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:42.859 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:43.460 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:43.460 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:43.479 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-09-16T04:51:42.773364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":123373248,"num_objects":15066,"num_object_clones":0,"num_object_copies":60264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186421,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7440,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161048,"ondisk_log_size":161048,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2748148,"kb_used_data":245156,"kb_used_omap":14,"kb_used_meta":2502897,"kb_avail":374739212,"statfs":{"total":386547056640,"available":383732953088,"internally_reserved":0,"allocated":251039744,"data_stored":250245708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2562967089},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,9,6,15,9,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15851520,"num_objects":-1935,"num_object_clones":0,"num_object_copies":-7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1935,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003931"},"pg_stats":[{"pgid":"1.f","version":"37'11725","reported_seq":11846,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.611181+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:40.611181+0000","last_peered":"2024-09-16T04:51:40.611181+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:40.611181+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7610368,"num_objects":929,"num_object_clones":0,"num_object_copies":3716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11768","reported_seq":11860,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.276821+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:40.276821+0000","last_peered":"2024-09-16T04:51:40.276821+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:40.276821+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7419148,"num_objects":906,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11766,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1297}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11717","reported_seq":11821,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:38.613866+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:38.613866+0000","last_peered":"2024-09-16T04:51:38.613866+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:38.613866+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7805598,"num_objects":954,"num_object_clones":0,"num_object_copies":3816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11716,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1265}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11680","reported_seq":16230,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:37.950783+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:37.950783+0000","last_peered":"2024-09-16T04:51:37.950783+0000","last_clean":"2024-09-16T04:51:37.950783+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:37.950783+0000","last_undegraded":"2024-09-16T04:51:37.950783+0000","last_fullsized":"2024-09-16T04:51:37.950783+0000","mapping_epoch":36,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:41.549520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8107276,"num_objects":990,"num_object_clones":0,"num_object_copies":3960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11768","reported_seq":11904,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.639488+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:40.639488+0000","last_peered":"2024-09-16T04:51:40.639488+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:40.639488+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7968012,"num_objects":973,"num_object_clones":0,"num_object_copies":3892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11766,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11411","reported_seq":11526,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.641666+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:40.641666+0000","last_peered":"2024-09-16T04:51:40.641666+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:40.641666+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7380992,"num_objects":901,"num_object_clones":0,"num_object_copies":3604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11411,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11687","reported_seq":11790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.062888+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:40.062888+0000","last_peered":"2024-09-16T04:51:40.062888+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:40.062888+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7476492,"num_objects":913,"num_object_clones":0,"num_object_copies":3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11685,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1300}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11769","reported_seq":16333,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:38.066705+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:38.066705+0000","last_peered":"2024-09-16T04:51:38.066705+0000","last_clean":"2024-09-16T04:51:38.066705+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:38.066705+0000","last_undegraded":"2024-09-16T04:51:38.066705+0000","last_fullsized":"2024-09-16T04:51:38.066705+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:41.549495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8320268,"num_objects":1016,"num_object_clones":0,"num_object_copies":4064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11767,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11798","reported_seq":16561,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:40.409668+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:40.409668+0000","last_peered":"2024-09-16T04:51:40.409668+0000","last_clean":"2024-09-16T04:51:40.409668+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:40.409668+0000","last_undegraded":"2024-09-16T04:51:40.409668+0000","last_fullsized":"2024-09-16T04:51:40.409668+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:41.713767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7842328,"num_objects":958,"num_object_clones":0,"num_object_copies":3832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11795,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11779","reported_seq":11891,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:38.743800+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:38.743800+0000","last_peered":"2024-09-16T04:51:38.743800+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:38.743800+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8017164,"num_objects":979,"num_object_clones":0,"num_object_copies":3916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11578","reported_seq":11715,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.712121+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:40.712121+0000","last_peered":"2024-09-16T04:51:40.712121+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:40.712121+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7475090,"num_objects":913,"num_object_clones":0,"num_object_copies":3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11575,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11433","reported_seq":11530,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.306334+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:40.306334+0000","last_peered":"2024-09-16T04:51:40.306334+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:40.306334+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7355014,"num_objects":898,"num_object_clones":0,"num_object_copies":3592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11432,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1207}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11600","reported_seq":11693,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:38.925873+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:38.925873+0000","last_peered":"2024-09-16T04:51:38.925873+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:38.925873+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7641734,"num_objects":933,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11599,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1288}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11838","reported_seq":11974,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.635485+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:40.635485+0000","last_peered":"2024-09-16T04:51:40.635485+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:40.635485+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7730444,"num_objects":944,"num_object_clones":0,"num_object_copies":3776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11836,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1254}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11498","reported_seq":12296,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:38.896187+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:38.896187+0000","last_peered":"2024-09-16T04:51:38.896187+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:38.896187+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7787788,"num_objects":951,"num_object_clones":0,"num_object_copies":3804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1781,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11496,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":355,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11399","reported_seq":11498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.242981+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:40.242981+0000","last_peered":"2024-09-16T04:51:40.242981+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:40.242981+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7435532,"num_objects":908,"num_object_clones":0,"num_object_copies":3632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11397,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":123373248,"num_objects":15066,"num_object_clones":0,"num_object_copies":60264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186421,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7440,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":248041472,"data_stored":248041472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161048,"ondisk_log_size":161048,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714168,"kb_used_data":60520,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93657672,"statfs":{"total":96636764160,"available":95905456128,"internally_reserved":0,"allocated":61972480,"data_stored":61773971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669314767},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,7,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724212,"kb_used_data":61348,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93647628,"statfs":{"total":96636764160,"available":95895171072,"internally_reserved":0,"allocated":62820352,"data_stored":62621843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,1,0,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608124,"kb_used_data":62748,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93763716,"statfs":{"total":96636764160,"available":96014045184,"internally_reserved":0,"allocated":64253952,"data_stored":64055443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558427855},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,7,5,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701644,"kb_used_data":60540,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93670196,"statfs":{"total":96636764160,"available":95918280704,"internally_reserved":0,"allocated":61992960,"data_stored":61794451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":656469710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61231104,"data_stored":61231104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63561728,"data_stored":63561728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":62345216,"data_stored":62345216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":60903424,"data_stored":60903424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:43.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:51:43.480 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11725", 'reported_seq': 11846, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:40.611181+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:51:40.611181+0000', 'last_peered': '2024-09-16T04:51:40.611181+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:51:40.611181+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7610368, 'num_objects': 929, 'num_object_clones': 0, 'num_object_copies': 3716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11725, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1257}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:43.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:51:43.480 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11768", 'reported_seq': 11860, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:40.276821+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:51:40.276821+0000', 'last_peered': '2024-09-16T04:51:40.276821+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:51:40.276821+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7419148, 'num_objects': 906, 'num_object_clones': 0, 'num_object_copies': 3624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11766, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1297}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:43.480 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:51:43.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11717", 'reported_seq': 11821, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:38.613866+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:51:38.613866+0000', 'last_peered': '2024-09-16T04:51:38.613866+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:51:38.613866+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7805598, 'num_objects': 954, 'num_object_clones': 0, 'num_object_copies': 3816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 954, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11716, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1265}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:43.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:51:43.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11768", 'reported_seq': 11904, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:40.639488+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:51:40.639488+0000', 'last_peered': '2024-09-16T04:51:40.639488+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:51:40.639488+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7968012, 'num_objects': 973, 'num_object_clones': 0, 'num_object_copies': 3892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11766, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1275}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:43.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:51:43.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11411", 'reported_seq': 11526, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:40.641666+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:51:40.641666+0000', 'last_peered': '2024-09-16T04:51:40.641666+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:51:40.641666+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7380992, 'num_objects': 901, 'num_object_clones': 0, 'num_object_copies': 3604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 901, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11411, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1277}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:43.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:51:43.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11687", 'reported_seq': 11790, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:40.062888+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:51:40.062888+0000', 'last_peered': '2024-09-16T04:51:40.062888+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:51:40.062888+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7476492, 'num_objects': 913, 'num_object_clones': 0, 'num_object_copies': 3652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11685, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1300}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:43.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:51:43.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11779", 'reported_seq': 11891, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:38.743800+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:51:38.743800+0000', 'last_peered': '2024-09-16T04:51:38.743800+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:51:38.743800+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8017164, 'num_objects': 979, 'num_object_clones': 0, 'num_object_copies': 3916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:43.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:51:43.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11578", 'reported_seq': 11715, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:40.712121+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:51:40.712121+0000', 'last_peered': '2024-09-16T04:51:40.712121+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:51:40.712121+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7475090, 'num_objects': 913, 'num_object_clones': 0, 'num_object_copies': 3652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11575, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1242}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:43.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:51:43.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11433", 'reported_seq': 11530, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:40.306334+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:51:40.306334+0000', 'last_peered': '2024-09-16T04:51:40.306334+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:51:40.306334+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7355014, 'num_objects': 898, 'num_object_clones': 0, 'num_object_copies': 3592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 898, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11432, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1207}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:43.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:51:43.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11600", 'reported_seq': 11693, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:38.925873+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:51:38.925873+0000', 'last_peered': '2024-09-16T04:51:38.925873+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:51:38.925873+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7641734, 'num_objects': 933, 'num_object_clones': 0, 'num_object_copies': 3732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 933, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11599, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1288}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:43.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:51:43.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11838", 'reported_seq': 11974, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:40.635485+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:51:40.635485+0000', 'last_peered': '2024-09-16T04:51:40.635485+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:51:40.635485+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7730444, 'num_objects': 944, 'num_object_clones': 0, 'num_object_copies': 3776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 944, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11836, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1254}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:43.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:51:43.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11498", 'reported_seq': 12296, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:38.896187+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:51:38.896187+0000', 'last_peered': '2024-09-16T04:51:38.896187+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:51:38.896187+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7787788, 'num_objects': 951, 'num_object_clones': 0, 'num_object_copies': 3804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1781, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 951, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11496, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 355, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -830}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:43.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:51:43.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11399", 'reported_seq': 11498, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:40.242981+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:51:40.242981+0000', 'last_peered': '2024-09-16T04:51:40.242981+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:51:40.242981+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7435532, 'num_objects': 908, 'num_object_clones': 0, 'num_object_copies': 3632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11397, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1320}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:43.484 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:51:44.215 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:44.215 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28142/60264 objects degraded (46.698%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":716,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":15066,"data_bytes":123373248,"bytes_used":2814103552,"bytes_avail":383732953088,"bytes_total":386547056640,"degraded_objects":28142,"degraded_total":60264,"degraded_ratio":0.46697862737289259,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":161},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:51:44.235 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:51:44.235 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:44.840 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:44.841 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:44.877 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-09-16T04:51:44.774144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120260288,"num_objects":14686,"num_object_clones":0,"num_object_copies":58744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14686,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186801,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7492,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2750084,"kb_used_data":237348,"kb_used_omap":14,"kb_used_meta":2512625,"kb_avail":374737276,"statfs":{"total":386547056640,"available":383730970624,"internally_reserved":0,"allocated":243044352,"data_stored":242250316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2572928561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,7,8,16,11,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18964480,"num_objects":-2315,"num_object_clones":0,"num_object_copies":-9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2315,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003884"},"pg_stats":[{"pgid":"1.f","version":"37'11725","reported_seq":11846,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.611181+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:40.611181+0000","last_peered":"2024-09-16T04:51:40.611181+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:40.611181+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7610368,"num_objects":929,"num_object_clones":0,"num_object_copies":3716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11768","reported_seq":11860,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.276821+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:40.276821+0000","last_peered":"2024-09-16T04:51:40.276821+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:40.276821+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7419148,"num_objects":906,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11766,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1297}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11790","reported_seq":11894,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.990260+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:43.990260+0000","last_peered":"2024-09-16T04:51:43.990260+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:43.990260+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7207582,"num_objects":881,"num_object_clones":0,"num_object_copies":3524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11789,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11739","reported_seq":16289,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:42.698996+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:42.698996+0000","last_peered":"2024-09-16T04:51:42.698996+0000","last_clean":"2024-09-16T04:51:42.698996+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:42.698996+0000","last_undegraded":"2024-09-16T04:51:42.698996+0000","last_fullsized":"2024-09-16T04:51:42.698996+0000","mapping_epoch":36,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:46.628430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7623948,"num_objects":931,"num_object_clones":0,"num_object_copies":3724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11737,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11768","reported_seq":11904,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.639488+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:40.639488+0000","last_peered":"2024-09-16T04:51:40.639488+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:40.639488+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7968012,"num_objects":973,"num_object_clones":0,"num_object_copies":3892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11766,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11411","reported_seq":11526,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.641666+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:40.641666+0000","last_peered":"2024-09-16T04:51:40.641666+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:40.641666+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7380992,"num_objects":901,"num_object_clones":0,"num_object_copies":3604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11411,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11687","reported_seq":11790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.062888+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:40.062888+0000","last_peered":"2024-09-16T04:51:40.062888+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:40.062888+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7476492,"num_objects":913,"num_object_clones":0,"num_object_copies":3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11685,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1300}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11837","reported_seq":16402,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:43.052268+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:43.052268+0000","last_peered":"2024-09-16T04:51:43.052268+0000","last_clean":"2024-09-16T04:51:43.052268+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:43.052268+0000","last_undegraded":"2024-09-16T04:51:43.052268+0000","last_fullsized":"2024-09-16T04:51:43.052268+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:46.628404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7763212,"num_objects":948,"num_object_clones":0,"num_object_copies":3792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11835,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11798","reported_seq":16561,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:40.409668+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:40.409668+0000","last_peered":"2024-09-16T04:51:40.409668+0000","last_clean":"2024-09-16T04:51:40.409668+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:40.409668+0000","last_undegraded":"2024-09-16T04:51:40.409668+0000","last_fullsized":"2024-09-16T04:51:40.409668+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:41.713767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7842328,"num_objects":958,"num_object_clones":0,"num_object_copies":3832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11795,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11837","reported_seq":11949,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.928382+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:43.928382+0000","last_peered":"2024-09-16T04:51:43.928382+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:43.928382+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7542028,"num_objects":921,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11835,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1268}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11578","reported_seq":11715,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.712121+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:40.712121+0000","last_peered":"2024-09-16T04:51:40.712121+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:40.712121+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7475090,"num_objects":913,"num_object_clones":0,"num_object_copies":3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11575,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11433","reported_seq":11530,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.306334+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:40.306334+0000","last_peered":"2024-09-16T04:51:40.306334+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:40.306334+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7355014,"num_objects":898,"num_object_clones":0,"num_object_copies":3592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11432,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1207}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11660","reported_seq":11754,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.752521+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:43.752521+0000","last_peered":"2024-09-16T04:51:43.752521+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:43.752521+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7150214,"num_objects":873,"num_object_clones":0,"num_object_copies":3492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11659,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1348}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11838","reported_seq":11974,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.635485+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:40.635485+0000","last_peered":"2024-09-16T04:51:40.635485+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:40.635485+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7730444,"num_objects":944,"num_object_clones":0,"num_object_copies":3776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11836,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1254}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11560","reported_seq":12462,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.987960+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:43.987960+0000","last_peered":"2024-09-16T04:51:43.987960+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:43.987960+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7279884,"num_objects":889,"num_object_clones":0,"num_object_copies":3556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1729,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11558,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-840}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11399","reported_seq":11498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:40.242981+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:40.242981+0000","last_peered":"2024-09-16T04:51:40.242981+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:40.242981+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7435532,"num_objects":908,"num_object_clones":0,"num_object_copies":3632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11397,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120260288,"num_objects":14686,"num_object_clones":0,"num_object_copies":58744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14686,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186801,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7492,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":239972352,"data_stored":239972352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714168,"kb_used_data":60520,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93657672,"statfs":{"total":96636764160,"available":95905456128,"internally_reserved":0,"allocated":61972480,"data_stored":61773971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":669314767},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,7,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742580,"kb_used_data":57476,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93629260,"statfs":{"total":96636764160,"available":95876362240,"internally_reserved":0,"allocated":58855424,"data_stored":58656915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591692,"kb_used_data":58812,"kb_used_omap":4,"kb_used_meta":532859,"kb_avail":93780148,"statfs":{"total":96636764160,"available":96030871552,"internally_reserved":0,"allocated":60223488,"data_stored":60024979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545648335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701644,"kb_used_data":60540,"kb_used_omap":4,"kb_used_meta":641083,"kb_avail":93670196,"statfs":{"total":96636764160,"available":95918280704,"internally_reserved":0,"allocated":61992960,"data_stored":61794451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":656469710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61231104,"data_stored":61231104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":59527168,"data_stored":59527168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58310656,"data_stored":58310656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":60903424,"data_stored":60903424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:47.879 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:48.511 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:48.511 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:48.541 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-09-16T04:51:46.775045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":115295936,"num_objects":14080,"num_object_clones":0,"num_object_copies":56320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14080,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187407,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7492,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160834,"ondisk_log_size":160834,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2770148,"kb_used_data":229596,"kb_used_omap":14,"kb_used_meta":2540401,"kb_avail":374717212,"statfs":{"total":386547056640,"available":383710425088,"internally_reserved":0,"allocated":235106304,"data_stored":234312268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2601371185},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,6,11,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20684800,"num_objects":-2525,"num_object_clones":0,"num_object_copies":-10100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2525,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004174"},"pg_stats":[{"pgid":"1.f","version":"37'11774","reported_seq":11895,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.537904+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:45.537904+0000","last_peered":"2024-09-16T04:51:45.537904+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:45.537904+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7208960,"num_objects":880,"num_object_clones":0,"num_object_copies":3520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11774,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1306}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11826","reported_seq":11919,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.279153+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:45.279153+0000","last_peered":"2024-09-16T04:51:45.279153+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:45.279153+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6944012,"num_objects":848,"num_object_clones":0,"num_object_copies":3392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11824,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1355}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11790","reported_seq":11894,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.990260+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:43.990260+0000","last_peered":"2024-09-16T04:51:43.990260+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:43.990260+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7207582,"num_objects":881,"num_object_clones":0,"num_object_copies":3524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11789,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11739","reported_seq":16289,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:42.698996+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:42.698996+0000","last_peered":"2024-09-16T04:51:42.698996+0000","last_clean":"2024-09-16T04:51:42.698996+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:42.698996+0000","last_undegraded":"2024-09-16T04:51:42.698996+0000","last_fullsized":"2024-09-16T04:51:42.698996+0000","mapping_epoch":36,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:46.628430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7623948,"num_objects":931,"num_object_clones":0,"num_object_copies":3724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11737,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11839","reported_seq":11977,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.713337+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:45.713337+0000","last_peered":"2024-09-16T04:51:45.713337+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:45.713337+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7386380,"num_objects":902,"num_object_clones":0,"num_object_copies":3608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11837,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11466","reported_seq":11581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.518114+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:45.518114+0000","last_peered":"2024-09-16T04:51:45.518114+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:45.518114+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6930432,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11466,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1332}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11748","reported_seq":11851,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.118760+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:45.118760+0000","last_peered":"2024-09-16T04:51:45.118760+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:45.118760+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6976780,"num_objects":852,"num_object_clones":0,"num_object_copies":3408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11746,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11837","reported_seq":16402,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:43.052268+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:43.052268+0000","last_peered":"2024-09-16T04:51:43.052268+0000","last_clean":"2024-09-16T04:51:43.052268+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:43.052268+0000","last_undegraded":"2024-09-16T04:51:43.052268+0000","last_fullsized":"2024-09-16T04:51:43.052268+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:46.628404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7763212,"num_objects":948,"num_object_clones":0,"num_object_copies":3792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11835,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11854","reported_seq":16617,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:45.606926+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:45.606926+0000","last_peered":"2024-09-16T04:51:45.606926+0000","last_clean":"2024-09-16T04:51:45.606926+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:45.606926+0000","last_undegraded":"2024-09-16T04:51:45.606926+0000","last_fullsized":"2024-09-16T04:51:45.606926+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:46.858627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7383576,"num_objects":902,"num_object_clones":0,"num_object_copies":3608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11837","reported_seq":11949,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.928382+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:43.928382+0000","last_peered":"2024-09-16T04:51:43.928382+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:43.928382+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7542028,"num_objects":921,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11835,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1268}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11645","reported_seq":11778,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.552346+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:45.552346+0000","last_peered":"2024-09-16T04:51:45.552346+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:45.552346+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6926226,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11642,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11496","reported_seq":11594,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.676288+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:45.676288+0000","last_peered":"2024-09-16T04:51:45.676288+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:45.676288+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6838918,"num_objects":835,"num_object_clones":0,"num_object_copies":3340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11495,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1270}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11660","reported_seq":11754,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.752521+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:43.752521+0000","last_peered":"2024-09-16T04:51:43.752521+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:43.752521+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7150214,"num_objects":873,"num_object_clones":0,"num_object_copies":3492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11659,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1348}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11904","reported_seq":12041,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.730111+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:45.730111+0000","last_peered":"2024-09-16T04:51:45.730111+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:45.730111+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7189772,"num_objects":878,"num_object_clones":0,"num_object_copies":3512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11902,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1320}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11560","reported_seq":12462,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.987960+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:43.987960+0000","last_peered":"2024-09-16T04:51:43.987960+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:43.987960+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7279884,"num_objects":889,"num_object_clones":0,"num_object_copies":3556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1729,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11558,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-840}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11459","reported_seq":11559,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.052687+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:45.052687+0000","last_peered":"2024-09-16T04:51:45.052687+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:45.052687+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6944012,"num_objects":848,"num_object_clones":0,"num_object_copies":3392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11457,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1380}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":115295936,"num_objects":14080,"num_object_clones":0,"num_object_copies":56320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14080,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187407,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7492,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":232103936,"data_stored":232103936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160834,"ondisk_log_size":160834,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715904,"kb_used_data":56648,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93655936,"statfs":{"total":96636764160,"available":95903678464,"internally_reserved":0,"allocated":58007552,"data_stored":57809043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":675016399},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742580,"kb_used_data":57476,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93629260,"statfs":{"total":96636764160,"available":95876362240,"internally_reserved":0,"allocated":58855424,"data_stored":58656915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591692,"kb_used_data":58812,"kb_used_omap":4,"kb_used_meta":532859,"kb_avail":93780148,"statfs":{"total":96636764160,"available":96030871552,"internally_reserved":0,"allocated":60223488,"data_stored":60024979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545648335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247946,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719972,"kb_used_data":56660,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93651868,"statfs":{"total":96636764160,"available":95899512832,"internally_reserved":0,"allocated":58019840,"data_stored":57821331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":679210702},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57319424,"data_stored":57319424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":59527168,"data_stored":59527168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58310656,"data_stored":58310656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":56946688,"data_stored":56946688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:48.542 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:49.123 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:49.123 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:49.145 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-09-16T04:51:48.775758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":114181824,"num_objects":13944,"num_object_clones":0,"num_object_copies":55776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13944,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187543,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7492,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160770,"ondisk_log_size":160770,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2770428,"kb_used_data":225708,"kb_used_omap":14,"kb_used_meta":2544561,"kb_avail":374716932,"statfs":{"total":386547056640,"available":383710138368,"internally_reserved":0,"allocated":231124992,"data_stored":230330956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2605631025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,5,10,17,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17039360,"num_objects":-2080,"num_object_clones":0,"num_object_copies":-8320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2080,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004386"},"pg_stats":[{"pgid":"1.f","version":"37'11774","reported_seq":11895,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.537904+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:45.537904+0000","last_peered":"2024-09-16T04:51:45.537904+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:45.537904+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7208960,"num_objects":880,"num_object_clones":0,"num_object_copies":3520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11774,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1306}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11826","reported_seq":11919,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.279153+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:45.279153+0000","last_peered":"2024-09-16T04:51:45.279153+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:45.279153+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6944012,"num_objects":848,"num_object_clones":0,"num_object_copies":3392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11824,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1355}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11790","reported_seq":11894,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.990260+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:43.990260+0000","last_peered":"2024-09-16T04:51:43.990260+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:43.990260+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7207582,"num_objects":881,"num_object_clones":0,"num_object_copies":3524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11789,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11802","reported_seq":16352,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:48.065394+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:48.065394+0000","last_peered":"2024-09-16T04:51:48.065394+0000","last_clean":"2024-09-16T04:51:48.065394+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:48.065394+0000","last_undegraded":"2024-09-16T04:51:48.065394+0000","last_fullsized":"2024-09-16T04:51:48.065394+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:51.736335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7107852,"num_objects":868,"num_object_clones":0,"num_object_copies":3472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11800,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11839","reported_seq":11977,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.713337+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:45.713337+0000","last_peered":"2024-09-16T04:51:45.713337+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:45.713337+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7386380,"num_objects":902,"num_object_clones":0,"num_object_copies":3608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11837,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11466","reported_seq":11581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.518114+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:45.518114+0000","last_peered":"2024-09-16T04:51:45.518114+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:45.518114+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6930432,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11466,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1332}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11748","reported_seq":11851,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.118760+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:45.118760+0000","last_peered":"2024-09-16T04:51:45.118760+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:45.118760+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6976780,"num_objects":852,"num_object_clones":0,"num_object_copies":3408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11746,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11910","reported_seq":16477,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:48.107838+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:48.107838+0000","last_peered":"2024-09-16T04:51:48.107838+0000","last_clean":"2024-09-16T04:51:48.107838+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:48.107838+0000","last_undegraded":"2024-09-16T04:51:48.107838+0000","last_fullsized":"2024-09-16T04:51:48.107838+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:51.736312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7165196,"num_objects":875,"num_object_clones":0,"num_object_copies":3500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11908,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11854","reported_seq":16617,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:45.606926+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:45.606926+0000","last_peered":"2024-09-16T04:51:45.606926+0000","last_clean":"2024-09-16T04:51:45.606926+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:45.606926+0000","last_undegraded":"2024-09-16T04:51:45.606926+0000","last_fullsized":"2024-09-16T04:51:45.606926+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:46.858627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7383576,"num_objects":902,"num_object_clones":0,"num_object_copies":3608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11837","reported_seq":11949,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.928382+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:43.928382+0000","last_peered":"2024-09-16T04:51:43.928382+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:43.928382+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7542028,"num_objects":921,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11835,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1268}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11645","reported_seq":11778,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.552346+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:45.552346+0000","last_peered":"2024-09-16T04:51:45.552346+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:45.552346+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6926226,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11642,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11496","reported_seq":11594,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.676288+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:45.676288+0000","last_peered":"2024-09-16T04:51:45.676288+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:45.676288+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6838918,"num_objects":835,"num_object_clones":0,"num_object_copies":3340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11495,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1270}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11660","reported_seq":11754,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.752521+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:43.752521+0000","last_peered":"2024-09-16T04:51:43.752521+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:43.752521+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7150214,"num_objects":873,"num_object_clones":0,"num_object_copies":3492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11659,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1348}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11904","reported_seq":12041,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.730111+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:45.730111+0000","last_peered":"2024-09-16T04:51:45.730111+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:45.730111+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7189772,"num_objects":878,"num_object_clones":0,"num_object_copies":3512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11902,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1320}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11560","reported_seq":12462,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.987960+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:43.987960+0000","last_peered":"2024-09-16T04:51:43.987960+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:43.987960+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7279884,"num_objects":889,"num_object_clones":0,"num_object_copies":3556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1729,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11558,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-840}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11459","reported_seq":11559,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.052687+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:45.052687+0000","last_peered":"2024-09-16T04:51:45.052687+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:45.052687+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6944012,"num_objects":848,"num_object_clones":0,"num_object_copies":3392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11457,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1380}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":114181824,"num_objects":13944,"num_object_clones":0,"num_object_copies":55776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13944,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187543,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7492,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":228159488,"data_stored":228159488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160770,"ondisk_log_size":160770,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715904,"kb_used_data":56648,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93655936,"statfs":{"total":96636764160,"available":95903678464,"internally_reserved":0,"allocated":58007552,"data_stored":57809043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":675016399},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742580,"kb_used_data":57476,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93629260,"statfs":{"total":96636764160,"available":95876362240,"internally_reserved":0,"allocated":58855424,"data_stored":58656915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591972,"kb_used_data":54924,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93779868,"statfs":{"total":96636764160,"available":96030584832,"internally_reserved":0,"allocated":56242176,"data_stored":56043667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549908175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247946,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719972,"kb_used_data":56660,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93651868,"statfs":{"total":96636764160,"available":95899512832,"internally_reserved":0,"allocated":58019840,"data_stored":57821331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":679210702},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57319424,"data_stored":57319424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55582720,"data_stored":55582720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58310656,"data_stored":58310656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":56946688,"data_stored":56946688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:49.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:51:49.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11774", 'reported_seq': 11895, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:45.537904+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:51:45.537904+0000', 'last_peered': '2024-09-16T04:51:45.537904+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:51:45.537904+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7208960, 'num_objects': 880, 'num_object_clones': 0, 'num_object_copies': 3520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11774, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1306}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:49.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:51:49.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11826", 'reported_seq': 11919, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:45.279153+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:51:45.279153+0000', 'last_peered': '2024-09-16T04:51:45.279153+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:51:45.279153+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6944012, 'num_objects': 848, 'num_object_clones': 0, 'num_object_copies': 3392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11824, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1355}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:49.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:51:49.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11790", 'reported_seq': 11894, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:43.990260+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:51:43.990260+0000', 'last_peered': '2024-09-16T04:51:43.990260+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:51:43.990260+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7207582, 'num_objects': 881, 'num_object_clones': 0, 'num_object_copies': 3524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 881, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11789, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1338}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:49.146 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:51:49.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11839", 'reported_seq': 11977, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:45.713337+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:51:45.713337+0000', 'last_peered': '2024-09-16T04:51:45.713337+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:51:45.713337+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7386380, 'num_objects': 902, 'num_object_clones': 0, 'num_object_copies': 3608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 902, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11837, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1346}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:49.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:51:49.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11466", 'reported_seq': 11581, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:45.518114+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:51:45.518114+0000', 'last_peered': '2024-09-16T04:51:45.518114+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:51:45.518114+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6930432, 'num_objects': 846, 'num_object_clones': 0, 'num_object_copies': 3384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 846, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11466, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1332}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:49.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:51:49.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11748", 'reported_seq': 11851, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:45.118760+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:51:45.118760+0000', 'last_peered': '2024-09-16T04:51:45.118760+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:51:45.118760+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6976780, 'num_objects': 852, 'num_object_clones': 0, 'num_object_copies': 3408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11746, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1361}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:49.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:51:49.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11837", 'reported_seq': 11949, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:43.928382+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:51:43.928382+0000', 'last_peered': '2024-09-16T04:51:43.928382+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:51:43.928382+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7542028, 'num_objects': 921, 'num_object_clones': 0, 'num_object_copies': 3684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 921, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11835, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1268}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:49.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:51:49.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11645", 'reported_seq': 11778, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:45.552346+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:51:45.552346+0000', 'last_peered': '2024-09-16T04:51:45.552346+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:51:45.552346+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6926226, 'num_objects': 846, 'num_object_clones': 0, 'num_object_copies': 3384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 846, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11642, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:49.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:51:49.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11496", 'reported_seq': 11594, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:45.676288+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:51:45.676288+0000', 'last_peered': '2024-09-16T04:51:45.676288+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:51:45.676288+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6838918, 'num_objects': 835, 'num_object_clones': 0, 'num_object_copies': 3340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 835, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11495, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1270}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:49.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:51:49.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11660", 'reported_seq': 11754, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:43.752521+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:51:43.752521+0000', 'last_peered': '2024-09-16T04:51:43.752521+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:51:43.752521+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7150214, 'num_objects': 873, 'num_object_clones': 0, 'num_object_copies': 3492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11659, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1348}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:49.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:51:49.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11904", 'reported_seq': 12041, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:45.730111+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:51:45.730111+0000', 'last_peered': '2024-09-16T04:51:45.730111+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:51:45.730111+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7189772, 'num_objects': 878, 'num_object_clones': 0, 'num_object_copies': 3512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 878, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11902, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1320}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:49.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:51:49.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11560", 'reported_seq': 12462, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:43.987960+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:51:43.987960+0000', 'last_peered': '2024-09-16T04:51:43.987960+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:51:43.987960+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7279884, 'num_objects': 889, 'num_object_clones': 0, 'num_object_copies': 3556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1729, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 889, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11558, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 407, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -840}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:49.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:51:49.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11459", 'reported_seq': 11559, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:45.052687+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:51:45.052687+0000', 'last_peered': '2024-09-16T04:51:45.052687+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:51:45.052687+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6944012, 'num_objects': 848, 'num_object_clones': 0, 'num_object_copies': 3392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11457, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1380}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:49.148 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:51:49.910 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:49.910 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28090/55776 objects degraded (50.362%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":722,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":13944,"data_bytes":114181824,"bytes_used":2836918272,"bytes_avail":383710138368,"bytes_total":386547056640,"degraded_objects":28090,"degraded_total":55776,"degraded_ratio":0.50362162937464139,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":173},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:51:49.933 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:51:49.934 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:50.492 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:50.492 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:50.521 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-09-16T04:51:48.775758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":114181824,"num_objects":13944,"num_object_clones":0,"num_object_copies":55776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13944,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187543,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7492,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160770,"ondisk_log_size":160770,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2770428,"kb_used_data":225708,"kb_used_omap":14,"kb_used_meta":2544561,"kb_avail":374716932,"statfs":{"total":386547056640,"available":383710138368,"internally_reserved":0,"allocated":231124992,"data_stored":230330956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2605631025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,5,10,17,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17039360,"num_objects":-2080,"num_object_clones":0,"num_object_copies":-8320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2080,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004386"},"pg_stats":[{"pgid":"1.f","version":"37'11774","reported_seq":11895,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.537904+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:45.537904+0000","last_peered":"2024-09-16T04:51:45.537904+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:45.537904+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7208960,"num_objects":880,"num_object_clones":0,"num_object_copies":3520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11774,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1306}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11826","reported_seq":11919,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.279153+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:45.279153+0000","last_peered":"2024-09-16T04:51:45.279153+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:45.279153+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6944012,"num_objects":848,"num_object_clones":0,"num_object_copies":3392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11824,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1355}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11790","reported_seq":11894,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.990260+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:43.990260+0000","last_peered":"2024-09-16T04:51:43.990260+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:43.990260+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7207582,"num_objects":881,"num_object_clones":0,"num_object_copies":3524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11789,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11802","reported_seq":16352,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:48.065394+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:48.065394+0000","last_peered":"2024-09-16T04:51:48.065394+0000","last_clean":"2024-09-16T04:51:48.065394+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:48.065394+0000","last_undegraded":"2024-09-16T04:51:48.065394+0000","last_fullsized":"2024-09-16T04:51:48.065394+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:51.736335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7107852,"num_objects":868,"num_object_clones":0,"num_object_copies":3472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11800,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11839","reported_seq":11977,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.713337+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:45.713337+0000","last_peered":"2024-09-16T04:51:45.713337+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:45.713337+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7386380,"num_objects":902,"num_object_clones":0,"num_object_copies":3608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11837,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11466","reported_seq":11581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.518114+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:45.518114+0000","last_peered":"2024-09-16T04:51:45.518114+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:45.518114+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6930432,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11466,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1332}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11748","reported_seq":11851,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.118760+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:45.118760+0000","last_peered":"2024-09-16T04:51:45.118760+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:45.118760+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6976780,"num_objects":852,"num_object_clones":0,"num_object_copies":3408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11746,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11910","reported_seq":16477,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:48.107838+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:48.107838+0000","last_peered":"2024-09-16T04:51:48.107838+0000","last_clean":"2024-09-16T04:51:48.107838+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:48.107838+0000","last_undegraded":"2024-09-16T04:51:48.107838+0000","last_fullsized":"2024-09-16T04:51:48.107838+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:51.736312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7165196,"num_objects":875,"num_object_clones":0,"num_object_copies":3500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11908,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11854","reported_seq":16617,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:45.606926+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:45.606926+0000","last_peered":"2024-09-16T04:51:45.606926+0000","last_clean":"2024-09-16T04:51:45.606926+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:45.606926+0000","last_undegraded":"2024-09-16T04:51:45.606926+0000","last_fullsized":"2024-09-16T04:51:45.606926+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:46.858627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7383576,"num_objects":902,"num_object_clones":0,"num_object_copies":3608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11837","reported_seq":11949,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.928382+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:43.928382+0000","last_peered":"2024-09-16T04:51:43.928382+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:43.928382+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7542028,"num_objects":921,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11835,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1268}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11645","reported_seq":11778,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.552346+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:45.552346+0000","last_peered":"2024-09-16T04:51:45.552346+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:45.552346+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6926226,"num_objects":846,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11642,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11496","reported_seq":11594,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.676288+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:45.676288+0000","last_peered":"2024-09-16T04:51:45.676288+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:45.676288+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6838918,"num_objects":835,"num_object_clones":0,"num_object_copies":3340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11495,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1270}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11660","reported_seq":11754,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.752521+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:43.752521+0000","last_peered":"2024-09-16T04:51:43.752521+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:43.752521+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7150214,"num_objects":873,"num_object_clones":0,"num_object_copies":3492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11659,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1348}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11904","reported_seq":12041,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.730111+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:45.730111+0000","last_peered":"2024-09-16T04:51:45.730111+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:45.730111+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7189772,"num_objects":878,"num_object_clones":0,"num_object_copies":3512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11902,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1320}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11560","reported_seq":12462,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:43.987960+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:43.987960+0000","last_peered":"2024-09-16T04:51:43.987960+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:43.987960+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7279884,"num_objects":889,"num_object_clones":0,"num_object_copies":3556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1729,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11558,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-840}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11459","reported_seq":11559,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:45.052687+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:45.052687+0000","last_peered":"2024-09-16T04:51:45.052687+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:45.052687+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6944012,"num_objects":848,"num_object_clones":0,"num_object_copies":3392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11457,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1380}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":114181824,"num_objects":13944,"num_object_clones":0,"num_object_copies":55776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13944,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187543,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7492,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":228159488,"data_stored":228159488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160770,"ondisk_log_size":160770,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715904,"kb_used_data":56648,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93655936,"statfs":{"total":96636764160,"available":95903678464,"internally_reserved":0,"allocated":58007552,"data_stored":57809043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":675016399},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742580,"kb_used_data":57476,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93629260,"statfs":{"total":96636764160,"available":95876362240,"internally_reserved":0,"allocated":58855424,"data_stored":58656915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591972,"kb_used_data":54924,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93779868,"statfs":{"total":96636764160,"available":96030584832,"internally_reserved":0,"allocated":56242176,"data_stored":56043667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549908175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247946,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719972,"kb_used_data":56660,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93651868,"statfs":{"total":96636764160,"available":95899512832,"internally_reserved":0,"allocated":58019840,"data_stored":57821331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":679210702},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57319424,"data_stored":57319424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55582720,"data_stored":55582720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58310656,"data_stored":58310656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":56946688,"data_stored":56946688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:53.523 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:54.115 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:54.115 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:54.146 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-09-16T04:51:52.777268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107415232,"num_objects":13118,"num_object_clones":0,"num_object_copies":52472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13118,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188369,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7552,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160696,"ondisk_log_size":160696,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771336,"kb_used_data":214184,"kb_used_omap":14,"kb_used_meta":2557041,"kb_avail":374716024,"statfs":{"total":386547056640,"available":383709208576,"internally_reserved":0,"allocated":219324416,"data_stored":218530380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2618410545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,6,6,16,23,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15958016,"num_objects":-1948,"num_object_clones":0,"num_object_copies":-7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1948,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004727"},"pg_stats":[{"pgid":"1.f","version":"37'11835","reported_seq":11956,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.659214+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:50.659214+0000","last_peered":"2024-09-16T04:51:50.659214+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:50.659214+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6709248,"num_objects":819,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11835,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11885","reported_seq":11978,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:49.882863+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:49.882863+0000","last_peered":"2024-09-16T04:51:49.882863+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:49.882863+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6460684,"num_objects":789,"num_object_clones":0,"num_object_copies":3156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11883,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1414}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11855","reported_seq":11957,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:48.800344+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:48.800344+0000","last_peered":"2024-09-16T04:51:48.800344+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:48.800344+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6675102,"num_objects":816,"num_object_clones":0,"num_object_copies":3264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11854,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1403}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11802","reported_seq":16352,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:48.065394+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:48.065394+0000","last_peered":"2024-09-16T04:51:48.065394+0000","last_clean":"2024-09-16T04:51:48.065394+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:48.065394+0000","last_undegraded":"2024-09-16T04:51:48.065394+0000","last_fullsized":"2024-09-16T04:51:48.065394+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:51.736335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7107852,"num_objects":868,"num_object_clones":0,"num_object_copies":3472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11800,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11921","reported_seq":12059,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.787867+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:50.787867+0000","last_peered":"2024-09-16T04:51:50.787867+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:50.787867+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6714636,"num_objects":820,"num_object_clones":0,"num_object_copies":3280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11919,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1428}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11534","reported_seq":11648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.809592+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:50.809592+0000","last_peered":"2024-09-16T04:51:50.809592+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:50.809592+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6373376,"num_objects":778,"num_object_clones":0,"num_object_copies":3112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11534,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1400}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11796","reported_seq":11899,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.208101+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:50.208101+0000","last_peered":"2024-09-16T04:51:50.208101+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:50.208101+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6583564,"num_objects":804,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11794,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1409}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11910","reported_seq":16477,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:48.107838+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:48.107838+0000","last_peered":"2024-09-16T04:51:48.107838+0000","last_clean":"2024-09-16T04:51:48.107838+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:48.107838+0000","last_undegraded":"2024-09-16T04:51:48.107838+0000","last_fullsized":"2024-09-16T04:51:48.107838+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:51.736312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7165196,"num_objects":875,"num_object_clones":0,"num_object_copies":3500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11908,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11917","reported_seq":16680,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:50.610956+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:50.610956+0000","last_peered":"2024-09-16T04:51:50.610956+0000","last_clean":"2024-09-16T04:51:50.610956+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:50.610956+0000","last_undegraded":"2024-09-16T04:51:50.610956+0000","last_fullsized":"2024-09-16T04:51:50.610956+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:52.877155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6867480,"num_objects":839,"num_object_clones":0,"num_object_copies":3356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11914,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11895","reported_seq":12007,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:49.066365+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:49.066365+0000","last_peered":"2024-09-16T04:51:49.066365+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:49.066365+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7066892,"num_objects":863,"num_object_clones":0,"num_object_copies":3452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11893,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1326}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11691","reported_seq":11828,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.840605+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:50.840605+0000","last_peered":"2024-09-16T04:51:50.840605+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:50.840605+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6549394,"num_objects":800,"num_object_clones":0,"num_object_copies":3200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11688,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1355}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11548","reported_seq":11646,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.652706+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:50.652706+0000","last_peered":"2024-09-16T04:51:50.652706+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:50.652706+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6412934,"num_objects":783,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1322}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11721","reported_seq":11815,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:49.036267+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:49.036267+0000","last_peered":"2024-09-16T04:51:49.036267+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:49.036267+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6650502,"num_objects":812,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1409}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11965","reported_seq":12102,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.776138+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:50.776138+0000","last_peered":"2024-09-16T04:51:50.776138+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:50.776138+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6690060,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1381}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11618","reported_seq":12640,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:49.163944+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:49.163944+0000","last_peered":"2024-09-16T04:51:49.163944+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:49.163944+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6804748,"num_objects":831,"num_object_clones":0,"num_object_copies":3324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11616,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":467,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-838}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11503","reported_seq":11603,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.002434+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:50.002434+0000","last_peered":"2024-09-16T04:51:50.002434+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:50.002434+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6583564,"num_objects":804,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11501,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107415232,"num_objects":13118,"num_object_clones":0,"num_object_copies":52472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13118,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188369,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7552,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216240128,"data_stored":216240128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160696,"ondisk_log_size":160696,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716204,"kb_used_data":52812,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93655636,"statfs":{"total":96636764160,"available":95903371264,"internally_reserved":0,"allocated":54079488,"data_stored":53880979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":679276239},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,8,9,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742912,"kb_used_data":53672,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93628928,"statfs":{"total":96636764160,"available":95876022272,"internally_reserved":0,"allocated":54960128,"data_stored":54761619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591972,"kb_used_data":54924,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93779868,"statfs":{"total":96636764160,"available":96030584832,"internally_reserved":0,"allocated":56242176,"data_stored":56043667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549908175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720248,"kb_used_data":52776,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93651592,"statfs":{"total":96636764160,"available":95899230208,"internally_reserved":0,"allocated":54042624,"data_stored":53844115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":683470542},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53420032,"data_stored":53420032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55582720,"data_stored":55582720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54251520,"data_stored":54251520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":52985856,"data_stored":52985856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:54.147 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:54.774 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:54.774 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:54.802 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-09-16T04:51:52.777268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107415232,"num_objects":13118,"num_object_clones":0,"num_object_copies":52472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13118,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188369,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7552,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160696,"ondisk_log_size":160696,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771336,"kb_used_data":214184,"kb_used_omap":14,"kb_used_meta":2557041,"kb_avail":374716024,"statfs":{"total":386547056640,"available":383709208576,"internally_reserved":0,"allocated":219324416,"data_stored":218530380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2618410545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,6,6,16,23,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15958016,"num_objects":-1948,"num_object_clones":0,"num_object_copies":-7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1948,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004727"},"pg_stats":[{"pgid":"1.f","version":"37'11835","reported_seq":11956,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.659214+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:50.659214+0000","last_peered":"2024-09-16T04:51:50.659214+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:50.659214+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6709248,"num_objects":819,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11835,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11885","reported_seq":11978,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:49.882863+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:49.882863+0000","last_peered":"2024-09-16T04:51:49.882863+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:49.882863+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6460684,"num_objects":789,"num_object_clones":0,"num_object_copies":3156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11883,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1414}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11855","reported_seq":11957,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:48.800344+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:48.800344+0000","last_peered":"2024-09-16T04:51:48.800344+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:48.800344+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6675102,"num_objects":816,"num_object_clones":0,"num_object_copies":3264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11854,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1403}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11802","reported_seq":16352,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:48.065394+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:48.065394+0000","last_peered":"2024-09-16T04:51:48.065394+0000","last_clean":"2024-09-16T04:51:48.065394+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:48.065394+0000","last_undegraded":"2024-09-16T04:51:48.065394+0000","last_fullsized":"2024-09-16T04:51:48.065394+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:51.736335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7107852,"num_objects":868,"num_object_clones":0,"num_object_copies":3472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11800,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11921","reported_seq":12059,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.787867+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:50.787867+0000","last_peered":"2024-09-16T04:51:50.787867+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:50.787867+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6714636,"num_objects":820,"num_object_clones":0,"num_object_copies":3280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11919,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1428}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11534","reported_seq":11648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.809592+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:50.809592+0000","last_peered":"2024-09-16T04:51:50.809592+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:50.809592+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6373376,"num_objects":778,"num_object_clones":0,"num_object_copies":3112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11534,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1400}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11796","reported_seq":11899,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.208101+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:50.208101+0000","last_peered":"2024-09-16T04:51:50.208101+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:50.208101+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6583564,"num_objects":804,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11794,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1409}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11910","reported_seq":16477,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:48.107838+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:48.107838+0000","last_peered":"2024-09-16T04:51:48.107838+0000","last_clean":"2024-09-16T04:51:48.107838+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:48.107838+0000","last_undegraded":"2024-09-16T04:51:48.107838+0000","last_fullsized":"2024-09-16T04:51:48.107838+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:51.736312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7165196,"num_objects":875,"num_object_clones":0,"num_object_copies":3500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11908,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11917","reported_seq":16680,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:50.610956+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:50.610956+0000","last_peered":"2024-09-16T04:51:50.610956+0000","last_clean":"2024-09-16T04:51:50.610956+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:50.610956+0000","last_undegraded":"2024-09-16T04:51:50.610956+0000","last_fullsized":"2024-09-16T04:51:50.610956+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:52.877155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6867480,"num_objects":839,"num_object_clones":0,"num_object_copies":3356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11914,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11895","reported_seq":12007,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:49.066365+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:49.066365+0000","last_peered":"2024-09-16T04:51:49.066365+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:49.066365+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7066892,"num_objects":863,"num_object_clones":0,"num_object_copies":3452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11893,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1326}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11691","reported_seq":11828,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.840605+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:50.840605+0000","last_peered":"2024-09-16T04:51:50.840605+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:50.840605+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6549394,"num_objects":800,"num_object_clones":0,"num_object_copies":3200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11688,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1355}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11548","reported_seq":11646,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.652706+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:50.652706+0000","last_peered":"2024-09-16T04:51:50.652706+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:50.652706+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6412934,"num_objects":783,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1322}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11721","reported_seq":11815,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:49.036267+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:49.036267+0000","last_peered":"2024-09-16T04:51:49.036267+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:49.036267+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6650502,"num_objects":812,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1409}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11965","reported_seq":12102,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.776138+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:50.776138+0000","last_peered":"2024-09-16T04:51:50.776138+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:50.776138+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6690060,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1381}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11618","reported_seq":12640,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:49.163944+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:49.163944+0000","last_peered":"2024-09-16T04:51:49.163944+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:49.163944+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6804748,"num_objects":831,"num_object_clones":0,"num_object_copies":3324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11616,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":467,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-838}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11503","reported_seq":11603,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.002434+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:50.002434+0000","last_peered":"2024-09-16T04:51:50.002434+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:50.002434+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6583564,"num_objects":804,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11501,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107415232,"num_objects":13118,"num_object_clones":0,"num_object_copies":52472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13118,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188369,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7552,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216240128,"data_stored":216240128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160696,"ondisk_log_size":160696,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716204,"kb_used_data":52812,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93655636,"statfs":{"total":96636764160,"available":95903371264,"internally_reserved":0,"allocated":54079488,"data_stored":53880979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":679276239},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,8,9,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742912,"kb_used_data":53672,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93628928,"statfs":{"total":96636764160,"available":95876022272,"internally_reserved":0,"allocated":54960128,"data_stored":54761619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591972,"kb_used_data":54924,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93779868,"statfs":{"total":96636764160,"available":96030584832,"internally_reserved":0,"allocated":56242176,"data_stored":56043667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549908175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720248,"kb_used_data":52776,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93651592,"statfs":{"total":96636764160,"available":95899230208,"internally_reserved":0,"allocated":54042624,"data_stored":53844115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":683470542},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53420032,"data_stored":53420032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55582720,"data_stored":55582720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54251520,"data_stored":54251520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":52985856,"data_stored":52985856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:54.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:51:54.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11835", 'reported_seq': 11956, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:50.659214+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:51:50.659214+0000', 'last_peered': '2024-09-16T04:51:50.659214+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:51:50.659214+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6709248, 'num_objects': 819, 'num_object_clones': 0, 'num_object_copies': 3276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11835, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:54.803 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:51:54.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11885", 'reported_seq': 11978, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:49.882863+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:51:49.882863+0000', 'last_peered': '2024-09-16T04:51:49.882863+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:51:49.882863+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6460684, 'num_objects': 789, 'num_object_clones': 0, 'num_object_copies': 3156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 789, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11883, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1414}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:54.804 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:51:54.804 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11855", 'reported_seq': 11957, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:48.800344+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:51:48.800344+0000', 'last_peered': '2024-09-16T04:51:48.800344+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:51:48.800344+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6675102, 'num_objects': 816, 'num_object_clones': 0, 'num_object_copies': 3264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 816, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11854, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1403}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:54.804 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:51:54.804 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11921", 'reported_seq': 12059, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:50.787867+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:51:50.787867+0000', 'last_peered': '2024-09-16T04:51:50.787867+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:51:50.787867+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6714636, 'num_objects': 820, 'num_object_clones': 0, 'num_object_copies': 3280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 820, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11919, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1428}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:54.804 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:51:54.804 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11534", 'reported_seq': 11648, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:50.809592+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:51:50.809592+0000', 'last_peered': '2024-09-16T04:51:50.809592+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:51:50.809592+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6373376, 'num_objects': 778, 'num_object_clones': 0, 'num_object_copies': 3112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 778, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11534, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1400}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:54.804 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:51:54.804 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11796", 'reported_seq': 11899, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:50.208101+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:51:50.208101+0000', 'last_peered': '2024-09-16T04:51:50.208101+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:51:50.208101+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6583564, 'num_objects': 804, 'num_object_clones': 0, 'num_object_copies': 3216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11794, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1409}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:54.804 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:51:54.804 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11895", 'reported_seq': 12007, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:49.066365+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:51:49.066365+0000', 'last_peered': '2024-09-16T04:51:49.066365+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:51:49.066365+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7066892, 'num_objects': 863, 'num_object_clones': 0, 'num_object_copies': 3452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11893, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1326}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:54.805 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:51:54.805 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11691", 'reported_seq': 11828, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:50.840605+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:51:50.840605+0000', 'last_peered': '2024-09-16T04:51:50.840605+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:51:50.840605+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6549394, 'num_objects': 800, 'num_object_clones': 0, 'num_object_copies': 3200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 800, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11688, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1355}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:54.805 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:51:54.805 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11548", 'reported_seq': 11646, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:50.652706+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:51:50.652706+0000', 'last_peered': '2024-09-16T04:51:50.652706+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:51:50.652706+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6412934, 'num_objects': 783, 'num_object_clones': 0, 'num_object_copies': 3132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11547, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1322}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:54.805 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:51:54.805 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11721", 'reported_seq': 11815, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:49.036267+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:51:49.036267+0000', 'last_peered': '2024-09-16T04:51:49.036267+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:51:49.036267+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6650502, 'num_objects': 812, 'num_object_clones': 0, 'num_object_copies': 3248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 812, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11720, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1409}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:54.805 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:51:54.805 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11965", 'reported_seq': 12102, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:50.776138+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:51:50.776138+0000', 'last_peered': '2024-09-16T04:51:50.776138+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:51:50.776138+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6690060, 'num_objects': 817, 'num_object_clones': 0, 'num_object_copies': 3268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11963, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1381}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:51:54.805 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:51:54.805 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11618", 'reported_seq': 12640, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:49.163944+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:51:49.163944+0000', 'last_peered': '2024-09-16T04:51:49.163944+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:51:49.163944+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6804748, 'num_objects': 831, 'num_object_clones': 0, 'num_object_copies': 3324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1669, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11616, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 467, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -838}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:51:54.805 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:51:54.806 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11503", 'reported_seq': 11603, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:50.002434+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:51:50.002434+0000', 'last_peered': '2024-09-16T04:51:50.002434+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:51:50.002434+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6583564, 'num_objects': 804, 'num_object_clones': 0, 'num_object_copies': 3216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11501, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1424}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:51:54.806 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:51:55.565 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:55.565 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27977/50972 objects degraded (54.887%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":727,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":12743,"data_bytes":104343232,"bytes_used":2839982080,"bytes_avail":383707074560,"bytes_total":386547056640,"degraded_objects":27977,"degraded_total":50972,"degraded_ratio":0.54886996782547282,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":193},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:51:55.577 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:51:55.577 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:56.140 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:56.141 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:56.180 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-09-16T04:51:54.778127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":104343232,"num_objects":12743,"num_object_clones":0,"num_object_copies":50972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188744,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7605,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773420,"kb_used_data":206548,"kb_used_omap":14,"kb_used_meta":2566769,"kb_avail":374713940,"statfs":{"total":386547056640,"available":383707074560,"internally_reserved":0,"allocated":211505152,"data_stored":210711116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2628372017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,9,8,17,20,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19030016,"num_objects":-2323,"num_object_clones":0,"num_object_copies":-9292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2323,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004763"},"pg_stats":[{"pgid":"1.f","version":"37'11835","reported_seq":11956,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.659214+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:50.659214+0000","last_peered":"2024-09-16T04:51:50.659214+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:50.659214+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6709248,"num_objects":819,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11835,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11885","reported_seq":11978,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:49.882863+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:49.882863+0000","last_peered":"2024-09-16T04:51:49.882863+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:49.882863+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6460684,"num_objects":789,"num_object_clones":0,"num_object_copies":3156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11883,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1414}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11919","reported_seq":12023,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:54.309191+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:54.309191+0000","last_peered":"2024-09-16T04:51:54.309191+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:54.309191+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6150814,"num_objects":752,"num_object_clones":0,"num_object_copies":3008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11918,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1467}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11864","reported_seq":16414,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:53.138558+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:53.138558+0000","last_peered":"2024-09-16T04:51:53.138558+0000","last_clean":"2024-09-16T04:51:53.138558+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:53.138558+0000","last_undegraded":"2024-09-16T04:51:53.138558+0000","last_fullsized":"2024-09-16T04:51:53.138558+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:57.588832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6599948,"num_objects":806,"num_object_clones":0,"num_object_copies":3224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11862,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11921","reported_seq":12059,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.787867+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:50.787867+0000","last_peered":"2024-09-16T04:51:50.787867+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:50.787867+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6714636,"num_objects":820,"num_object_clones":0,"num_object_copies":3280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11919,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1428}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11534","reported_seq":11648,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.809592+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:50.809592+0000","last_peered":"2024-09-16T04:51:50.809592+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:50.809592+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6373376,"num_objects":778,"num_object_clones":0,"num_object_copies":3112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11534,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1400}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11796","reported_seq":11899,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.208101+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:50.208101+0000","last_peered":"2024-09-16T04:51:50.208101+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:50.208101+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6583564,"num_objects":804,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11794,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1409}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11974","reported_seq":16541,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:52.933170+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:52.933170+0000","last_peered":"2024-09-16T04:51:52.933170+0000","last_clean":"2024-09-16T04:51:52.933170+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:52.933170+0000","last_undegraded":"2024-09-16T04:51:52.933170+0000","last_fullsized":"2024-09-16T04:51:52.933170+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:57.588802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6640908,"num_objects":811,"num_object_clones":0,"num_object_copies":3244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11972,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11917","reported_seq":16680,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:50.610956+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:50.610956+0000","last_peered":"2024-09-16T04:51:50.610956+0000","last_clean":"2024-09-16T04:51:50.610956+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:50.610956+0000","last_undegraded":"2024-09-16T04:51:50.610956+0000","last_fullsized":"2024-09-16T04:51:50.610956+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:52.877155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6867480,"num_objects":839,"num_object_clones":0,"num_object_copies":3356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11914,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11959","reported_seq":12071,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:54.324660+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:54.324660+0000","last_peered":"2024-09-16T04:51:54.324660+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:54.324660+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6542604,"num_objects":799,"num_object_clones":0,"num_object_copies":3196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11957,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1390}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11691","reported_seq":11828,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.840605+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:50.840605+0000","last_peered":"2024-09-16T04:51:50.840605+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:50.840605+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6549394,"num_objects":800,"num_object_clones":0,"num_object_copies":3200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11688,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1355}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11548","reported_seq":11646,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.652706+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:50.652706+0000","last_peered":"2024-09-16T04:51:50.652706+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:50.652706+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6412934,"num_objects":783,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1322}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11770","reported_seq":11864,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:54.158911+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:54.158911+0000","last_peered":"2024-09-16T04:51:54.158911+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:54.158911+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6249094,"num_objects":763,"num_object_clones":0,"num_object_copies":3052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11769,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1458}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'11965","reported_seq":12102,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.776138+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:50.776138+0000","last_peered":"2024-09-16T04:51:50.776138+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:50.776138+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6690060,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1381}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11690","reported_seq":12817,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:54.336849+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:54.336849+0000","last_peered":"2024-09-16T04:51:54.336849+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:54.336849+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6214924,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11688,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-857}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11503","reported_seq":11603,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:50.002434+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:50.002434+0000","last_peered":"2024-09-16T04:51:50.002434+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:50.002434+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6583564,"num_objects":804,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11501,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":104343232,"num_objects":12743,"num_object_clones":0,"num_object_copies":50972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188744,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7605,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208203776,"data_stored":208203776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716204,"kb_used_data":52812,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93655636,"statfs":{"total":96636764160,"available":95903371264,"internally_reserved":0,"allocated":54079488,"data_stored":53880979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":679276239},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,8,9,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744536,"kb_used_data":49728,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93627304,"statfs":{"total":96636764160,"available":95874359296,"internally_reserved":0,"allocated":50921472,"data_stored":50722963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,5,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592432,"kb_used_data":51232,"kb_used_omap":4,"kb_used_meta":541179,"kb_avail":93779408,"statfs":{"total":96636764160,"available":96030113792,"internally_reserved":0,"allocated":52461568,"data_stored":52263059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554168015},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720248,"kb_used_data":52776,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93651592,"statfs":{"total":96636764160,"available":95899230208,"internally_reserved":0,"allocated":54042624,"data_stored":53844115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":683470542},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53420032,"data_stored":53420032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51625984,"data_stored":51625984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50171904,"data_stored":50171904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":52985856,"data_stored":52985856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:59.182 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:51:59.785 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:51:59.785 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:51:59.813 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-09-16T04:51:58.779686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98346688,"num_objects":12011,"num_object_clones":0,"num_object_copies":48044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189476,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7605,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160903,"ondisk_log_size":160903,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2809740,"kb_used_data":194288,"kb_used_omap":14,"kb_used_meta":2615345,"kb_avail":374677620,"statfs":{"total":386547056640,"available":383669882880,"internally_reserved":0,"allocated":198950912,"data_stored":198156876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2678113841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,3,8,10,15,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16949248,"num_objects":-2069,"num_object_clones":0,"num_object_copies":-8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2069,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004640"},"pg_stats":[{"pgid":"1.f","version":"37'11899","reported_seq":12020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.678246+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:55.678246+0000","last_peered":"2024-09-16T04:51:55.678246+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:55.678246+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6184960,"num_objects":755,"num_object_clones":0,"num_object_copies":3020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11899,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1431}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11935","reported_seq":12028,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.282658+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:55.282658+0000","last_peered":"2024-09-16T04:51:55.282658+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:55.282658+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6051084,"num_objects":739,"num_object_clones":0,"num_object_copies":2956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11933,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11919","reported_seq":12023,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:54.309191+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:54.309191+0000","last_peered":"2024-09-16T04:51:54.309191+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:54.309191+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6150814,"num_objects":752,"num_object_clones":0,"num_object_copies":3008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11918,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1467}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11928","reported_seq":16478,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:58.110465+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:58.110465+0000","last_peered":"2024-09-16T04:51:58.110465+0000","last_clean":"2024-09-16T04:51:58.110465+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:58.110465+0000","last_undegraded":"2024-09-16T04:51:58.110465+0000","last_fullsized":"2024-09-16T04:51:58.110465+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:02.731019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6075660,"num_objects":742,"num_object_clones":0,"num_object_copies":2968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11926,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11993","reported_seq":12130,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.804388+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:55.804388+0000","last_peered":"2024-09-16T04:51:55.804388+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:55.804388+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6124812,"num_objects":748,"num_object_clones":0,"num_object_copies":2992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11991,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1500}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11584","reported_seq":11699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.809809+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:55.809809+0000","last_peered":"2024-09-16T04:51:55.809809+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:55.809809+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5963776,"num_objects":728,"num_object_clones":0,"num_object_copies":2912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1450}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11865","reported_seq":11967,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.401798+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:55.401798+0000","last_peered":"2024-09-16T04:51:55.401798+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:55.401798+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6018316,"num_objects":735,"num_object_clones":0,"num_object_copies":2940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1478}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12041","reported_seq":16608,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:58.114494+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:58.114494+0000","last_peered":"2024-09-16T04:51:58.114494+0000","last_clean":"2024-09-16T04:51:58.114494+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:58.114494+0000","last_undegraded":"2024-09-16T04:51:58.114494+0000","last_fullsized":"2024-09-16T04:51:58.114494+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:02.717238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6092044,"num_objects":744,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12039,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11974","reported_seq":16737,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:55.651143+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:55.651143+0000","last_peered":"2024-09-16T04:51:55.651143+0000","last_clean":"2024-09-16T04:51:55.651143+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:55.651143+0000","last_undegraded":"2024-09-16T04:51:55.651143+0000","last_fullsized":"2024-09-16T04:51:55.651143+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:58.806017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6400536,"num_objects":782,"num_object_clones":0,"num_object_copies":3128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11971,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11959","reported_seq":12071,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:54.324660+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:54.324660+0000","last_peered":"2024-09-16T04:51:54.324660+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:54.324660+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6542604,"num_objects":799,"num_object_clones":0,"num_object_copies":3196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11957,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1390}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11752","reported_seq":11890,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.835345+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:55.835345+0000","last_peered":"2024-09-16T04:51:55.835345+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:55.835345+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6049682,"num_objects":739,"num_object_clones":0,"num_object_copies":2956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11749,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1416}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11611","reported_seq":11709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.870222+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:55.870222+0000","last_peered":"2024-09-16T04:51:55.870222+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:55.870222+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5896838,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11610,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1385}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11770","reported_seq":11864,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:54.158911+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:54.158911+0000","last_peered":"2024-09-16T04:51:54.158911+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:54.158911+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6249094,"num_objects":763,"num_object_clones":0,"num_object_copies":3052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11769,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1458}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12024","reported_seq":12162,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.606652+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:55.606652+0000","last_peered":"2024-09-16T04:51:55.606652+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:55.606652+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6206732,"num_objects":758,"num_object_clones":0,"num_object_copies":3032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1440}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11690","reported_seq":12817,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:54.336849+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:54.336849+0000","last_peered":"2024-09-16T04:51:54.336849+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:54.336849+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6214924,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11688,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-857}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11559","reported_seq":11659,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.024674+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:55.024674+0000","last_peered":"2024-09-16T04:51:55.024674+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:55.024674+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6124812,"num_objects":748,"num_object_clones":0,"num_object_copies":2992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11557,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1480}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":98346688,"num_objects":12011,"num_object_clones":0,"num_object_copies":48044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189476,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7605,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196296704,"data_stored":196296704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160903,"ondisk_log_size":160903,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733904,"kb_used_data":48304,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93637936,"statfs":{"total":96636764160,"available":95885246464,"internally_reserved":0,"allocated":49463296,"data_stored":49264787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":702017231},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,6,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744536,"kb_used_data":49728,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93627304,"statfs":{"total":96636764160,"available":95874359296,"internally_reserved":0,"allocated":50921472,"data_stored":50722963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,5,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592712,"kb_used_data":47348,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93779128,"statfs":{"total":96636764160,"available":96029827072,"internally_reserved":0,"allocated":48484352,"data_stored":48285843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558427855},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247948,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738588,"kb_used_data":48908,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93633252,"statfs":{"total":96636764160,"available":95880450048,"internally_reserved":0,"allocated":50081792,"data_stored":49883283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":706211534},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49373184,"data_stored":49373184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47689728,"data_stored":47689728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50171904,"data_stored":50171904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49061888,"data_stored":49061888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:51:59.814 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:00.436 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:00.437 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:00.463 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-09-16T04:51:58.779686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98346688,"num_objects":12011,"num_object_clones":0,"num_object_copies":48044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189476,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7605,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160903,"ondisk_log_size":160903,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2809740,"kb_used_data":194288,"kb_used_omap":14,"kb_used_meta":2615345,"kb_avail":374677620,"statfs":{"total":386547056640,"available":383669882880,"internally_reserved":0,"allocated":198950912,"data_stored":198156876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2678113841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,3,8,10,15,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16949248,"num_objects":-2069,"num_object_clones":0,"num_object_copies":-8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2069,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004640"},"pg_stats":[{"pgid":"1.f","version":"37'11899","reported_seq":12020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.678246+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:55.678246+0000","last_peered":"2024-09-16T04:51:55.678246+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:55.678246+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6184960,"num_objects":755,"num_object_clones":0,"num_object_copies":3020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11899,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1431}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11935","reported_seq":12028,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.282658+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:51:55.282658+0000","last_peered":"2024-09-16T04:51:55.282658+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:51:55.282658+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6051084,"num_objects":739,"num_object_clones":0,"num_object_copies":2956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11933,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1464}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11919","reported_seq":12023,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:54.309191+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:54.309191+0000","last_peered":"2024-09-16T04:51:54.309191+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:54.309191+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6150814,"num_objects":752,"num_object_clones":0,"num_object_copies":3008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11918,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1467}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11928","reported_seq":16478,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:58.110465+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:58.110465+0000","last_peered":"2024-09-16T04:51:58.110465+0000","last_clean":"2024-09-16T04:51:58.110465+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:58.110465+0000","last_undegraded":"2024-09-16T04:51:58.110465+0000","last_fullsized":"2024-09-16T04:51:58.110465+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:02.731019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6075660,"num_objects":742,"num_object_clones":0,"num_object_copies":2968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11926,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11993","reported_seq":12130,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.804388+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:55.804388+0000","last_peered":"2024-09-16T04:51:55.804388+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:55.804388+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6124812,"num_objects":748,"num_object_clones":0,"num_object_copies":2992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11991,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1500}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11584","reported_seq":11699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.809809+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:55.809809+0000","last_peered":"2024-09-16T04:51:55.809809+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:55.809809+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5963776,"num_objects":728,"num_object_clones":0,"num_object_copies":2912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1450}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11865","reported_seq":11967,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.401798+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:51:55.401798+0000","last_peered":"2024-09-16T04:51:55.401798+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:51:55.401798+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6018316,"num_objects":735,"num_object_clones":0,"num_object_copies":2940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1478}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12041","reported_seq":16608,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:58.114494+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:58.114494+0000","last_peered":"2024-09-16T04:51:58.114494+0000","last_clean":"2024-09-16T04:51:58.114494+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:58.114494+0000","last_undegraded":"2024-09-16T04:51:58.114494+0000","last_fullsized":"2024-09-16T04:51:58.114494+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:02.717238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6092044,"num_objects":744,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12039,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11974","reported_seq":16737,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:55.651143+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:55.651143+0000","last_peered":"2024-09-16T04:51:55.651143+0000","last_clean":"2024-09-16T04:51:55.651143+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:55.651143+0000","last_undegraded":"2024-09-16T04:51:55.651143+0000","last_fullsized":"2024-09-16T04:51:55.651143+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:58.806017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6400536,"num_objects":782,"num_object_clones":0,"num_object_copies":3128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11971,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11959","reported_seq":12071,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:54.324660+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:54.324660+0000","last_peered":"2024-09-16T04:51:54.324660+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:54.324660+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6542604,"num_objects":799,"num_object_clones":0,"num_object_copies":3196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11957,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1390}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11752","reported_seq":11890,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.835345+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:55.835345+0000","last_peered":"2024-09-16T04:51:55.835345+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:55.835345+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6049682,"num_objects":739,"num_object_clones":0,"num_object_copies":2956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11749,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1416}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11611","reported_seq":11709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.870222+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:55.870222+0000","last_peered":"2024-09-16T04:51:55.870222+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:55.870222+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5896838,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11610,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1385}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11770","reported_seq":11864,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:54.158911+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:54.158911+0000","last_peered":"2024-09-16T04:51:54.158911+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:54.158911+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6249094,"num_objects":763,"num_object_clones":0,"num_object_copies":3052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11769,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1458}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12024","reported_seq":12162,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.606652+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:55.606652+0000","last_peered":"2024-09-16T04:51:55.606652+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:55.606652+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6206732,"num_objects":758,"num_object_clones":0,"num_object_copies":3032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1440}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11690","reported_seq":12817,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:54.336849+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:54.336849+0000","last_peered":"2024-09-16T04:51:54.336849+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:54.336849+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6214924,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11688,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-857}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11559","reported_seq":11659,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.024674+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:51:55.024674+0000","last_peered":"2024-09-16T04:51:55.024674+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:51:55.024674+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6124812,"num_objects":748,"num_object_clones":0,"num_object_copies":2992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11557,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1480}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":98346688,"num_objects":12011,"num_object_clones":0,"num_object_copies":48044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189476,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7605,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196296704,"data_stored":196296704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160903,"ondisk_log_size":160903,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733904,"kb_used_data":48304,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93637936,"statfs":{"total":96636764160,"available":95885246464,"internally_reserved":0,"allocated":49463296,"data_stored":49264787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":702017231},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,6,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744536,"kb_used_data":49728,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93627304,"statfs":{"total":96636764160,"available":95874359296,"internally_reserved":0,"allocated":50921472,"data_stored":50722963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,5,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592712,"kb_used_data":47348,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93779128,"statfs":{"total":96636764160,"available":96029827072,"internally_reserved":0,"allocated":48484352,"data_stored":48285843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558427855},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247948,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738588,"kb_used_data":48908,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93633252,"statfs":{"total":96636764160,"available":95880450048,"internally_reserved":0,"allocated":50081792,"data_stored":49883283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":706211534},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49373184,"data_stored":49373184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47689728,"data_stored":47689728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50171904,"data_stored":50171904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49061888,"data_stored":49061888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:00.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:52:00.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11899", 'reported_seq': 12020, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:55.678246+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:51:55.678246+0000', 'last_peered': '2024-09-16T04:51:55.678246+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:51:55.678246+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6184960, 'num_objects': 755, 'num_object_clones': 0, 'num_object_copies': 3020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11899, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1431}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:00.465 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:52:00.465 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11935", 'reported_seq': 12028, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:55.282658+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:51:55.282658+0000', 'last_peered': '2024-09-16T04:51:55.282658+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:51:55.282658+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6051084, 'num_objects': 739, 'num_object_clones': 0, 'num_object_copies': 2956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 739, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11933, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1464}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:00.465 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:52:00.465 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11919", 'reported_seq': 12023, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:54.309191+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:51:54.309191+0000', 'last_peered': '2024-09-16T04:51:54.309191+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:51:54.309191+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6150814, 'num_objects': 752, 'num_object_clones': 0, 'num_object_copies': 3008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 752, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11918, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1467}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:00.465 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:52:00.465 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11993", 'reported_seq': 12130, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:55.804388+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:51:55.804388+0000', 'last_peered': '2024-09-16T04:51:55.804388+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:51:55.804388+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6124812, 'num_objects': 748, 'num_object_clones': 0, 'num_object_copies': 2992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11991, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1500}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:00.465 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:52:00.465 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11584", 'reported_seq': 11699, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:55.809809+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:51:55.809809+0000', 'last_peered': '2024-09-16T04:51:55.809809+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:51:55.809809+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5963776, 'num_objects': 728, 'num_object_clones': 0, 'num_object_copies': 2912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 728, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11584, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1450}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:00.465 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:52:00.466 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11865", 'reported_seq': 11967, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:55.401798+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:51:55.401798+0000', 'last_peered': '2024-09-16T04:51:55.401798+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:51:55.401798+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6018316, 'num_objects': 735, 'num_object_clones': 0, 'num_object_copies': 2940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 735, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11863, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1478}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:00.466 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:52:00.466 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11959", 'reported_seq': 12071, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:54.324660+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:51:54.324660+0000', 'last_peered': '2024-09-16T04:51:54.324660+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:51:54.324660+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6542604, 'num_objects': 799, 'num_object_clones': 0, 'num_object_copies': 3196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 799, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11957, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1390}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:00.466 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:52:00.466 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11752", 'reported_seq': 11890, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:55.835345+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:51:55.835345+0000', 'last_peered': '2024-09-16T04:51:55.835345+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:51:55.835345+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6049682, 'num_objects': 739, 'num_object_clones': 0, 'num_object_copies': 2956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 739, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11749, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1416}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:00.466 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:52:00.466 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11611", 'reported_seq': 11709, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:55.870222+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:51:55.870222+0000', 'last_peered': '2024-09-16T04:51:55.870222+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:51:55.870222+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5896838, 'num_objects': 720, 'num_object_clones': 0, 'num_object_copies': 2880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11610, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1385}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:00.466 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:52:00.466 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11770", 'reported_seq': 11864, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:54.158911+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:51:54.158911+0000', 'last_peered': '2024-09-16T04:51:54.158911+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:51:54.158911+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6249094, 'num_objects': 763, 'num_object_clones': 0, 'num_object_copies': 3052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 763, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11769, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1458}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:00.466 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:52:00.466 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12024", 'reported_seq': 12162, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:55.606652+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:51:55.606652+0000', 'last_peered': '2024-09-16T04:51:55.606652+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:51:55.606652+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6206732, 'num_objects': 758, 'num_object_clones': 0, 'num_object_copies': 3032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 758, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12022, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1440}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:00.466 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:52:00.467 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11690", 'reported_seq': 12817, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:54.336849+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:51:54.336849+0000', 'last_peered': '2024-09-16T04:51:54.336849+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:51:54.336849+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6214924, 'num_objects': 759, 'num_object_clones': 0, 'num_object_copies': 3036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1616, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11688, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 520, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -857}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:00.467 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:52:00.467 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11559", 'reported_seq': 11659, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:51:55.024674+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:51:55.024674+0000', 'last_peered': '2024-09-16T04:51:55.024674+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:51:55.024674+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6124812, 'num_objects': 748, 'num_object_clones': 0, 'num_object_copies': 2992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11557, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1480}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:00.467 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:52:01.230 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:01.230 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27924/46372 objects degraded (60.217%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":733,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":11593,"data_bytes":94922432,"bytes_used":2873585664,"bytes_avail":383673470976,"bytes_total":386547056640,"degraded_objects":27924,"degraded_total":46372,"degraded_ratio":0.60217372552402315,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":166,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":195},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:52:01.258 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:52:01.258 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:01.861 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:01.861 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:01.883 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-09-16T04:52:00.780259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94922432,"num_objects":11593,"num_object_clones":0,"num_object_copies":46372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11593,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189894,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7658,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160721,"ondisk_log_size":160721,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2806236,"kb_used_data":186628,"kb_used_omap":14,"kb_used_meta":2619505,"kb_avail":374681124,"statfs":{"total":386547056640,"available":383673470976,"internally_reserved":0,"allocated":191107072,"data_stored":190313036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2682373681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,8,7,12,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19259392,"num_objects":-2351,"num_object_clones":0,"num_object_copies":-9404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2351,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004501"},"pg_stats":[{"pgid":"1.f","version":"37'11899","reported_seq":12020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.678246+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:51:55.678246+0000","last_peered":"2024-09-16T04:51:55.678246+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:51:55.678246+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6184960,"num_objects":755,"num_object_clones":0,"num_object_copies":3020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11899,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1431}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12003","reported_seq":12096,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.218965+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:00.218965+0000","last_peered":"2024-09-16T04:52:00.218965+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:00.218965+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5494028,"num_objects":671,"num_object_clones":0,"num_object_copies":2684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12001,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1532}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11979","reported_seq":12083,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:59.088594+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:51:59.088594+0000","last_peered":"2024-09-16T04:51:59.088594+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:51:59.088594+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5659294,"num_objects":692,"num_object_clones":0,"num_object_copies":2768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":692,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11978,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1527}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11928","reported_seq":16478,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:58.110465+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:51:58.110465+0000","last_peered":"2024-09-16T04:51:58.110465+0000","last_clean":"2024-09-16T04:51:58.110465+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:51:58.110465+0000","last_undegraded":"2024-09-16T04:51:58.110465+0000","last_fullsized":"2024-09-16T04:51:58.110465+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:02.731019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6075660,"num_objects":742,"num_object_clones":0,"num_object_copies":2968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11926,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11993","reported_seq":12130,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.804388+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:51:55.804388+0000","last_peered":"2024-09-16T04:51:55.804388+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:51:55.804388+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6124812,"num_objects":748,"num_object_clones":0,"num_object_copies":2992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11991,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1500}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11584","reported_seq":11699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.809809+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:51:55.809809+0000","last_peered":"2024-09-16T04:51:55.809809+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:51:55.809809+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5963776,"num_objects":728,"num_object_clones":0,"num_object_copies":2912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1450}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11935","reported_seq":12038,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.363185+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:00.363185+0000","last_peered":"2024-09-16T04:52:00.363185+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:00.363185+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5444876,"num_objects":665,"num_object_clones":0,"num_object_copies":2660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11933,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1548}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12041","reported_seq":16608,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:58.114494+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:51:58.114494+0000","last_peered":"2024-09-16T04:51:58.114494+0000","last_clean":"2024-09-16T04:51:58.114494+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:51:58.114494+0000","last_undegraded":"2024-09-16T04:51:58.114494+0000","last_fullsized":"2024-09-16T04:51:58.114494+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:02.717238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6092044,"num_objects":744,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12039,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11974","reported_seq":16737,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:51:55.651143+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:51:55.651143+0000","last_peered":"2024-09-16T04:51:55.651143+0000","last_clean":"2024-09-16T04:51:55.651143+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:51:55.651143+0000","last_undegraded":"2024-09-16T04:51:55.651143+0000","last_fullsized":"2024-09-16T04:51:55.651143+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:58.806017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6400536,"num_objects":782,"num_object_clones":0,"num_object_copies":3128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11971,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12022","reported_seq":12134,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:59.091052+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:51:59.091052+0000","last_peered":"2024-09-16T04:51:59.091052+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:51:59.091052+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6026508,"num_objects":736,"num_object_clones":0,"num_object_copies":2944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12020,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1453}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11752","reported_seq":11890,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.835345+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:51:55.835345+0000","last_peered":"2024-09-16T04:51:55.835345+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:51:55.835345+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6049682,"num_objects":739,"num_object_clones":0,"num_object_copies":2956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11749,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1416}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11611","reported_seq":11709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.870222+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:51:55.870222+0000","last_peered":"2024-09-16T04:51:55.870222+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:51:55.870222+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5896838,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11610,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1385}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11824","reported_seq":11918,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:59.298472+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:51:59.298472+0000","last_peered":"2024-09-16T04:51:59.298472+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:51:59.298472+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5806726,"num_objects":709,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11823,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1512}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12024","reported_seq":12162,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:55.606652+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:51:55.606652+0000","last_peered":"2024-09-16T04:51:55.606652+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:51:55.606652+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6206732,"num_objects":758,"num_object_clones":0,"num_object_copies":3032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1440}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11736","reported_seq":12969,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:51:59.434879+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:51:59.434879+0000","last_peered":"2024-09-16T04:51:59.434879+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:51:59.434879+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5838092,"num_objects":713,"num_object_clones":0,"num_object_copies":2852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11734,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-850}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11616","reported_seq":11716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.361766+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:00.361766+0000","last_peered":"2024-09-16T04:52:00.361766+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:00.361766+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5657868,"num_objects":691,"num_object_clones":0,"num_object_copies":2764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11614,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1537}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94922432,"num_objects":11593,"num_object_clones":0,"num_object_copies":46372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11593,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189894,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7658,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188313600,"data_stored":188313600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160721,"ondisk_log_size":160721,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733904,"kb_used_data":48304,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93637936,"statfs":{"total":96636764160,"available":95885246464,"internally_reserved":0,"allocated":49463296,"data_stored":49264787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":702017231},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,6,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744908,"kb_used_data":45944,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":93626932,"statfs":{"total":96636764160,"available":95873978368,"internally_reserved":0,"allocated":47046656,"data_stored":46848147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592712,"kb_used_data":47348,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93779128,"statfs":{"total":96636764160,"available":96029827072,"internally_reserved":0,"allocated":48484352,"data_stored":48285843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558427855},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734712,"kb_used_data":45032,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93637128,"statfs":{"total":96636764160,"available":95884419072,"internally_reserved":0,"allocated":46112768,"data_stored":45914259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":706211534},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45379584,"data_stored":45379584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47689728,"data_stored":47689728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46182400,"data_stored":46182400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49061888,"data_stored":49061888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:04.886 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:05.478 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:05.478 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:05.523 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-09-16T04:52:04.781830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":88360640,"num_objects":10792,"num_object_clones":0,"num_object_copies":43168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10792,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190695,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7715,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2790408,"kb_used_data":174976,"kb_used_omap":14,"kb_used_meta":2615345,"kb_avail":374696952,"statfs":{"total":386547056640,"available":383689678848,"internally_reserved":0,"allocated":179175424,"data_stored":178381388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2678113841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,9,10,13,23,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19054592,"num_objects":-2326,"num_object_clones":0,"num_object_copies":-9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2326,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004561"},"pg_stats":[{"pgid":"1.f","version":"37'11964","reported_seq":12085,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.898168+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:00.898168+0000","last_peered":"2024-09-16T04:52:00.898168+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:00.898168+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5652480,"num_objects":690,"num_object_clones":0,"num_object_copies":2760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12003","reported_seq":12096,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.218965+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:00.218965+0000","last_peered":"2024-09-16T04:52:00.218965+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:00.218965+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5494028,"num_objects":671,"num_object_clones":0,"num_object_copies":2684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12001,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1532}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12047","reported_seq":12151,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:04.401686+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:04.401686+0000","last_peered":"2024-09-16T04:52:04.401686+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:04.401686+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5102238,"num_objects":624,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12046,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1595}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12006","reported_seq":16556,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:03.370756+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:03.370756+0000","last_peered":"2024-09-16T04:52:03.370756+0000","last_clean":"2024-09-16T04:52:03.370756+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:03.370756+0000","last_undegraded":"2024-09-16T04:52:03.370756+0000","last_fullsized":"2024-09-16T04:52:03.370756+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:07.800773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5436684,"num_objects":664,"num_object_clones":0,"num_object_copies":2656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12004,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12058","reported_seq":12196,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.854269+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:00.854269+0000","last_peered":"2024-09-16T04:52:00.854269+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:00.854269+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5592332,"num_objects":683,"num_object_clones":0,"num_object_copies":2732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12056,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1565}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11649","reported_seq":11765,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.776253+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:00.776253+0000","last_peered":"2024-09-16T04:52:00.776253+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:00.776253+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5431296,"num_objects":663,"num_object_clones":0,"num_object_copies":2652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11649,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1515}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11935","reported_seq":12038,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.363185+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:00.363185+0000","last_peered":"2024-09-16T04:52:00.363185+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:00.363185+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5444876,"num_objects":665,"num_object_clones":0,"num_object_copies":2660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11933,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1548}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12106","reported_seq":16672,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:03.334001+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:03.334001+0000","last_peered":"2024-09-16T04:52:03.334001+0000","last_clean":"2024-09-16T04:52:03.334001+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:03.334001+0000","last_undegraded":"2024-09-16T04:52:03.334001+0000","last_fullsized":"2024-09-16T04:52:03.334001+0000","mapping_epoch":32,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:07.770943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5559564,"num_objects":679,"num_object_clones":0,"num_object_copies":2716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12104,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12036","reported_seq":16799,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:00.933323+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:00.933323+0000","last_peered":"2024-09-16T04:52:00.933323+0000","last_clean":"2024-09-16T04:52:00.933323+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:00.933323+0000","last_undegraded":"2024-09-16T04:52:00.933323+0000","last_fullsized":"2024-09-16T04:52:00.933323+0000","mapping_epoch":34,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:03.861716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5892632,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12033,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12080","reported_seq":12191,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:04.479350+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:04.479350+0000","last_peered":"2024-09-16T04:52:04.479350+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:04.479350+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5551372,"num_objects":678,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12078,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1511}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11810","reported_seq":11948,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.852871+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:00.852871+0000","last_peered":"2024-09-16T04:52:00.852871+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:00.852871+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5574546,"num_objects":681,"num_object_clones":0,"num_object_copies":2724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11807,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11678","reported_seq":11776,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.934088+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:00.934088+0000","last_peered":"2024-09-16T04:52:00.934088+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:00.934088+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5347974,"num_objects":653,"num_object_clones":0,"num_object_copies":2612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11677,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1452}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11873","reported_seq":11965,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:04.442490+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:04.442490+0000","last_peered":"2024-09-16T04:52:04.442490+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:04.442490+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5405318,"num_objects":660,"num_object_clones":0,"num_object_copies":2640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11872,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1561}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12073","reported_seq":12211,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.849111+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:00.849111+0000","last_peered":"2024-09-16T04:52:00.849111+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:00.849111+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5805324,"num_objects":709,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12071,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1489}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11788","reported_seq":13135,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:04.491473+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:04.491473+0000","last_peered":"2024-09-16T04:52:04.491473+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:04.491473+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5412108,"num_objects":661,"num_object_clones":0,"num_object_copies":2644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11786,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":630,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-845}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11616","reported_seq":11716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.361766+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:00.361766+0000","last_peered":"2024-09-16T04:52:00.361766+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:00.361766+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5657868,"num_objects":691,"num_object_clones":0,"num_object_copies":2764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11614,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1537}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":88360640,"num_objects":10792,"num_object_clones":0,"num_object_copies":43168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10792,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190695,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7715,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":176287744,"data_stored":176287744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734060,"kb_used_data":44316,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93637780,"statfs":{"total":96636764160,"available":95885086720,"internally_reserved":0,"allocated":45379584,"data_stored":45181075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":706277071},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,0,3,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745232,"kb_used_data":42108,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93626608,"statfs":{"total":96636764160,"available":95873646592,"internally_reserved":0,"allocated":43118592,"data_stored":42920083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576404,"kb_used_data":43520,"kb_used_omap":4,"kb_used_meta":532859,"kb_avail":93795436,"statfs":{"total":96636764160,"available":96046526464,"internally_reserved":0,"allocated":44564480,"data_stored":44365971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545648335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,8,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734712,"kb_used_data":45032,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93637128,"statfs":{"total":96636764160,"available":95884419072,"internally_reserved":0,"allocated":46112768,"data_stored":45914259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":706211534},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45379584,"data_stored":45379584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43655168,"data_stored":43655168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":42217472,"data_stored":42217472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45035520,"data_stored":45035520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:05.524 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:06.123 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:06.124 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:06.146 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-09-16T04:52:04.781830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":88360640,"num_objects":10792,"num_object_clones":0,"num_object_copies":43168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10792,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190695,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7715,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2790408,"kb_used_data":174976,"kb_used_omap":14,"kb_used_meta":2615345,"kb_avail":374696952,"statfs":{"total":386547056640,"available":383689678848,"internally_reserved":0,"allocated":179175424,"data_stored":178381388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2678113841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,9,10,13,23,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19054592,"num_objects":-2326,"num_object_clones":0,"num_object_copies":-9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2326,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004561"},"pg_stats":[{"pgid":"1.f","version":"37'11964","reported_seq":12085,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.898168+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:00.898168+0000","last_peered":"2024-09-16T04:52:00.898168+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:00.898168+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5652480,"num_objects":690,"num_object_clones":0,"num_object_copies":2760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12003","reported_seq":12096,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.218965+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:00.218965+0000","last_peered":"2024-09-16T04:52:00.218965+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:00.218965+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5494028,"num_objects":671,"num_object_clones":0,"num_object_copies":2684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12001,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1532}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12047","reported_seq":12151,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:04.401686+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:04.401686+0000","last_peered":"2024-09-16T04:52:04.401686+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:04.401686+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5102238,"num_objects":624,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12046,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1595}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12006","reported_seq":16556,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:03.370756+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:03.370756+0000","last_peered":"2024-09-16T04:52:03.370756+0000","last_clean":"2024-09-16T04:52:03.370756+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:03.370756+0000","last_undegraded":"2024-09-16T04:52:03.370756+0000","last_fullsized":"2024-09-16T04:52:03.370756+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:07.800773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5436684,"num_objects":664,"num_object_clones":0,"num_object_copies":2656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12004,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12058","reported_seq":12196,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.854269+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:00.854269+0000","last_peered":"2024-09-16T04:52:00.854269+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:00.854269+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5592332,"num_objects":683,"num_object_clones":0,"num_object_copies":2732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12056,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1565}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11649","reported_seq":11765,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.776253+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:00.776253+0000","last_peered":"2024-09-16T04:52:00.776253+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:00.776253+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5431296,"num_objects":663,"num_object_clones":0,"num_object_copies":2652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11649,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1515}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11935","reported_seq":12038,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.363185+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:00.363185+0000","last_peered":"2024-09-16T04:52:00.363185+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:00.363185+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5444876,"num_objects":665,"num_object_clones":0,"num_object_copies":2660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11933,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1548}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12106","reported_seq":16672,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:03.334001+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:03.334001+0000","last_peered":"2024-09-16T04:52:03.334001+0000","last_clean":"2024-09-16T04:52:03.334001+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:03.334001+0000","last_undegraded":"2024-09-16T04:52:03.334001+0000","last_fullsized":"2024-09-16T04:52:03.334001+0000","mapping_epoch":32,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:07.770943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5559564,"num_objects":679,"num_object_clones":0,"num_object_copies":2716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12104,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12036","reported_seq":16799,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:00.933323+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:00.933323+0000","last_peered":"2024-09-16T04:52:00.933323+0000","last_clean":"2024-09-16T04:52:00.933323+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:00.933323+0000","last_undegraded":"2024-09-16T04:52:00.933323+0000","last_fullsized":"2024-09-16T04:52:00.933323+0000","mapping_epoch":34,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:03.861716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5892632,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12033,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12080","reported_seq":12191,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:04.479350+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:04.479350+0000","last_peered":"2024-09-16T04:52:04.479350+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:04.479350+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5551372,"num_objects":678,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12078,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1511}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11810","reported_seq":11948,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.852871+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:00.852871+0000","last_peered":"2024-09-16T04:52:00.852871+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:00.852871+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5574546,"num_objects":681,"num_object_clones":0,"num_object_copies":2724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11807,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11678","reported_seq":11776,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.934088+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:00.934088+0000","last_peered":"2024-09-16T04:52:00.934088+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:00.934088+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5347974,"num_objects":653,"num_object_clones":0,"num_object_copies":2612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11677,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1452}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11873","reported_seq":11965,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:04.442490+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:04.442490+0000","last_peered":"2024-09-16T04:52:04.442490+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:04.442490+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5405318,"num_objects":660,"num_object_clones":0,"num_object_copies":2640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11872,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1561}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12073","reported_seq":12211,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.849111+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:00.849111+0000","last_peered":"2024-09-16T04:52:00.849111+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:00.849111+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5805324,"num_objects":709,"num_object_clones":0,"num_object_copies":2836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12071,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1489}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11788","reported_seq":13135,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:04.491473+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:04.491473+0000","last_peered":"2024-09-16T04:52:04.491473+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:04.491473+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5412108,"num_objects":661,"num_object_clones":0,"num_object_copies":2644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11786,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":630,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-845}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11616","reported_seq":11716,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:00.361766+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:00.361766+0000","last_peered":"2024-09-16T04:52:00.361766+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:00.361766+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5657868,"num_objects":691,"num_object_clones":0,"num_object_copies":2764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11614,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1537}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":88360640,"num_objects":10792,"num_object_clones":0,"num_object_copies":43168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10792,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190695,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7715,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":176287744,"data_stored":176287744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734060,"kb_used_data":44316,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93637780,"statfs":{"total":96636764160,"available":95885086720,"internally_reserved":0,"allocated":45379584,"data_stored":45181075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":706277071},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,0,3,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745232,"kb_used_data":42108,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93626608,"statfs":{"total":96636764160,"available":95873646592,"internally_reserved":0,"allocated":43118592,"data_stored":42920083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576404,"kb_used_data":43520,"kb_used_omap":4,"kb_used_meta":532859,"kb_avail":93795436,"statfs":{"total":96636764160,"available":96046526464,"internally_reserved":0,"allocated":44564480,"data_stored":44365971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545648335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,8,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734712,"kb_used_data":45032,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93637128,"statfs":{"total":96636764160,"available":95884419072,"internally_reserved":0,"allocated":46112768,"data_stored":45914259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":706211534},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45379584,"data_stored":45379584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43655168,"data_stored":43655168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":42217472,"data_stored":42217472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45035520,"data_stored":45035520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:06.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:52:06.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11964", 'reported_seq': 12085, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:00.898168+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:52:00.898168+0000', 'last_peered': '2024-09-16T04:52:00.898168+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:52:00.898168+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5652480, 'num_objects': 690, 'num_object_clones': 0, 'num_object_copies': 2760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11964, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1496}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:06.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:52:06.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12003", 'reported_seq': 12096, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:00.218965+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:52:00.218965+0000', 'last_peered': '2024-09-16T04:52:00.218965+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:52:00.218965+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5494028, 'num_objects': 671, 'num_object_clones': 0, 'num_object_copies': 2684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 671, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12001, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1532}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:06.147 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:52:06.147 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12047", 'reported_seq': 12151, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:04.401686+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:52:04.401686+0000', 'last_peered': '2024-09-16T04:52:04.401686+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:52:04.401686+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5102238, 'num_objects': 624, 'num_object_clones': 0, 'num_object_copies': 2496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 624, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12046, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1595}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:06.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:52:06.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12058", 'reported_seq': 12196, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:00.854269+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:52:00.854269+0000', 'last_peered': '2024-09-16T04:52:00.854269+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:52:00.854269+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5592332, 'num_objects': 683, 'num_object_clones': 0, 'num_object_copies': 2732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 683, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12056, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1565}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:06.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:52:06.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11649", 'reported_seq': 11765, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:00.776253+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:52:00.776253+0000', 'last_peered': '2024-09-16T04:52:00.776253+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:52:00.776253+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5431296, 'num_objects': 663, 'num_object_clones': 0, 'num_object_copies': 2652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11649, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1515}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:06.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:52:06.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11935", 'reported_seq': 12038, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:00.363185+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:52:00.363185+0000', 'last_peered': '2024-09-16T04:52:00.363185+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:52:00.363185+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5444876, 'num_objects': 665, 'num_object_clones': 0, 'num_object_copies': 2660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11933, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:06.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:52:06.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12080", 'reported_seq': 12191, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:04.479350+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:52:04.479350+0000', 'last_peered': '2024-09-16T04:52:04.479350+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:52:04.479350+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5551372, 'num_objects': 678, 'num_object_clones': 0, 'num_object_copies': 2712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12078, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1511}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:06.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:52:06.148 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11810", 'reported_seq': 11948, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:00.852871+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:52:00.852871+0000', 'last_peered': '2024-09-16T04:52:00.852871+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:52:00.852871+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5574546, 'num_objects': 681, 'num_object_clones': 0, 'num_object_copies': 2724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 681, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11807, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1474}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:06.148 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:52:06.149 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11678", 'reported_seq': 11776, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:00.934088+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:52:00.934088+0000', 'last_peered': '2024-09-16T04:52:00.934088+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:52:00.934088+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5347974, 'num_objects': 653, 'num_object_clones': 0, 'num_object_copies': 2612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11677, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1452}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:06.149 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:52:06.149 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11873", 'reported_seq': 11965, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:04.442490+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:52:04.442490+0000', 'last_peered': '2024-09-16T04:52:04.442490+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:52:04.442490+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5405318, 'num_objects': 660, 'num_object_clones': 0, 'num_object_copies': 2640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 660, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11872, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1561}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:06.149 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:52:06.149 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12073", 'reported_seq': 12211, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:00.849111+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:52:00.849111+0000', 'last_peered': '2024-09-16T04:52:00.849111+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:52:00.849111+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5805324, 'num_objects': 709, 'num_object_clones': 0, 'num_object_copies': 2836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12071, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1489}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:06.149 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:52:06.149 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11788", 'reported_seq': 13135, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:04.491473+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:52:04.491473+0000', 'last_peered': '2024-09-16T04:52:04.491473+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:52:04.491473+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5412108, 'num_objects': 661, 'num_object_clones': 0, 'num_object_copies': 2644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1506, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 661, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11786, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 630, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -845}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:06.149 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:52:06.149 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11616", 'reported_seq': 11716, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:00.361766+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:52:00.361766+0000', 'last_peered': '2024-09-16T04:52:00.361766+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:52:00.361766+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5657868, 'num_objects': 691, 'num_object_clones': 0, 'num_object_copies': 2764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11614, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1537}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:06.150 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:52:06.908 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:06.908 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27867/43168 objects degraded (64.555%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":739,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":10792,"data_bytes":88360640,"bytes_used":2857377792,"bytes_avail":383689678848,"bytes_total":386547056640,"degraded_objects":27867,"degraded_total":43168,"degraded_ratio":0.64554762787249809,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":193},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:52:06.936 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:52:06.937 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:07.523 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:07.524 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:07.552 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-09-16T04:52:06.782482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":83412672,"num_objects":10188,"num_object_clones":0,"num_object_copies":40752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10188,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191299,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7715,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160726,"ondisk_log_size":160726,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2757768,"kb_used_data":167280,"kb_used_omap":14,"kb_used_meta":2590385,"kb_avail":374729592,"statfs":{"total":386547056640,"available":383723102208,"internally_reserved":0,"allocated":171294720,"data_stored":170500684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2652554801},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,7,5,20,25,12,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20930560,"num_objects":-2555,"num_object_clones":0,"num_object_copies":-10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2555,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004355"},"pg_stats":[{"pgid":"1.f","version":"37'12028","reported_seq":12149,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:06.030545+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:06.030545+0000","last_peered":"2024-09-16T04:52:06.030545+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:06.030545+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5128192,"num_objects":626,"num_object_clones":0,"num_object_copies":2504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12028,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12062","reported_seq":12156,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.349138+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:05.349138+0000","last_peered":"2024-09-16T04:52:05.349138+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:05.349138+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5010700,"num_objects":612,"num_object_clones":0,"num_object_copies":2448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12060,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12047","reported_seq":12151,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:04.401686+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:04.401686+0000","last_peered":"2024-09-16T04:52:04.401686+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:04.401686+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5102238,"num_objects":624,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12046,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1595}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12006","reported_seq":16556,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:03.370756+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:03.370756+0000","last_peered":"2024-09-16T04:52:03.370756+0000","last_clean":"2024-09-16T04:52:03.370756+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:03.370756+0000","last_undegraded":"2024-09-16T04:52:03.370756+0000","last_fullsized":"2024-09-16T04:52:03.370756+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:07.800773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5436684,"num_objects":664,"num_object_clones":0,"num_object_copies":2656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12004,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12126","reported_seq":12264,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.812143+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:05.812143+0000","last_peered":"2024-09-16T04:52:05.812143+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:05.812143+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5035276,"num_objects":615,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12124,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11702","reported_seq":11818,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.847854+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:05.847854+0000","last_peered":"2024-09-16T04:52:05.847854+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:05.847854+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4997120,"num_objects":610,"num_object_clones":0,"num_object_copies":2440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11702,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1568}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11987","reported_seq":12090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.481087+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:05.481087+0000","last_peered":"2024-09-16T04:52:05.481087+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:05.481087+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5018892,"num_objects":613,"num_object_clones":0,"num_object_copies":2452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1600}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12106","reported_seq":16672,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:03.334001+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:03.334001+0000","last_peered":"2024-09-16T04:52:03.334001+0000","last_clean":"2024-09-16T04:52:03.334001+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:03.334001+0000","last_undegraded":"2024-09-16T04:52:03.334001+0000","last_fullsized":"2024-09-16T04:52:03.334001+0000","mapping_epoch":32,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:07.770943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5559564,"num_objects":679,"num_object_clones":0,"num_object_copies":2716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12104,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12108","reported_seq":16871,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:05.955866+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:05.955866+0000","last_peered":"2024-09-16T04:52:05.955866+0000","last_clean":"2024-09-16T04:52:05.955866+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:05.955866+0000","last_undegraded":"2024-09-16T04:52:05.955866+0000","last_fullsized":"2024-09-16T04:52:05.955866+0000","mapping_epoch":34,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:08.881934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5302808,"num_objects":648,"num_object_clones":0,"num_object_copies":2592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12105,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12080","reported_seq":12191,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:04.479350+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:04.479350+0000","last_peered":"2024-09-16T04:52:04.479350+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:04.479350+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5551372,"num_objects":678,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12078,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1511}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11864","reported_seq":12002,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.666893+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:05.666893+0000","last_peered":"2024-09-16T04:52:05.666893+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:05.666893+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5132178,"num_objects":627,"num_object_clones":0,"num_object_copies":2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11861,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11732","reported_seq":11830,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.984989+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:05.984989+0000","last_peered":"2024-09-16T04:52:05.984989+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:05.984989+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4905606,"num_objects":599,"num_object_clones":0,"num_object_copies":2396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11731,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1506}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11873","reported_seq":11965,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:04.442490+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:04.442490+0000","last_peered":"2024-09-16T04:52:04.442490+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:04.442490+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5405318,"num_objects":660,"num_object_clones":0,"num_object_copies":2640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11872,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1561}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12125","reported_seq":12263,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.596593+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:05.596593+0000","last_peered":"2024-09-16T04:52:05.596593+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:05.596593+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5379340,"num_objects":657,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12123,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1541}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11788","reported_seq":13135,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:04.491473+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:04.491473+0000","last_peered":"2024-09-16T04:52:04.491473+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:04.491473+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5412108,"num_objects":661,"num_object_clones":0,"num_object_copies":2644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11786,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":630,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-845}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11692","reported_seq":11792,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.450132+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:05.450132+0000","last_peered":"2024-09-16T04:52:05.450132+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:05.450132+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5035276,"num_objects":615,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11690,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1613}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":83412672,"num_objects":10188,"num_object_clones":0,"num_object_copies":40752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10188,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191299,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7715,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":168271872,"data_stored":168271872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160726,"ondisk_log_size":160726,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717720,"kb_used_data":40440,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93654120,"statfs":{"total":96636764160,"available":95901818880,"internally_reserved":0,"allocated":41410560,"data_stored":41212051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":693497551},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,8,6,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745232,"kb_used_data":42108,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93626608,"statfs":{"total":96636764160,"available":95873646592,"internally_reserved":0,"allocated":43118592,"data_stored":42920083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576404,"kb_used_data":43520,"kb_used_omap":4,"kb_used_meta":532859,"kb_avail":93795436,"statfs":{"total":96636764160,"available":96046526464,"internally_reserved":0,"allocated":44564480,"data_stored":44365971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545648335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,8,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718412,"kb_used_data":41212,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93653428,"statfs":{"total":96636764160,"available":95901110272,"internally_reserved":0,"allocated":42201088,"data_stored":42002579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":693432014},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,0,0,1,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41357312,"data_stored":41357312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43655168,"data_stored":43655168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":42217472,"data_stored":42217472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41041920,"data_stored":41041920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:10.555 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:11.184 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:11.184 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:11.226 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2024-09-16T04:52:10.784139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":79128256,"num_objects":9665,"num_object_clones":0,"num_object_copies":38660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9665,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191822,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7765,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160549,"ondisk_log_size":160549,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758820,"kb_used_data":155840,"kb_used_omap":14,"kb_used_meta":2602865,"kb_avail":374728540,"statfs":{"total":386547056640,"available":383722024960,"internally_reserved":0,"allocated":159580160,"data_stored":158786124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2665334321},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,9,20,21,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19218432,"num_objects":-2346,"num_object_clones":0,"num_object_copies":-9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2346,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004453"},"pg_stats":[{"pgid":"1.f","version":"37'12028","reported_seq":12149,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:06.030545+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:06.030545+0000","last_peered":"2024-09-16T04:52:06.030545+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:06.030545+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5128192,"num_objects":626,"num_object_clones":0,"num_object_copies":2504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12028,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12115","reported_seq":12209,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:10.655176+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:10.655176+0000","last_peered":"2024-09-16T04:52:10.655176+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:10.655176+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4576524,"num_objects":559,"num_object_clones":0,"num_object_copies":2236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12113,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1644}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12118","reported_seq":12222,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:09.244098+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:09.244098+0000","last_peered":"2024-09-16T04:52:09.244098+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:09.244098+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4520606,"num_objects":553,"num_object_clones":0,"num_object_copies":2212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12117,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12061","reported_seq":16611,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:08.108656+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:08.108656+0000","last_peered":"2024-09-16T04:52:08.108656+0000","last_clean":"2024-09-16T04:52:08.108656+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:08.108656+0000","last_undegraded":"2024-09-16T04:52:08.108656+0000","last_fullsized":"2024-09-16T04:52:08.108656+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4986124,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12059,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12126","reported_seq":12264,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.812143+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:05.812143+0000","last_peered":"2024-09-16T04:52:05.812143+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:05.812143+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5035276,"num_objects":615,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12124,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11702","reported_seq":11818,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.847854+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:05.847854+0000","last_peered":"2024-09-16T04:52:05.847854+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:05.847854+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4997120,"num_objects":610,"num_object_clones":0,"num_object_copies":2440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11702,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1568}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12048","reported_seq":12151,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:10.431466+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:10.431466+0000","last_peered":"2024-09-16T04:52:10.431466+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:10.431466+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4519180,"num_objects":552,"num_object_clones":0,"num_object_copies":2208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12046,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1661}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12178","reported_seq":16745,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:08.340907+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:08.340907+0000","last_peered":"2024-09-16T04:52:08.340907+0000","last_clean":"2024-09-16T04:52:08.340907+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:08.340907+0000","last_undegraded":"2024-09-16T04:52:08.340907+0000","last_fullsized":"2024-09-16T04:52:08.340907+0000","mapping_epoch":32,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4969740,"num_objects":607,"num_object_clones":0,"num_object_copies":2428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12176,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12108","reported_seq":16871,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:05.955866+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:05.955866+0000","last_peered":"2024-09-16T04:52:05.955866+0000","last_clean":"2024-09-16T04:52:05.955866+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:05.955866+0000","last_undegraded":"2024-09-16T04:52:05.955866+0000","last_fullsized":"2024-09-16T04:52:05.955866+0000","mapping_epoch":34,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:08.881934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5302808,"num_objects":648,"num_object_clones":0,"num_object_copies":2592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12105,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12134","reported_seq":12246,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:09.257127+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:09.257127+0000","last_peered":"2024-09-16T04:52:09.257127+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:09.257127+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5109004,"num_objects":624,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12132,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1565}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11864","reported_seq":12002,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.666893+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:05.666893+0000","last_peered":"2024-09-16T04:52:05.666893+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:05.666893+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5132178,"num_objects":627,"num_object_clones":0,"num_object_copies":2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11861,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11732","reported_seq":11830,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.984989+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:05.984989+0000","last_peered":"2024-09-16T04:52:05.984989+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:05.984989+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4905606,"num_objects":599,"num_object_clones":0,"num_object_copies":2396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11731,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1506}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11924","reported_seq":12018,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:09.326330+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:09.326330+0000","last_peered":"2024-09-16T04:52:09.326330+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:09.326330+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4987526,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11923,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1612}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12125","reported_seq":12263,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.596593+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:05.596593+0000","last_peered":"2024-09-16T04:52:05.596593+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:05.596593+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5379340,"num_objects":657,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12123,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1541}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11839","reported_seq":13287,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:09.451889+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:09.451889+0000","last_peered":"2024-09-16T04:52:09.451889+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:09.451889+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4994316,"num_objects":610,"num_object_clones":0,"num_object_copies":2440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1456,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11837,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-846}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11747","reported_seq":11847,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:10.589265+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:10.589265+0000","last_peered":"2024-09-16T04:52:10.589265+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:10.589265+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4584716,"num_objects":560,"num_object_clones":0,"num_object_copies":2240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11745,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1668}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":79128256,"num_objects":9665,"num_object_clones":0,"num_object_copies":38660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9665,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191822,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7765,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":156352512,"data_stored":156352512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160549,"ondisk_log_size":160549,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717720,"kb_used_data":40440,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93654120,"statfs":{"total":96636764160,"available":95901818880,"internally_reserved":0,"allocated":41410560,"data_stored":41212051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":693497551},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,8,6,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745652,"kb_used_data":38352,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93626188,"statfs":{"total":96636764160,"available":95873216512,"internally_reserved":0,"allocated":39272448,"data_stored":39073939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576740,"kb_used_data":39700,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93795100,"statfs":{"total":96636764160,"available":96046182400,"internally_reserved":0,"allocated":40652800,"data_stored":40454291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549908175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718708,"kb_used_data":37348,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93653132,"statfs":{"total":96636764160,"available":95900807168,"internally_reserved":0,"allocated":38244352,"data_stored":38045843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":697691854},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37318656,"data_stored":37318656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":39698432,"data_stored":39698432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38293504,"data_stored":38293504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41041920,"data_stored":41041920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:11.227 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:11.817 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:11.817 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:11.841 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2024-09-16T04:52:10.784139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":79128256,"num_objects":9665,"num_object_clones":0,"num_object_copies":38660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9665,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191822,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7765,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160549,"ondisk_log_size":160549,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758820,"kb_used_data":155840,"kb_used_omap":14,"kb_used_meta":2602865,"kb_avail":374728540,"statfs":{"total":386547056640,"available":383722024960,"internally_reserved":0,"allocated":159580160,"data_stored":158786124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2665334321},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,9,20,21,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19218432,"num_objects":-2346,"num_object_clones":0,"num_object_copies":-9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2346,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004453"},"pg_stats":[{"pgid":"1.f","version":"37'12028","reported_seq":12149,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:06.030545+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:06.030545+0000","last_peered":"2024-09-16T04:52:06.030545+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:06.030545+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5128192,"num_objects":626,"num_object_clones":0,"num_object_copies":2504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12028,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12115","reported_seq":12209,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:10.655176+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:10.655176+0000","last_peered":"2024-09-16T04:52:10.655176+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:10.655176+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4576524,"num_objects":559,"num_object_clones":0,"num_object_copies":2236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12113,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1644}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12118","reported_seq":12222,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:09.244098+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:09.244098+0000","last_peered":"2024-09-16T04:52:09.244098+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:09.244098+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4520606,"num_objects":553,"num_object_clones":0,"num_object_copies":2212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12117,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12061","reported_seq":16611,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:08.108656+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:08.108656+0000","last_peered":"2024-09-16T04:52:08.108656+0000","last_clean":"2024-09-16T04:52:08.108656+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:08.108656+0000","last_undegraded":"2024-09-16T04:52:08.108656+0000","last_fullsized":"2024-09-16T04:52:08.108656+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4986124,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12059,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12126","reported_seq":12264,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.812143+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:05.812143+0000","last_peered":"2024-09-16T04:52:05.812143+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:05.812143+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5035276,"num_objects":615,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12124,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11702","reported_seq":11818,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.847854+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:05.847854+0000","last_peered":"2024-09-16T04:52:05.847854+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:05.847854+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4997120,"num_objects":610,"num_object_clones":0,"num_object_copies":2440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11702,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1568}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12048","reported_seq":12151,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:10.431466+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:10.431466+0000","last_peered":"2024-09-16T04:52:10.431466+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:10.431466+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4519180,"num_objects":552,"num_object_clones":0,"num_object_copies":2208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12046,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1661}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12178","reported_seq":16745,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:08.340907+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:08.340907+0000","last_peered":"2024-09-16T04:52:08.340907+0000","last_clean":"2024-09-16T04:52:08.340907+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:08.340907+0000","last_undegraded":"2024-09-16T04:52:08.340907+0000","last_fullsized":"2024-09-16T04:52:08.340907+0000","mapping_epoch":32,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4969740,"num_objects":607,"num_object_clones":0,"num_object_copies":2428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12176,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12108","reported_seq":16871,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:05.955866+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:05.955866+0000","last_peered":"2024-09-16T04:52:05.955866+0000","last_clean":"2024-09-16T04:52:05.955866+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:05.955866+0000","last_undegraded":"2024-09-16T04:52:05.955866+0000","last_fullsized":"2024-09-16T04:52:05.955866+0000","mapping_epoch":34,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:08.881934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5302808,"num_objects":648,"num_object_clones":0,"num_object_copies":2592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12105,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12134","reported_seq":12246,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:09.257127+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:09.257127+0000","last_peered":"2024-09-16T04:52:09.257127+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:09.257127+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5109004,"num_objects":624,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12132,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1565}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11864","reported_seq":12002,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.666893+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:05.666893+0000","last_peered":"2024-09-16T04:52:05.666893+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:05.666893+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5132178,"num_objects":627,"num_object_clones":0,"num_object_copies":2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11861,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11732","reported_seq":11830,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.984989+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:05.984989+0000","last_peered":"2024-09-16T04:52:05.984989+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:05.984989+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4905606,"num_objects":599,"num_object_clones":0,"num_object_copies":2396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11731,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1506}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11924","reported_seq":12018,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:09.326330+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:09.326330+0000","last_peered":"2024-09-16T04:52:09.326330+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:09.326330+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4987526,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11923,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1612}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12125","reported_seq":12263,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:05.596593+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:05.596593+0000","last_peered":"2024-09-16T04:52:05.596593+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:05.596593+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5379340,"num_objects":657,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12123,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1541}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11839","reported_seq":13287,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:09.451889+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:09.451889+0000","last_peered":"2024-09-16T04:52:09.451889+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:09.451889+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4994316,"num_objects":610,"num_object_clones":0,"num_object_copies":2440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1456,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11837,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-846}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11747","reported_seq":11847,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:10.589265+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:10.589265+0000","last_peered":"2024-09-16T04:52:10.589265+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:10.589265+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4584716,"num_objects":560,"num_object_clones":0,"num_object_copies":2240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11745,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1668}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":79128256,"num_objects":9665,"num_object_clones":0,"num_object_copies":38660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9665,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191822,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7765,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":156352512,"data_stored":156352512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160549,"ondisk_log_size":160549,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717720,"kb_used_data":40440,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93654120,"statfs":{"total":96636764160,"available":95901818880,"internally_reserved":0,"allocated":41410560,"data_stored":41212051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":693497551},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,8,6,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745652,"kb_used_data":38352,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93626188,"statfs":{"total":96636764160,"available":95873216512,"internally_reserved":0,"allocated":39272448,"data_stored":39073939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576740,"kb_used_data":39700,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93795100,"statfs":{"total":96636764160,"available":96046182400,"internally_reserved":0,"allocated":40652800,"data_stored":40454291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549908175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718708,"kb_used_data":37348,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93653132,"statfs":{"total":96636764160,"available":95900807168,"internally_reserved":0,"allocated":38244352,"data_stored":38045843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":697691854},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37318656,"data_stored":37318656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":39698432,"data_stored":39698432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38293504,"data_stored":38293504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41041920,"data_stored":41041920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:11.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:52:11.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12028", 'reported_seq': 12149, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:06.030545+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:52:06.030545+0000', 'last_peered': '2024-09-16T04:52:06.030545+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:52:06.030545+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5128192, 'num_objects': 626, 'num_object_clones': 0, 'num_object_copies': 2504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 626, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12028, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1560}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:11.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:52:11.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12115", 'reported_seq': 12209, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:10.655176+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:52:10.655176+0000', 'last_peered': '2024-09-16T04:52:10.655176+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:52:10.655176+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4576524, 'num_objects': 559, 'num_object_clones': 0, 'num_object_copies': 2236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 559, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12113, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1644}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:11.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:52:11.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12118", 'reported_seq': 12222, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:09.244098+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:52:09.244098+0000', 'last_peered': '2024-09-16T04:52:09.244098+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:52:09.244098+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4520606, 'num_objects': 553, 'num_object_clones': 0, 'num_object_copies': 2212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 553, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12117, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1666}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:11.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:52:11.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12126", 'reported_seq': 12264, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:05.812143+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:52:05.812143+0000', 'last_peered': '2024-09-16T04:52:05.812143+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:52:05.812143+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5035276, 'num_objects': 615, 'num_object_clones': 0, 'num_object_copies': 2460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 615, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12124, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1633}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:11.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:52:11.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11702", 'reported_seq': 11818, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:05.847854+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:52:05.847854+0000', 'last_peered': '2024-09-16T04:52:05.847854+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:52:05.847854+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4997120, 'num_objects': 610, 'num_object_clones': 0, 'num_object_copies': 2440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 610, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11702, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1568}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:11.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:52:11.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12048", 'reported_seq': 12151, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:10.431466+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:52:10.431466+0000', 'last_peered': '2024-09-16T04:52:10.431466+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:52:10.431466+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4519180, 'num_objects': 552, 'num_object_clones': 0, 'num_object_copies': 2208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 552, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12046, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1661}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:11.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:52:11.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12134", 'reported_seq': 12246, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:09.257127+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:52:09.257127+0000', 'last_peered': '2024-09-16T04:52:09.257127+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:52:09.257127+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5109004, 'num_objects': 624, 'num_object_clones': 0, 'num_object_copies': 2496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12132, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1565}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:11.844 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:52:11.844 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11864", 'reported_seq': 12002, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:05.666893+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:52:05.666893+0000', 'last_peered': '2024-09-16T04:52:05.666893+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:52:05.666893+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5132178, 'num_objects': 627, 'num_object_clones': 0, 'num_object_copies': 2508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11861, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1528}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:11.844 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:52:11.844 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11732", 'reported_seq': 11830, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:05.984989+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:52:05.984989+0000', 'last_peered': '2024-09-16T04:52:05.984989+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:52:05.984989+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4905606, 'num_objects': 599, 'num_object_clones': 0, 'num_object_copies': 2396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11731, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1506}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:11.844 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:52:11.844 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11924", 'reported_seq': 12018, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:09.326330+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:52:09.326330+0000', 'last_peered': '2024-09-16T04:52:09.326330+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:52:09.326330+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4987526, 'num_objects': 609, 'num_object_clones': 0, 'num_object_copies': 2436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11923, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1612}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:11.844 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:52:11.844 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12125", 'reported_seq': 12263, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:05.596593+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:52:05.596593+0000', 'last_peered': '2024-09-16T04:52:05.596593+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:52:05.596593+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5379340, 'num_objects': 657, 'num_object_clones': 0, 'num_object_copies': 2628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 657, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12123, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1541}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:11.844 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:52:11.844 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11839", 'reported_seq': 13287, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:09.451889+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:52:09.451889+0000', 'last_peered': '2024-09-16T04:52:09.451889+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:52:09.451889+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4994316, 'num_objects': 610, 'num_object_clones': 0, 'num_object_copies': 2440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1456, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 610, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11837, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 680, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -846}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:11.845 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:52:11.845 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11747", 'reported_seq': 11847, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:10.589265+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:52:10.589265+0000', 'last_peered': '2024-09-16T04:52:10.589265+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:52:10.589265+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4584716, 'num_objects': 560, 'num_object_clones': 0, 'num_object_copies': 2240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 560, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11745, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1668}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:11.845 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:52:12.573 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:12.574 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27817/38660 objects degraded (71.953%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":744,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":9665,"data_bytes":79128256,"bytes_used":2825031680,"bytes_avail":383722024960,"bytes_total":386547056640,"degraded_objects":27817,"degraded_total":38660,"degraded_ratio":0.71952922917744444,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":195},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:52:12.605 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:52:12.605 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:13.216 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:13.216 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:13.245 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-16T04:52:12.785085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":75507392,"num_objects":9223,"num_object_clones":0,"num_object_copies":36892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192264,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7765,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160891,"ondisk_log_size":160891,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2759104,"kb_used_data":151980,"kb_used_omap":14,"kb_used_meta":2607025,"kb_avail":374728256,"statfs":{"total":386547056640,"available":383721734144,"internally_reserved":0,"allocated":155627520,"data_stored":154833484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2669594161},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,8,20,26,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19415040,"num_objects":-2370,"num_object_clones":0,"num_object_copies":-9480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2370,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004825"},"pg_stats":[{"pgid":"1.f","version":"37'12095","reported_seq":12215,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:10.907395+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:10.907395+0000","last_peered":"2024-09-16T04:52:10.907395+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:10.907395+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4579328,"num_objects":559,"num_object_clones":0,"num_object_copies":2236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12095,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12115","reported_seq":12209,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:10.655176+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:10.655176+0000","last_peered":"2024-09-16T04:52:10.655176+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:10.655176+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4576524,"num_objects":559,"num_object_clones":0,"num_object_copies":2236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12113,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1644}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12118","reported_seq":12222,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:09.244098+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:09.244098+0000","last_peered":"2024-09-16T04:52:09.244098+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:09.244098+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4520606,"num_objects":553,"num_object_clones":0,"num_object_copies":2212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12117,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12061","reported_seq":16611,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:08.108656+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:08.108656+0000","last_peered":"2024-09-16T04:52:08.108656+0000","last_clean":"2024-09-16T04:52:08.108656+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:08.108656+0000","last_undegraded":"2024-09-16T04:52:08.108656+0000","last_fullsized":"2024-09-16T04:52:08.108656+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4986124,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12059,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12181","reported_seq":12319,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:10.704887+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:10.704887+0000","last_peered":"2024-09-16T04:52:10.704887+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:10.704887+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4584716,"num_objects":560,"num_object_clones":0,"num_object_copies":2240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12179,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11761","reported_seq":11877,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:11.070273+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:11.070273+0000","last_peered":"2024-09-16T04:52:11.070273+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:11.070273+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4513792,"num_objects":551,"num_object_clones":0,"num_object_copies":2204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11761,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12048","reported_seq":12151,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:10.431466+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:10.431466+0000","last_peered":"2024-09-16T04:52:10.431466+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:10.431466+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4519180,"num_objects":552,"num_object_clones":0,"num_object_copies":2208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12046,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1661}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12178","reported_seq":16745,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:08.340907+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:08.340907+0000","last_peered":"2024-09-16T04:52:08.340907+0000","last_clean":"2024-09-16T04:52:08.340907+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:08.340907+0000","last_undegraded":"2024-09-16T04:52:08.340907+0000","last_fullsized":"2024-09-16T04:52:08.340907+0000","mapping_epoch":32,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4969740,"num_objects":607,"num_object_clones":0,"num_object_copies":2428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12176,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12174","reported_seq":16937,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:10.905711+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:10.905711+0000","last_peered":"2024-09-16T04:52:10.905711+0000","last_clean":"2024-09-16T04:52:10.905711+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:10.905711+0000","last_undegraded":"2024-09-16T04:52:10.905711+0000","last_fullsized":"2024-09-16T04:52:10.905711+0000","mapping_epoch":34,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:13.909143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4762136,"num_objects":582,"num_object_clones":0,"num_object_copies":2328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12171,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12134","reported_seq":12246,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:09.257127+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:09.257127+0000","last_peered":"2024-09-16T04:52:09.257127+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:09.257127+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5109004,"num_objects":624,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12132,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1565}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11931","reported_seq":12069,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:11.006256+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:11.006256+0000","last_peered":"2024-09-16T04:52:11.006256+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:11.006256+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4583314,"num_objects":560,"num_object_clones":0,"num_object_copies":2240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11928,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1595}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11797","reported_seq":11895,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:10.910171+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:10.910171+0000","last_peered":"2024-09-16T04:52:10.910171+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:10.910171+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4373126,"num_objects":534,"num_object_clones":0,"num_object_copies":2136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11796,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1571}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11924","reported_seq":12018,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:09.326330+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:09.326330+0000","last_peered":"2024-09-16T04:52:09.326330+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:09.326330+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4987526,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11923,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1612}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12188","reported_seq":12326,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:11.033758+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:11.033758+0000","last_peered":"2024-09-16T04:52:11.033758+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:11.033758+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4863244,"num_objects":594,"num_object_clones":0,"num_object_copies":2376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12186,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1604}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11839","reported_seq":13287,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:09.451889+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:09.451889+0000","last_peered":"2024-09-16T04:52:09.451889+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:09.451889+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4994316,"num_objects":610,"num_object_clones":0,"num_object_copies":2440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1456,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11837,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-846}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11747","reported_seq":11847,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:10.589265+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:10.589265+0000","last_peered":"2024-09-16T04:52:10.589265+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:10.589265+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4584716,"num_objects":560,"num_object_clones":0,"num_object_copies":2240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11745,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1668}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":75507392,"num_objects":9223,"num_object_clones":0,"num_object_copies":36892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192264,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7765,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":152276992,"data_stored":152276992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160891,"ondisk_log_size":160891,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718004,"kb_used_data":36580,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93653836,"statfs":{"total":96636764160,"available":95901528064,"internally_reserved":0,"allocated":37457920,"data_stored":37259411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697757391},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,8,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745652,"kb_used_data":38352,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93626188,"statfs":{"total":96636764160,"available":95873216512,"internally_reserved":0,"allocated":39272448,"data_stored":39073939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576740,"kb_used_data":39700,"kb_used_omap":4,"kb_used_meta":537019,"kb_avail":93795100,"statfs":{"total":96636764160,"available":96046182400,"internally_reserved":0,"allocated":40652800,"data_stored":40454291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549908175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718708,"kb_used_data":37348,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93653132,"statfs":{"total":96636764160,"available":95900807168,"internally_reserved":0,"allocated":38244352,"data_stored":38045843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":697691854},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37318656,"data_stored":37318656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":39698432,"data_stored":39698432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38293504,"data_stored":38293504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36966400,"data_stored":36966400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:16.248 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:16.849 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:16.850 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:16.882 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-09-16T04:52:16.786940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67372736,"num_objects":8230,"num_object_clones":0,"num_object_copies":32920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8230,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193257,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7804,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2743700,"kb_used_data":136592,"kb_used_omap":14,"kb_used_meta":2607025,"kb_avail":374743660,"statfs":{"total":386547056640,"available":383737507840,"internally_reserved":0,"allocated":139870208,"data_stored":139076172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2669594161},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,5,10,15,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20987904,"num_objects":-2562,"num_object_clones":0,"num_object_copies":-10248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2562,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005109"},"pg_stats":[{"pgid":"1.f","version":"37'12150","reported_seq":12272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.028986+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:16.028986+0000","last_peered":"2024-09-16T04:52:16.028986+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:16.028986+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4128768,"num_objects":504,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12150,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1682}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12172","reported_seq":12266,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.693783+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:15.693783+0000","last_peered":"2024-09-16T04:52:15.693783+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:15.693783+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4109580,"num_objects":502,"num_object_clones":0,"num_object_copies":2008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12170,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12171","reported_seq":12275,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:14.520562+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:14.520562+0000","last_peered":"2024-09-16T04:52:14.520562+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:14.520562+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4086430,"num_objects":500,"num_object_clones":0,"num_object_copies":2000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":500,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12170,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1719}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12124","reported_seq":16673,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:13.361997+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:13.361997+0000","last_peered":"2024-09-16T04:52:13.361997+0000","last_clean":"2024-09-16T04:52:13.361997+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:13.361997+0000","last_undegraded":"2024-09-16T04:52:13.361997+0000","last_fullsized":"2024-09-16T04:52:13.361997+0000","mapping_epoch":36,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:13.798055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4470028,"num_objects":546,"num_object_clones":0,"num_object_copies":2184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12244","reported_seq":12380,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.000867+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:16.000867+0000","last_peered":"2024-09-16T04:52:16.000867+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:16.000867+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4068620,"num_objects":497,"num_object_clones":0,"num_object_copies":1988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12242,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11821","reported_seq":11937,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.055992+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:16.055992+0000","last_peered":"2024-09-16T04:52:16.055992+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:16.055992+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4022272,"num_objects":491,"num_object_clones":0,"num_object_copies":1964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11821,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1687}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12116","reported_seq":12219,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.579232+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:15.579232+0000","last_peered":"2024-09-16T04:52:15.579232+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:15.579232+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3962124,"num_objects":484,"num_object_clones":0,"num_object_copies":1936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12114,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1729}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12234","reported_seq":16801,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:13.580630+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:13.580630+0000","last_peered":"2024-09-16T04:52:13.580630+0000","last_clean":"2024-09-16T04:52:13.580630+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:13.580630+0000","last_undegraded":"2024-09-16T04:52:13.580630+0000","last_fullsized":"2024-09-16T04:52:13.580630+0000","mapping_epoch":32,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:13.774360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4510988,"num_objects":551,"num_object_clones":0,"num_object_copies":2204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12232,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12233","reported_seq":16996,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:15.859462+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:15.859462+0000","last_peered":"2024-09-16T04:52:15.859462+0000","last_clean":"2024-09-16T04:52:15.859462+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:15.859462+0000","last_undegraded":"2024-09-16T04:52:15.859462+0000","last_fullsized":"2024-09-16T04:52:15.859462+0000","mapping_epoch":34,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:19.937793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4278808,"num_objects":523,"num_object_clones":0,"num_object_copies":2092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12230,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12196","reported_seq":12307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:14.513512+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:14.513512+0000","last_peered":"2024-09-16T04:52:14.513512+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:14.513512+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4601100,"num_objects":562,"num_object_clones":0,"num_object_copies":2248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12194,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1627}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12009","reported_seq":12145,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.170718+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:16.170718+0000","last_peered":"2024-09-16T04:52:16.170718+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:16.170718+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3944338,"num_objects":482,"num_object_clones":0,"num_object_copies":1928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12006,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1673}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11857","reported_seq":11955,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.956477+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:15.956477+0000","last_peered":"2024-09-16T04:52:15.956477+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:15.956477+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3881606,"num_objects":474,"num_object_clones":0,"num_object_copies":1896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11856,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1631}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11991","reported_seq":12085,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:14.388908+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:14.388908+0000","last_peered":"2024-09-16T04:52:14.388908+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:14.388908+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4438662,"num_objects":542,"num_object_clones":0,"num_object_copies":2168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11990,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1679}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12252","reported_seq":12390,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.785774+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:15.785774+0000","last_peered":"2024-09-16T04:52:15.785774+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:15.785774+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4338956,"num_objects":530,"num_object_clones":0,"num_object_copies":2120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12250,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1668}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11918","reported_seq":13444,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:14.611535+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:14.611535+0000","last_peered":"2024-09-16T04:52:14.611535+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:14.611535+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4347148,"num_objects":531,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11916,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":719,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11796","reported_seq":11896,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.595809+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:15.595809+0000","last_peered":"2024-09-16T04:52:15.595809+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:15.595809+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4183308,"num_objects":511,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11794,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1717}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67372736,"num_objects":8230,"num_object_clones":0,"num_object_copies":32920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8230,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193257,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7804,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":136294400,"data_stored":136294400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718308,"kb_used_data":32720,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93653532,"statfs":{"total":96636764160,"available":95901216768,"internally_reserved":0,"allocated":33505280,"data_stored":33306771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":702017231},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729364,"kb_used_data":34564,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93642476,"statfs":{"total":96636764160,"available":95889895424,"internally_reserved":0,"allocated":35393536,"data_stored":35195027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577036,"kb_used_data":35836,"kb_used_omap":4,"kb_used_meta":541179,"kb_avail":93794804,"statfs":{"total":96636764160,"available":96045879296,"internally_reserved":0,"allocated":36696064,"data_stored":36497555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554168015},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718992,"kb_used_data":33472,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93652848,"statfs":{"total":96636764160,"available":95900516352,"internally_reserved":0,"allocated":34275328,"data_stored":34076819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":701951694},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33378304,"data_stored":33378304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35704832,"data_stored":35704832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34258944,"data_stored":34258944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32952320,"data_stored":32952320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:16.883 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:17.516 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:17.516 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:17.550 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-09-16T04:52:16.786940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67372736,"num_objects":8230,"num_object_clones":0,"num_object_copies":32920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8230,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193257,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7804,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2743700,"kb_used_data":136592,"kb_used_omap":14,"kb_used_meta":2607025,"kb_avail":374743660,"statfs":{"total":386547056640,"available":383737507840,"internally_reserved":0,"allocated":139870208,"data_stored":139076172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2669594161},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,5,10,15,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20987904,"num_objects":-2562,"num_object_clones":0,"num_object_copies":-10248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2562,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005109"},"pg_stats":[{"pgid":"1.f","version":"37'12150","reported_seq":12272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.028986+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:16.028986+0000","last_peered":"2024-09-16T04:52:16.028986+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:16.028986+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4128768,"num_objects":504,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12150,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1682}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12172","reported_seq":12266,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.693783+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:15.693783+0000","last_peered":"2024-09-16T04:52:15.693783+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:15.693783+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4109580,"num_objects":502,"num_object_clones":0,"num_object_copies":2008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12170,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12171","reported_seq":12275,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:14.520562+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:14.520562+0000","last_peered":"2024-09-16T04:52:14.520562+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:14.520562+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4086430,"num_objects":500,"num_object_clones":0,"num_object_copies":2000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":500,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12170,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1719}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12124","reported_seq":16673,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:13.361997+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:13.361997+0000","last_peered":"2024-09-16T04:52:13.361997+0000","last_clean":"2024-09-16T04:52:13.361997+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:13.361997+0000","last_undegraded":"2024-09-16T04:52:13.361997+0000","last_fullsized":"2024-09-16T04:52:13.361997+0000","mapping_epoch":36,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:13.798055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4470028,"num_objects":546,"num_object_clones":0,"num_object_copies":2184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12244","reported_seq":12380,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.000867+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:16.000867+0000","last_peered":"2024-09-16T04:52:16.000867+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:16.000867+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4068620,"num_objects":497,"num_object_clones":0,"num_object_copies":1988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12242,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11821","reported_seq":11937,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.055992+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:16.055992+0000","last_peered":"2024-09-16T04:52:16.055992+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:16.055992+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4022272,"num_objects":491,"num_object_clones":0,"num_object_copies":1964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11821,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1687}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12116","reported_seq":12219,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.579232+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:15.579232+0000","last_peered":"2024-09-16T04:52:15.579232+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:15.579232+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3962124,"num_objects":484,"num_object_clones":0,"num_object_copies":1936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12114,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1729}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12234","reported_seq":16801,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:13.580630+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:13.580630+0000","last_peered":"2024-09-16T04:52:13.580630+0000","last_clean":"2024-09-16T04:52:13.580630+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:13.580630+0000","last_undegraded":"2024-09-16T04:52:13.580630+0000","last_fullsized":"2024-09-16T04:52:13.580630+0000","mapping_epoch":32,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:13.774360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4510988,"num_objects":551,"num_object_clones":0,"num_object_copies":2204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12232,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12233","reported_seq":16996,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:15.859462+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:15.859462+0000","last_peered":"2024-09-16T04:52:15.859462+0000","last_clean":"2024-09-16T04:52:15.859462+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:15.859462+0000","last_undegraded":"2024-09-16T04:52:15.859462+0000","last_fullsized":"2024-09-16T04:52:15.859462+0000","mapping_epoch":34,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:19.937793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4278808,"num_objects":523,"num_object_clones":0,"num_object_copies":2092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12230,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12196","reported_seq":12307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:14.513512+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:14.513512+0000","last_peered":"2024-09-16T04:52:14.513512+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:14.513512+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4601100,"num_objects":562,"num_object_clones":0,"num_object_copies":2248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12194,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1627}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12009","reported_seq":12145,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.170718+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:16.170718+0000","last_peered":"2024-09-16T04:52:16.170718+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:16.170718+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3944338,"num_objects":482,"num_object_clones":0,"num_object_copies":1928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12006,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1673}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11857","reported_seq":11955,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.956477+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:15.956477+0000","last_peered":"2024-09-16T04:52:15.956477+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:15.956477+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3881606,"num_objects":474,"num_object_clones":0,"num_object_copies":1896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11856,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1631}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11991","reported_seq":12085,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:14.388908+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:14.388908+0000","last_peered":"2024-09-16T04:52:14.388908+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:14.388908+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4438662,"num_objects":542,"num_object_clones":0,"num_object_copies":2168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11990,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1679}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12252","reported_seq":12390,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.785774+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:15.785774+0000","last_peered":"2024-09-16T04:52:15.785774+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:15.785774+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4338956,"num_objects":530,"num_object_clones":0,"num_object_copies":2120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12250,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1668}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11918","reported_seq":13444,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:14.611535+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:14.611535+0000","last_peered":"2024-09-16T04:52:14.611535+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:14.611535+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4347148,"num_objects":531,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11916,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":719,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11796","reported_seq":11896,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.595809+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:15.595809+0000","last_peered":"2024-09-16T04:52:15.595809+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:15.595809+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4183308,"num_objects":511,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11794,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1717}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67372736,"num_objects":8230,"num_object_clones":0,"num_object_copies":32920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8230,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193257,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7804,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":136294400,"data_stored":136294400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718308,"kb_used_data":32720,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93653532,"statfs":{"total":96636764160,"available":95901216768,"internally_reserved":0,"allocated":33505280,"data_stored":33306771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":702017231},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729364,"kb_used_data":34564,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93642476,"statfs":{"total":96636764160,"available":95889895424,"internally_reserved":0,"allocated":35393536,"data_stored":35195027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577036,"kb_used_data":35836,"kb_used_omap":4,"kb_used_meta":541179,"kb_avail":93794804,"statfs":{"total":96636764160,"available":96045879296,"internally_reserved":0,"allocated":36696064,"data_stored":36497555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554168015},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718992,"kb_used_data":33472,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93652848,"statfs":{"total":96636764160,"available":95900516352,"internally_reserved":0,"allocated":34275328,"data_stored":34076819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":701951694},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33378304,"data_stored":33378304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35704832,"data_stored":35704832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34258944,"data_stored":34258944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32952320,"data_stored":32952320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:17.551 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:52:17.551 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12150", 'reported_seq': 12272, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:16.028986+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:52:16.028986+0000', 'last_peered': '2024-09-16T04:52:16.028986+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:52:16.028986+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4128768, 'num_objects': 504, 'num_object_clones': 0, 'num_object_copies': 2016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 504, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12150, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1682}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:17.551 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:52:17.551 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12172", 'reported_seq': 12266, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:15.693783+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:52:15.693783+0000', 'last_peered': '2024-09-16T04:52:15.693783+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:52:15.693783+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4109580, 'num_objects': 502, 'num_object_clones': 0, 'num_object_copies': 2008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 502, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12170, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1701}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:17.551 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:52:17.551 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12171", 'reported_seq': 12275, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:14.520562+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:52:14.520562+0000', 'last_peered': '2024-09-16T04:52:14.520562+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:52:14.520562+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4086430, 'num_objects': 500, 'num_object_clones': 0, 'num_object_copies': 2000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 500, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12170, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1719}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:17.551 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:52:17.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12244", 'reported_seq': 12380, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:16.000867+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:52:16.000867+0000', 'last_peered': '2024-09-16T04:52:16.000867+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:52:16.000867+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4068620, 'num_objects': 497, 'num_object_clones': 0, 'num_object_copies': 1988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 497, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12242, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1751}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:17.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:52:17.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11821", 'reported_seq': 11937, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:16.055992+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:52:16.055992+0000', 'last_peered': '2024-09-16T04:52:16.055992+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:52:16.055992+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4022272, 'num_objects': 491, 'num_object_clones': 0, 'num_object_copies': 1964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11821, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1687}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:17.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:52:17.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12116", 'reported_seq': 12219, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:15.579232+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:52:15.579232+0000', 'last_peered': '2024-09-16T04:52:15.579232+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:52:15.579232+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3962124, 'num_objects': 484, 'num_object_clones': 0, 'num_object_copies': 1936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12114, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1729}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:17.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:52:17.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12196", 'reported_seq': 12307, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:14.513512+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:52:14.513512+0000', 'last_peered': '2024-09-16T04:52:14.513512+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:52:14.513512+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4601100, 'num_objects': 562, 'num_object_clones': 0, 'num_object_copies': 2248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 562, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12194, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1627}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:17.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:52:17.552 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12009", 'reported_seq': 12145, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:16.170718+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:52:16.170718+0000', 'last_peered': '2024-09-16T04:52:16.170718+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:52:16.170718+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3944338, 'num_objects': 482, 'num_object_clones': 0, 'num_object_copies': 1928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12006, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1673}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:17.552 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:52:17.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11857", 'reported_seq': 11955, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:15.956477+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:52:15.956477+0000', 'last_peered': '2024-09-16T04:52:15.956477+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:52:15.956477+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3881606, 'num_objects': 474, 'num_object_clones': 0, 'num_object_copies': 1896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11856, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1631}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:17.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:52:17.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11991", 'reported_seq': 12085, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:14.388908+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:52:14.388908+0000', 'last_peered': '2024-09-16T04:52:14.388908+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:52:14.388908+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4438662, 'num_objects': 542, 'num_object_clones': 0, 'num_object_copies': 2168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 542, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11990, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1679}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:17.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:52:17.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12252", 'reported_seq': 12390, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:15.785774+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:52:15.785774+0000', 'last_peered': '2024-09-16T04:52:15.785774+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:52:15.785774+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4338956, 'num_objects': 530, 'num_object_clones': 0, 'num_object_copies': 2120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 530, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12250, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1668}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:17.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:52:17.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11918", 'reported_seq': 13444, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:14.611535+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:52:14.611535+0000', 'last_peered': '2024-09-16T04:52:14.611535+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:52:14.611535+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4347148, 'num_objects': 531, 'num_object_clones': 0, 'num_object_copies': 2124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1417, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11916, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 719, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -886}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:17.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:52:17.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11796", 'reported_seq': 11896, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:15.595809+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:52:15.595809+0000', 'last_peered': '2024-09-16T04:52:15.595809+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:52:15.595809+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4183308, 'num_objects': 511, 'num_object_clones': 0, 'num_object_copies': 2044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11794, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1717}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:17.554 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:52:18.250 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:18.250 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27778/32920 objects degraded (84.380%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":750,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":8230,"data_bytes":67372736,"bytes_used":2809548800,"bytes_avail":383737507840,"bytes_total":386547056640,"degraded_objects":27778,"degraded_total":32920,"degraded_ratio":0.84380315917375459,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":213},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:52:18.279 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:52:18.279 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:18.901 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:18.901 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:18.939 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-09-16T04:52:18.787877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":66275008,"num_objects":8096,"num_object_clones":0,"num_object_copies":32384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8096,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193391,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7804,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160818,"ondisk_log_size":160818,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2743972,"kb_used_data":132692,"kb_used_omap":14,"kb_used_meta":2611185,"kb_avail":374743388,"statfs":{"total":386547056640,"available":383737229312,"internally_reserved":0,"allocated":135876608,"data_stored":135082572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2673854001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,6,12,15,7,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17137664,"num_objects":-2092,"num_object_clones":0,"num_object_copies":-8368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2092,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005394"},"pg_stats":[{"pgid":"1.f","version":"37'12150","reported_seq":12272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.028986+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:16.028986+0000","last_peered":"2024-09-16T04:52:16.028986+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:16.028986+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4128768,"num_objects":504,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12150,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1682}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12172","reported_seq":12266,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.693783+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:15.693783+0000","last_peered":"2024-09-16T04:52:15.693783+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:15.693783+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4109580,"num_objects":502,"num_object_clones":0,"num_object_copies":2008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12170,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12171","reported_seq":12275,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:14.520562+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:14.520562+0000","last_peered":"2024-09-16T04:52:14.520562+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:14.520562+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4086430,"num_objects":500,"num_object_clones":0,"num_object_copies":2000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":500,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12170,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1719}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12185","reported_seq":16735,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:18.157822+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:18.157822+0000","last_peered":"2024-09-16T04:52:18.157822+0000","last_clean":"2024-09-16T04:52:18.157822+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:18.157822+0000","last_undegraded":"2024-09-16T04:52:18.157822+0000","last_fullsized":"2024-09-16T04:52:18.157822+0000","mapping_epoch":36,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:19.709359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3970316,"num_objects":485,"num_object_clones":0,"num_object_copies":1940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12183,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12244","reported_seq":12380,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.000867+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:16.000867+0000","last_peered":"2024-09-16T04:52:16.000867+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:16.000867+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4068620,"num_objects":497,"num_object_clones":0,"num_object_copies":1988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12242,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11821","reported_seq":11937,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.055992+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:16.055992+0000","last_peered":"2024-09-16T04:52:16.055992+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:16.055992+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4022272,"num_objects":491,"num_object_clones":0,"num_object_copies":1964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11821,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1687}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12116","reported_seq":12219,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.579232+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:15.579232+0000","last_peered":"2024-09-16T04:52:15.579232+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:15.579232+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3962124,"num_objects":484,"num_object_clones":0,"num_object_copies":1936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12114,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1729}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12307","reported_seq":16874,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:18.426514+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:18.426514+0000","last_peered":"2024-09-16T04:52:18.426514+0000","last_clean":"2024-09-16T04:52:18.426514+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:18.426514+0000","last_undegraded":"2024-09-16T04:52:18.426514+0000","last_fullsized":"2024-09-16T04:52:18.426514+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:19.704983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3912972,"num_objects":478,"num_object_clones":0,"num_object_copies":1912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12305,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12233","reported_seq":16996,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:15.859462+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:15.859462+0000","last_peered":"2024-09-16T04:52:15.859462+0000","last_clean":"2024-09-16T04:52:15.859462+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:15.859462+0000","last_undegraded":"2024-09-16T04:52:15.859462+0000","last_fullsized":"2024-09-16T04:52:15.859462+0000","mapping_epoch":34,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:19.937793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4278808,"num_objects":523,"num_object_clones":0,"num_object_copies":2092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12230,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12196","reported_seq":12307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:14.513512+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:14.513512+0000","last_peered":"2024-09-16T04:52:14.513512+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:14.513512+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4601100,"num_objects":562,"num_object_clones":0,"num_object_copies":2248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12194,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1627}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12009","reported_seq":12145,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.170718+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:16.170718+0000","last_peered":"2024-09-16T04:52:16.170718+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:16.170718+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3944338,"num_objects":482,"num_object_clones":0,"num_object_copies":1928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12006,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1673}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11857","reported_seq":11955,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.956477+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:15.956477+0000","last_peered":"2024-09-16T04:52:15.956477+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:15.956477+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3881606,"num_objects":474,"num_object_clones":0,"num_object_copies":1896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11856,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1631}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11991","reported_seq":12085,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:14.388908+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:14.388908+0000","last_peered":"2024-09-16T04:52:14.388908+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:14.388908+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4438662,"num_objects":542,"num_object_clones":0,"num_object_copies":2168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11990,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1679}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12252","reported_seq":12390,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.785774+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:15.785774+0000","last_peered":"2024-09-16T04:52:15.785774+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:15.785774+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4338956,"num_objects":530,"num_object_clones":0,"num_object_copies":2120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12250,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1668}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11918","reported_seq":13444,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:14.611535+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:14.611535+0000","last_peered":"2024-09-16T04:52:14.611535+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:14.611535+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4347148,"num_objects":531,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11916,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":719,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11796","reported_seq":11896,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.595809+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:15.595809+0000","last_peered":"2024-09-16T04:52:15.595809+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:15.595809+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4183308,"num_objects":511,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11794,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1717}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":66275008,"num_objects":8096,"num_object_clones":0,"num_object_copies":32384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8096,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193391,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7804,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":132276224,"data_stored":132276224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160818,"ondisk_log_size":160818,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718308,"kb_used_data":32720,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93653532,"statfs":{"total":96636764160,"available":95901216768,"internally_reserved":0,"allocated":33505280,"data_stored":33306771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":702017231},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729364,"kb_used_data":34564,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93642476,"statfs":{"total":96636764160,"available":95889895424,"internally_reserved":0,"allocated":35393536,"data_stored":35195027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577308,"kb_used_data":31936,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93794532,"statfs":{"total":96636764160,"available":96045600768,"internally_reserved":0,"allocated":32702464,"data_stored":32503955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558427855},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718992,"kb_used_data":33472,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93652848,"statfs":{"total":96636764160,"available":95900516352,"internally_reserved":0,"allocated":34275328,"data_stored":34076819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":701951694},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33378304,"data_stored":33378304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31686656,"data_stored":31686656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34258944,"data_stored":34258944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32952320,"data_stored":32952320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:21.941 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:22.519 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:22.519 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:22.538 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-09-16T04:52:20.788709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":62842560,"num_objects":7677,"num_object_clones":0,"num_object_copies":30708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7677,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193810,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7846,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160737,"ondisk_log_size":160737,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2743788,"kb_used_data":124188,"kb_used_omap":14,"kb_used_meta":2619505,"kb_avail":374743572,"statfs":{"total":386547056640,"available":383737417728,"internally_reserved":0,"allocated":127168512,"data_stored":126374476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2682373681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,8,12,15,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19529728,"num_objects":-2384,"num_object_clones":0,"num_object_copies":-9536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2384,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005441"},"pg_stats":[{"pgid":"1.f","version":"37'12150","reported_seq":12272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.028986+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:16.028986+0000","last_peered":"2024-09-16T04:52:16.028986+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:16.028986+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4128768,"num_objects":504,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12150,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1682}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12230","reported_seq":12324,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:20.657404+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:20.657404+0000","last_peered":"2024-09-16T04:52:20.657404+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:20.657404+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3634444,"num_objects":444,"num_object_clones":0,"num_object_copies":1776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1759}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12224","reported_seq":12328,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:19.592258+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:19.592258+0000","last_peered":"2024-09-16T04:52:19.592258+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:19.592258+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3652254,"num_objects":447,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12223,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1772}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12185","reported_seq":16735,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:18.157822+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:18.157822+0000","last_peered":"2024-09-16T04:52:18.157822+0000","last_clean":"2024-09-16T04:52:18.157822+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:18.157822+0000","last_undegraded":"2024-09-16T04:52:18.157822+0000","last_fullsized":"2024-09-16T04:52:18.157822+0000","mapping_epoch":36,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:19.709359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3970316,"num_objects":485,"num_object_clones":0,"num_object_copies":1940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12183,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12244","reported_seq":12380,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.000867+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:16.000867+0000","last_peered":"2024-09-16T04:52:16.000867+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:16.000867+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4068620,"num_objects":497,"num_object_clones":0,"num_object_copies":1988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12242,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11821","reported_seq":11937,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.055992+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:16.055992+0000","last_peered":"2024-09-16T04:52:16.055992+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:16.055992+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4022272,"num_objects":491,"num_object_clones":0,"num_object_copies":1964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11821,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1687}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12186","reported_seq":12289,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:20.506255+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:20.506255+0000","last_peered":"2024-09-16T04:52:20.506255+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:20.506255+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3388684,"num_objects":414,"num_object_clones":0,"num_object_copies":1656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1799}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12307","reported_seq":16874,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:18.426514+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:18.426514+0000","last_peered":"2024-09-16T04:52:18.426514+0000","last_clean":"2024-09-16T04:52:18.426514+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:18.426514+0000","last_undegraded":"2024-09-16T04:52:18.426514+0000","last_fullsized":"2024-09-16T04:52:18.426514+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:19.704983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3912972,"num_objects":478,"num_object_clones":0,"num_object_copies":1912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12305,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12233","reported_seq":16996,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:15.859462+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:15.859462+0000","last_peered":"2024-09-16T04:52:15.859462+0000","last_clean":"2024-09-16T04:52:15.859462+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:15.859462+0000","last_undegraded":"2024-09-16T04:52:15.859462+0000","last_fullsized":"2024-09-16T04:52:15.859462+0000","mapping_epoch":34,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:19.937793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4278808,"num_objects":523,"num_object_clones":0,"num_object_copies":2092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12230,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12257","reported_seq":12369,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:19.648910+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:19.648910+0000","last_peered":"2024-09-16T04:52:19.648910+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:19.648910+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4101388,"num_objects":501,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12255,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1688}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12009","reported_seq":12145,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:16.170718+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:16.170718+0000","last_peered":"2024-09-16T04:52:16.170718+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:16.170718+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3944338,"num_objects":482,"num_object_clones":0,"num_object_copies":1928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12006,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1673}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11857","reported_seq":11955,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.956477+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:15.956477+0000","last_peered":"2024-09-16T04:52:15.956477+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:15.956477+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3881606,"num_objects":474,"num_object_clones":0,"num_object_copies":1896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11856,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1631}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12044","reported_seq":12138,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:19.369753+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:19.369753+0000","last_peered":"2024-09-16T04:52:19.369753+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:19.369753+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4004486,"num_objects":489,"num_object_clones":0,"num_object_copies":1956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12043,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1732}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12252","reported_seq":12390,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:15.785774+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:15.785774+0000","last_peered":"2024-09-16T04:52:15.785774+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:15.785774+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4338956,"num_objects":530,"num_object_clones":0,"num_object_copies":2120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12250,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1668}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11985","reported_seq":13594,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:19.697750+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:19.697750+0000","last_peered":"2024-09-16T04:52:19.697750+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:19.697750+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798284,"num_objects":464,"num_object_clones":0,"num_object_copies":1856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11983,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-911}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11853","reported_seq":11953,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:20.426215+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:20.426215+0000","last_peered":"2024-09-16T04:52:20.426215+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:20.426215+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3716364,"num_objects":454,"num_object_clones":0,"num_object_copies":1816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1774}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":62842560,"num_objects":7677,"num_object_clones":0,"num_object_copies":30708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7677,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193810,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7846,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":124387328,"data_stored":124387328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160737,"ondisk_log_size":160737,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718308,"kb_used_data":32720,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93653532,"statfs":{"total":96636764160,"available":95901216768,"internally_reserved":0,"allocated":33505280,"data_stored":33306771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":702017231},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728892,"kb_used_data":29932,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":93642948,"statfs":{"total":96636764160,"available":95890378752,"internally_reserved":0,"allocated":30650368,"data_stored":30451859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,0,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577308,"kb_used_data":31936,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93794532,"statfs":{"total":96636764160,"available":96045600768,"internally_reserved":0,"allocated":32702464,"data_stored":32503955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558427855},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719280,"kb_used_data":29600,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93652560,"statfs":{"total":96636764160,"available":95900221440,"internally_reserved":0,"allocated":30310400,"data_stored":30111891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":706211534},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,3,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29446144,"data_stored":29446144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31686656,"data_stored":31686656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30302208,"data_stored":30302208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32952320,"data_stored":32952320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:22.539 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:23.140 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:23.140 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:23.159 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2024-09-16T04:52:22.789550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":59516608,"num_objects":7271,"num_object_clones":0,"num_object_copies":29084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7271,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194216,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7846,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160743,"ondisk_log_size":160743,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2744188,"kb_used_data":120432,"kb_used_omap":14,"kb_used_meta":2623665,"kb_avail":374743172,"statfs":{"total":386547056640,"available":383737008128,"internally_reserved":0,"allocated":123322368,"data_stored":122528332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2686633521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,6,12,11,14,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19611648,"num_objects":-2394,"num_object_clones":0,"num_object_copies":-9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2394,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005410"},"pg_stats":[{"pgid":"1.f","version":"37'12211","reported_seq":12333,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:21.215999+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:21.215999+0000","last_peered":"2024-09-16T04:52:21.215999+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:21.215999+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3629056,"num_objects":443,"num_object_clones":0,"num_object_copies":1772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12211,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12230","reported_seq":12324,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:20.657404+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:20.657404+0000","last_peered":"2024-09-16T04:52:20.657404+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:20.657404+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3634444,"num_objects":444,"num_object_clones":0,"num_object_copies":1776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1759}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12224","reported_seq":12328,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:19.592258+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:19.592258+0000","last_peered":"2024-09-16T04:52:19.592258+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:19.592258+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3652254,"num_objects":447,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12223,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1772}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12185","reported_seq":16735,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:18.157822+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:18.157822+0000","last_peered":"2024-09-16T04:52:18.157822+0000","last_clean":"2024-09-16T04:52:18.157822+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:18.157822+0000","last_undegraded":"2024-09-16T04:52:18.157822+0000","last_fullsized":"2024-09-16T04:52:18.157822+0000","mapping_epoch":36,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:19.709359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3970316,"num_objects":485,"num_object_clones":0,"num_object_copies":1940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12183,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12304","reported_seq":12441,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:21.129893+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:21.129893+0000","last_peered":"2024-09-16T04:52:21.129893+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:21.129893+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3577100,"num_objects":437,"num_object_clones":0,"num_object_copies":1748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12302,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11883","reported_seq":11998,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:21.256804+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:21.256804+0000","last_peered":"2024-09-16T04:52:21.256804+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:21.256804+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3514368,"num_objects":429,"num_object_clones":0,"num_object_copies":1716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11883,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12186","reported_seq":12289,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:20.506255+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:20.506255+0000","last_peered":"2024-09-16T04:52:20.506255+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:20.506255+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3388684,"num_objects":414,"num_object_clones":0,"num_object_copies":1656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1799}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12307","reported_seq":16874,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:18.426514+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:18.426514+0000","last_peered":"2024-09-16T04:52:18.426514+0000","last_clean":"2024-09-16T04:52:18.426514+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:18.426514+0000","last_undegraded":"2024-09-16T04:52:18.426514+0000","last_fullsized":"2024-09-16T04:52:18.426514+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:19.704983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3912972,"num_objects":478,"num_object_clones":0,"num_object_copies":1912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12305,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12285","reported_seq":17048,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:21.012008+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:21.012008+0000","last_peered":"2024-09-16T04:52:21.012008+0000","last_clean":"2024-09-16T04:52:21.012008+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:21.012008+0000","last_undegraded":"2024-09-16T04:52:21.012008+0000","last_fullsized":"2024-09-16T04:52:21.012008+0000","mapping_epoch":34,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:25.857109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3852824,"num_objects":471,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12282,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12257","reported_seq":12369,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:19.648910+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:19.648910+0000","last_peered":"2024-09-16T04:52:19.648910+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:19.648910+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4101388,"num_objects":501,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12255,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1688}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12051","reported_seq":12189,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:21.180394+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:21.180394+0000","last_peered":"2024-09-16T04:52:21.180394+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:21.180394+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3600274,"num_objects":440,"num_object_clones":0,"num_object_copies":1760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1715}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11922","reported_seq":12020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:21.217924+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:21.217924+0000","last_peered":"2024-09-16T04:52:21.217924+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:21.217924+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3349126,"num_objects":409,"num_object_clones":0,"num_object_copies":1636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11921,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1696}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12044","reported_seq":12138,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:19.369753+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:19.369753+0000","last_peered":"2024-09-16T04:52:19.369753+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:19.369753+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4004486,"num_objects":489,"num_object_clones":0,"num_object_copies":1956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12043,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1732}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12316","reported_seq":12454,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:21.038380+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:21.038380+0000","last_peered":"2024-09-16T04:52:21.038380+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:21.038380+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3814668,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12314,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1732}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11985","reported_seq":13594,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:19.697750+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:19.697750+0000","last_peered":"2024-09-16T04:52:19.697750+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:19.697750+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798284,"num_objects":464,"num_object_clones":0,"num_object_copies":1856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11983,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-911}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11853","reported_seq":11953,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:20.426215+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:20.426215+0000","last_peered":"2024-09-16T04:52:20.426215+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:20.426215+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3716364,"num_objects":454,"num_object_clones":0,"num_object_copies":1816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1774}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":59516608,"num_objects":7271,"num_object_clones":0,"num_object_copies":29084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7271,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194216,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7846,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":120397824,"data_stored":120397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160743,"ondisk_log_size":160743,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718708,"kb_used_data":28964,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93653132,"statfs":{"total":96636764160,"available":95900807168,"internally_reserved":0,"allocated":29659136,"data_stored":29460627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":706277071},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,5,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728892,"kb_used_data":29932,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":93642948,"statfs":{"total":96636764160,"available":95890378752,"internally_reserved":0,"allocated":30650368,"data_stored":30451859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,0,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577308,"kb_used_data":31936,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93794532,"statfs":{"total":96636764160,"available":96045600768,"internally_reserved":0,"allocated":32702464,"data_stored":32503955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558427855},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719280,"kb_used_data":29600,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93652560,"statfs":{"total":96636764160,"available":95900221440,"internally_reserved":0,"allocated":30310400,"data_stored":30111891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":706211534},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,3,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29446144,"data_stored":29446144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31686656,"data_stored":31686656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30302208,"data_stored":30302208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28962816,"data_stored":28962816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:23.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:52:23.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12211", 'reported_seq': 12333, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:21.215999+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:52:21.215999+0000', 'last_peered': '2024-09-16T04:52:21.215999+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:52:21.215999+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3629056, 'num_objects': 443, 'num_object_clones': 0, 'num_object_copies': 1772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 443, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12211, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1743}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:23.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:52:23.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12230", 'reported_seq': 12324, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:20.657404+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:52:20.657404+0000', 'last_peered': '2024-09-16T04:52:20.657404+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:52:20.657404+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3634444, 'num_objects': 444, 'num_object_clones': 0, 'num_object_copies': 1776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 444, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12228, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1759}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:23.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:52:23.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12224", 'reported_seq': 12328, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:19.592258+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:52:19.592258+0000', 'last_peered': '2024-09-16T04:52:19.592258+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:52:19.592258+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3652254, 'num_objects': 447, 'num_object_clones': 0, 'num_object_copies': 1788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 447, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12223, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1772}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:23.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:52:23.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12304", 'reported_seq': 12441, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:21.129893+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:52:21.129893+0000', 'last_peered': '2024-09-16T04:52:21.129893+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:52:21.129893+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3577100, 'num_objects': 437, 'num_object_clones': 0, 'num_object_copies': 1748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 437, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12302, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1811}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:23.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:52:23.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11883", 'reported_seq': 11998, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:21.256804+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:52:21.256804+0000', 'last_peered': '2024-09-16T04:52:21.256804+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:52:21.256804+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3514368, 'num_objects': 429, 'num_object_clones': 0, 'num_object_copies': 1716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 429, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11883, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1749}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:23.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:52:23.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12186", 'reported_seq': 12289, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:20.506255+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:52:20.506255+0000', 'last_peered': '2024-09-16T04:52:20.506255+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:52:20.506255+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3388684, 'num_objects': 414, 'num_object_clones': 0, 'num_object_copies': 1656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12184, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1799}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:23.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:52:23.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12257", 'reported_seq': 12369, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:19.648910+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:52:19.648910+0000', 'last_peered': '2024-09-16T04:52:19.648910+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:52:19.648910+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4101388, 'num_objects': 501, 'num_object_clones': 0, 'num_object_copies': 2004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12255, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1688}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:23.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:52:23.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12051", 'reported_seq': 12189, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:21.180394+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:52:21.180394+0000', 'last_peered': '2024-09-16T04:52:21.180394+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:52:21.180394+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3600274, 'num_objects': 440, 'num_object_clones': 0, 'num_object_copies': 1760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12048, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1715}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:23.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:52:23.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11922", 'reported_seq': 12020, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:21.217924+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:52:21.217924+0000', 'last_peered': '2024-09-16T04:52:21.217924+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:52:21.217924+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3349126, 'num_objects': 409, 'num_object_clones': 0, 'num_object_copies': 1636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11921, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1696}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:23.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:52:23.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12044", 'reported_seq': 12138, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:19.369753+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:52:19.369753+0000', 'last_peered': '2024-09-16T04:52:19.369753+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:52:19.369753+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4004486, 'num_objects': 489, 'num_object_clones': 0, 'num_object_copies': 1956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 489, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12043, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1732}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:23.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:52:23.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12316", 'reported_seq': 12454, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:21.038380+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:52:21.038380+0000', 'last_peered': '2024-09-16T04:52:21.038380+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:52:21.038380+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3814668, 'num_objects': 466, 'num_object_clones': 0, 'num_object_copies': 1864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12314, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1732}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:23.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:52:23.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11985", 'reported_seq': 13594, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:19.697750+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:52:19.697750+0000', 'last_peered': '2024-09-16T04:52:19.697750+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:52:19.697750+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3798284, 'num_objects': 464, 'num_object_clones': 0, 'num_object_copies': 1856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1375, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11983, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 761, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -911}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:23.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:52:23.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11853", 'reported_seq': 11953, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:20.426215+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:52:20.426215+0000', 'last_peered': '2024-09-16T04:52:20.426215+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:52:20.426215+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3716364, 'num_objects': 454, 'num_object_clones': 0, 'num_object_copies': 1816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 454, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11851, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1774}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:23.163 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:52:23.931 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:23.931 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27736/29084 objects degraded (95.365%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":756,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7271,"data_bytes":59516608,"bytes_used":2810048512,"bytes_avail":383737008128,"bytes_total":386547056640,"degraded_objects":27736,"degraded_total":29084,"degraded_ratio":0.95365149222940448,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":199},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:52:23.952 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:52:23.952 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:24.572 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:24.573 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:24.597 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2024-09-16T04:52:22.789550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":59516608,"num_objects":7271,"num_object_clones":0,"num_object_copies":29084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7271,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194216,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7846,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160743,"ondisk_log_size":160743,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2744188,"kb_used_data":120432,"kb_used_omap":14,"kb_used_meta":2623665,"kb_avail":374743172,"statfs":{"total":386547056640,"available":383737008128,"internally_reserved":0,"allocated":123322368,"data_stored":122528332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2686633521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,6,12,11,14,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19611648,"num_objects":-2394,"num_object_clones":0,"num_object_copies":-9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2394,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005410"},"pg_stats":[{"pgid":"1.f","version":"37'12211","reported_seq":12333,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:21.215999+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:21.215999+0000","last_peered":"2024-09-16T04:52:21.215999+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:21.215999+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3629056,"num_objects":443,"num_object_clones":0,"num_object_copies":1772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12211,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12230","reported_seq":12324,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:20.657404+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:20.657404+0000","last_peered":"2024-09-16T04:52:20.657404+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:20.657404+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3634444,"num_objects":444,"num_object_clones":0,"num_object_copies":1776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1759}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12224","reported_seq":12328,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:19.592258+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:19.592258+0000","last_peered":"2024-09-16T04:52:19.592258+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:19.592258+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3652254,"num_objects":447,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12223,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1772}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12185","reported_seq":16735,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:18.157822+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:18.157822+0000","last_peered":"2024-09-16T04:52:18.157822+0000","last_clean":"2024-09-16T04:52:18.157822+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:18.157822+0000","last_undegraded":"2024-09-16T04:52:18.157822+0000","last_fullsized":"2024-09-16T04:52:18.157822+0000","mapping_epoch":36,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:19.709359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3970316,"num_objects":485,"num_object_clones":0,"num_object_copies":1940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12183,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12304","reported_seq":12441,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:21.129893+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:21.129893+0000","last_peered":"2024-09-16T04:52:21.129893+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:21.129893+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3577100,"num_objects":437,"num_object_clones":0,"num_object_copies":1748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12302,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11883","reported_seq":11998,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:21.256804+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:21.256804+0000","last_peered":"2024-09-16T04:52:21.256804+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:21.256804+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3514368,"num_objects":429,"num_object_clones":0,"num_object_copies":1716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11883,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12186","reported_seq":12289,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:20.506255+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:20.506255+0000","last_peered":"2024-09-16T04:52:20.506255+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:20.506255+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3388684,"num_objects":414,"num_object_clones":0,"num_object_copies":1656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1799}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12307","reported_seq":16874,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:18.426514+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:18.426514+0000","last_peered":"2024-09-16T04:52:18.426514+0000","last_clean":"2024-09-16T04:52:18.426514+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:18.426514+0000","last_undegraded":"2024-09-16T04:52:18.426514+0000","last_fullsized":"2024-09-16T04:52:18.426514+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:19.704983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3912972,"num_objects":478,"num_object_clones":0,"num_object_copies":1912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12305,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12285","reported_seq":17048,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:21.012008+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:21.012008+0000","last_peered":"2024-09-16T04:52:21.012008+0000","last_clean":"2024-09-16T04:52:21.012008+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:21.012008+0000","last_undegraded":"2024-09-16T04:52:21.012008+0000","last_fullsized":"2024-09-16T04:52:21.012008+0000","mapping_epoch":34,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:25.857109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3852824,"num_objects":471,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12282,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12257","reported_seq":12369,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:19.648910+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:19.648910+0000","last_peered":"2024-09-16T04:52:19.648910+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:19.648910+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4101388,"num_objects":501,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12255,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1688}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12051","reported_seq":12189,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:21.180394+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:21.180394+0000","last_peered":"2024-09-16T04:52:21.180394+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:21.180394+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3600274,"num_objects":440,"num_object_clones":0,"num_object_copies":1760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1715}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11922","reported_seq":12020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:21.217924+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:21.217924+0000","last_peered":"2024-09-16T04:52:21.217924+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:21.217924+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3349126,"num_objects":409,"num_object_clones":0,"num_object_copies":1636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11921,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1696}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12044","reported_seq":12138,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:19.369753+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:19.369753+0000","last_peered":"2024-09-16T04:52:19.369753+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:19.369753+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4004486,"num_objects":489,"num_object_clones":0,"num_object_copies":1956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12043,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1732}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12316","reported_seq":12454,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:21.038380+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:21.038380+0000","last_peered":"2024-09-16T04:52:21.038380+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:21.038380+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3814668,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12314,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1732}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11985","reported_seq":13594,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:19.697750+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:19.697750+0000","last_peered":"2024-09-16T04:52:19.697750+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:19.697750+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798284,"num_objects":464,"num_object_clones":0,"num_object_copies":1856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11983,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-911}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11853","reported_seq":11953,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:20.426215+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:20.426215+0000","last_peered":"2024-09-16T04:52:20.426215+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:20.426215+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3716364,"num_objects":454,"num_object_clones":0,"num_object_copies":1816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1774}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":59516608,"num_objects":7271,"num_object_clones":0,"num_object_copies":29084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7271,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194216,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7846,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":120397824,"data_stored":120397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160743,"ondisk_log_size":160743,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718708,"kb_used_data":28964,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93653132,"statfs":{"total":96636764160,"available":95900807168,"internally_reserved":0,"allocated":29659136,"data_stored":29460627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":706277071},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,5,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728892,"kb_used_data":29932,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":93642948,"statfs":{"total":96636764160,"available":95890378752,"internally_reserved":0,"allocated":30650368,"data_stored":30451859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,0,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577308,"kb_used_data":31936,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93794532,"statfs":{"total":96636764160,"available":96045600768,"internally_reserved":0,"allocated":32702464,"data_stored":32503955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558427855},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719280,"kb_used_data":29600,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93652560,"statfs":{"total":96636764160,"available":95900221440,"internally_reserved":0,"allocated":30310400,"data_stored":30111891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":706211534},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,3,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29446144,"data_stored":29446144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31686656,"data_stored":31686656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30302208,"data_stored":30302208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28962816,"data_stored":28962816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:27.599 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:28.193 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:28.194 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:28.219 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-09-16T04:52:26.791318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":51537600,"num_objects":6297,"num_object_clones":0,"num_object_copies":25188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195190,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7893,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160817,"ondisk_log_size":160817,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2713524,"kb_used_data":105000,"kb_used_omap":14,"kb_used_meta":2608433,"kb_avail":374773836,"statfs":{"total":386547056640,"available":383768408064,"internally_reserved":0,"allocated":107520000,"data_stored":106725964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2671035953},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,5,14,9,16,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20856832,"num_objects":-2546,"num_object_clones":0,"num_object_copies":-10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2546,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005292"},"pg_stats":[{"pgid":"1.f","version":"37'12278","reported_seq":12400,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.912110+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:25.912110+0000","last_peered":"2024-09-16T04:52:25.912110+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:25.912110+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3080192,"num_objects":376,"num_object_clones":0,"num_object_copies":1504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12278,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12298","reported_seq":12391,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.724534+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:25.724534+0000","last_peered":"2024-09-16T04:52:25.724534+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:25.724534+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3077388,"num_objects":376,"num_object_clones":0,"num_object_copies":1504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12296,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1827}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12281","reported_seq":12384,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:24.682729+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:24.682729+0000","last_peered":"2024-09-16T04:52:24.682729+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:24.682729+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3185310,"num_objects":390,"num_object_clones":0,"num_object_copies":1560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12280,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1829}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12258","reported_seq":16808,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:23.650397+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:23.650397+0000","last_peered":"2024-09-16T04:52:23.650397+0000","last_clean":"2024-09-16T04:52:23.650397+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:23.650397+0000","last_undegraded":"2024-09-16T04:52:23.650397+0000","last_fullsized":"2024-09-16T04:52:23.650397+0000","mapping_epoch":36,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:24.782344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3372300,"num_objects":412,"num_object_clones":0,"num_object_copies":1648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12256,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12364","reported_seq":12502,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.999464+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:25.999464+0000","last_peered":"2024-09-16T04:52:25.999464+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:25.999464+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3085580,"num_objects":377,"num_object_clones":0,"num_object_copies":1508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12362,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1871}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11949","reported_seq":12065,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:26.261092+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:26.261092+0000","last_peered":"2024-09-16T04:52:26.261092+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:26.261092+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2973696,"num_objects":363,"num_object_clones":0,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1815}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12247","reported_seq":12350,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.739141+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:25.739141+0000","last_peered":"2024-09-16T04:52:25.739141+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:25.739141+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2888972,"num_objects":353,"num_object_clones":0,"num_object_copies":1412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12245,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1860}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12359","reported_seq":16926,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:23.298929+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:23.298929+0000","last_peered":"2024-09-16T04:52:23.298929+0000","last_clean":"2024-09-16T04:52:23.298929+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:23.298929+0000","last_undegraded":"2024-09-16T04:52:23.298929+0000","last_fullsized":"2024-09-16T04:52:23.298929+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:24.782316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3486988,"num_objects":426,"num_object_clones":0,"num_object_copies":1704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12357,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12349","reported_seq":17112,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:26.259270+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:26.259270+0000","last_peered":"2024-09-16T04:52:26.259270+0000","last_clean":"2024-09-16T04:52:26.259270+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:26.259270+0000","last_undegraded":"2024-09-16T04:52:26.259270+0000","last_fullsized":"2024-09-16T04:52:26.259270+0000","mapping_epoch":34,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:30.950331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3328536,"num_objects":407,"num_object_clones":0,"num_object_copies":1628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12346,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12305","reported_seq":12417,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:24.594581+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:24.594581+0000","last_peered":"2024-09-16T04:52:24.594581+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:24.594581+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3708172,"num_objects":453,"num_object_clones":0,"num_object_copies":1812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1736}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12113","reported_seq":12251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:26.261274+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:26.261274+0000","last_peered":"2024-09-16T04:52:26.261274+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:26.261274+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092370,"num_objects":378,"num_object_clones":0,"num_object_copies":1512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12110,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1777}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11971","reported_seq":12069,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.915763+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:25.915763+0000","last_peered":"2024-09-16T04:52:25.915763+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:25.915763+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2947718,"num_objects":360,"num_object_clones":0,"num_object_copies":1440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11970,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1745}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12101","reported_seq":12195,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:24.462001+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:24.462001+0000","last_peered":"2024-09-16T04:52:24.462001+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:24.462001+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3537542,"num_objects":432,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12100,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1789}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12381","reported_seq":12519,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:26.143837+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:26.143837+0000","last_peered":"2024-09-16T04:52:26.143837+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:26.143837+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3282188,"num_objects":401,"num_object_clones":0,"num_object_copies":1604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12379,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1797}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12046","reported_seq":13750,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:24.684288+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:24.684288+0000","last_peered":"2024-09-16T04:52:24.684288+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:24.684288+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3298572,"num_objects":403,"num_object_clones":0,"num_object_copies":1612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12044,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":808,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-925}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11917","reported_seq":12016,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.736993+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:25.736993+0000","last_peered":"2024-09-16T04:52:25.736993+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:25.736993+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3192076,"num_objects":390,"num_object_clones":0,"num_object_copies":1560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11915,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1838}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":51537600,"num_objects":6297,"num_object_clones":0,"num_object_copies":25188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195190,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7893,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104677376,"data_stored":104677376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160817,"ondisk_log_size":160817,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702388,"kb_used_data":25116,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93669452,"statfs":{"total":96636764160,"available":95917518848,"internally_reserved":0,"allocated":25718784,"data_stored":25520275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":693497551},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,5,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729256,"kb_used_data":26136,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93642584,"statfs":{"total":96636764160,"available":95890006016,"internally_reserved":0,"allocated":26763264,"data_stored":26564755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578932,"kb_used_data":28000,"kb_used_omap":4,"kb_used_meta":550907,"kb_avail":93792908,"statfs":{"total":96636764160,"available":96043937792,"internally_reserved":0,"allocated":28672000,"data_stored":28473491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":564129487},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702948,"kb_used_data":25748,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93668892,"statfs":{"total":96636764160,"available":95916945408,"internally_reserved":0,"allocated":26365952,"data_stored":26167443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":693432014},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1,1,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25468928,"data_stored":25468928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27713536,"data_stored":27713536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":26419200,"data_stored":26419200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25075712,"data_stored":25075712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:28.220 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:28.814 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:28.814 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:28.845 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-09-16T04:52:28.792240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":51537600,"num_objects":6297,"num_object_clones":0,"num_object_copies":25188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195190,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7893,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160817,"ondisk_log_size":160817,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2713524,"kb_used_data":105000,"kb_used_omap":14,"kb_used_meta":2608433,"kb_avail":374773836,"statfs":{"total":386547056640,"available":383768408064,"internally_reserved":0,"allocated":107520000,"data_stored":106725964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2671035953},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,5,14,9,16,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15835136,"num_objects":-1933,"num_object_clones":0,"num_object_copies":-7732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1933,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005299"},"pg_stats":[{"pgid":"1.f","version":"37'12278","reported_seq":12400,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.912110+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:25.912110+0000","last_peered":"2024-09-16T04:52:25.912110+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:25.912110+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3080192,"num_objects":376,"num_object_clones":0,"num_object_copies":1504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12278,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12298","reported_seq":12391,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.724534+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:25.724534+0000","last_peered":"2024-09-16T04:52:25.724534+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:25.724534+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3077388,"num_objects":376,"num_object_clones":0,"num_object_copies":1504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12296,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1827}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12281","reported_seq":12384,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:24.682729+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:24.682729+0000","last_peered":"2024-09-16T04:52:24.682729+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:24.682729+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3185310,"num_objects":390,"num_object_clones":0,"num_object_copies":1560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12280,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1829}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12258","reported_seq":16808,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:23.650397+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:23.650397+0000","last_peered":"2024-09-16T04:52:23.650397+0000","last_clean":"2024-09-16T04:52:23.650397+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:23.650397+0000","last_undegraded":"2024-09-16T04:52:23.650397+0000","last_fullsized":"2024-09-16T04:52:23.650397+0000","mapping_epoch":36,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:24.782344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3372300,"num_objects":412,"num_object_clones":0,"num_object_copies":1648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12256,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12364","reported_seq":12502,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.999464+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:25.999464+0000","last_peered":"2024-09-16T04:52:25.999464+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:25.999464+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3085580,"num_objects":377,"num_object_clones":0,"num_object_copies":1508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12362,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1871}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11949","reported_seq":12065,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:26.261092+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:26.261092+0000","last_peered":"2024-09-16T04:52:26.261092+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:26.261092+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2973696,"num_objects":363,"num_object_clones":0,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1815}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12247","reported_seq":12350,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.739141+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:25.739141+0000","last_peered":"2024-09-16T04:52:25.739141+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:25.739141+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2888972,"num_objects":353,"num_object_clones":0,"num_object_copies":1412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12245,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1860}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12359","reported_seq":16926,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:23.298929+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:23.298929+0000","last_peered":"2024-09-16T04:52:23.298929+0000","last_clean":"2024-09-16T04:52:23.298929+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:23.298929+0000","last_undegraded":"2024-09-16T04:52:23.298929+0000","last_fullsized":"2024-09-16T04:52:23.298929+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:24.782316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3486988,"num_objects":426,"num_object_clones":0,"num_object_copies":1704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12357,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12349","reported_seq":17112,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:26.259270+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:26.259270+0000","last_peered":"2024-09-16T04:52:26.259270+0000","last_clean":"2024-09-16T04:52:26.259270+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:26.259270+0000","last_undegraded":"2024-09-16T04:52:26.259270+0000","last_fullsized":"2024-09-16T04:52:26.259270+0000","mapping_epoch":34,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:30.950331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3328536,"num_objects":407,"num_object_clones":0,"num_object_copies":1628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12346,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12305","reported_seq":12417,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:24.594581+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:24.594581+0000","last_peered":"2024-09-16T04:52:24.594581+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:24.594581+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3708172,"num_objects":453,"num_object_clones":0,"num_object_copies":1812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1736}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12113","reported_seq":12251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:26.261274+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:26.261274+0000","last_peered":"2024-09-16T04:52:26.261274+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:26.261274+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092370,"num_objects":378,"num_object_clones":0,"num_object_copies":1512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12110,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1777}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11971","reported_seq":12069,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.915763+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:25.915763+0000","last_peered":"2024-09-16T04:52:25.915763+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:25.915763+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2947718,"num_objects":360,"num_object_clones":0,"num_object_copies":1440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11970,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1745}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12101","reported_seq":12195,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:24.462001+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:24.462001+0000","last_peered":"2024-09-16T04:52:24.462001+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:24.462001+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3537542,"num_objects":432,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12100,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1789}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12381","reported_seq":12519,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:26.143837+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:26.143837+0000","last_peered":"2024-09-16T04:52:26.143837+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:26.143837+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3282188,"num_objects":401,"num_object_clones":0,"num_object_copies":1604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12379,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1797}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12046","reported_seq":13750,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:24.684288+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:24.684288+0000","last_peered":"2024-09-16T04:52:24.684288+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:24.684288+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3298572,"num_objects":403,"num_object_clones":0,"num_object_copies":1612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12044,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":808,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-925}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11917","reported_seq":12016,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.736993+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:25.736993+0000","last_peered":"2024-09-16T04:52:25.736993+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:25.736993+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3192076,"num_objects":390,"num_object_clones":0,"num_object_copies":1560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11915,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1838}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":51537600,"num_objects":6297,"num_object_clones":0,"num_object_copies":25188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195190,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7893,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104677376,"data_stored":104677376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160817,"ondisk_log_size":160817,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702388,"kb_used_data":25116,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93669452,"statfs":{"total":96636764160,"available":95917518848,"internally_reserved":0,"allocated":25718784,"data_stored":25520275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":693497551},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,5,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729256,"kb_used_data":26136,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93642584,"statfs":{"total":96636764160,"available":95890006016,"internally_reserved":0,"allocated":26763264,"data_stored":26564755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578932,"kb_used_data":28000,"kb_used_omap":4,"kb_used_meta":550907,"kb_avail":93792908,"statfs":{"total":96636764160,"available":96043937792,"internally_reserved":0,"allocated":28672000,"data_stored":28473491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":564129487},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702948,"kb_used_data":25748,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93668892,"statfs":{"total":96636764160,"available":95916945408,"internally_reserved":0,"allocated":26365952,"data_stored":26167443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":693432014},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1,1,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25468928,"data_stored":25468928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27713536,"data_stored":27713536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":26419200,"data_stored":26419200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25075712,"data_stored":25075712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:28.846 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:52:28.846 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12278", 'reported_seq': 12400, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:25.912110+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:52:25.912110+0000', 'last_peered': '2024-09-16T04:52:25.912110+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:52:25.912110+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3080192, 'num_objects': 376, 'num_object_clones': 0, 'num_object_copies': 1504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 376, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12278, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1810}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:28.846 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:52:28.847 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12298", 'reported_seq': 12391, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:25.724534+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:52:25.724534+0000', 'last_peered': '2024-09-16T04:52:25.724534+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:52:25.724534+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3077388, 'num_objects': 376, 'num_object_clones': 0, 'num_object_copies': 1504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 376, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12296, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1827}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:28.847 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:52:28.847 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12281", 'reported_seq': 12384, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:24.682729+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:52:24.682729+0000', 'last_peered': '2024-09-16T04:52:24.682729+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:52:24.682729+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3185310, 'num_objects': 390, 'num_object_clones': 0, 'num_object_copies': 1560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 390, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12280, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1829}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:28.847 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:52:28.847 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12364", 'reported_seq': 12502, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:25.999464+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:52:25.999464+0000', 'last_peered': '2024-09-16T04:52:25.999464+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:52:25.999464+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3085580, 'num_objects': 377, 'num_object_clones': 0, 'num_object_copies': 1508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12362, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1871}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:28.847 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:52:28.847 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11949", 'reported_seq': 12065, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:26.261092+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:52:26.261092+0000', 'last_peered': '2024-09-16T04:52:26.261092+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:52:26.261092+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2973696, 'num_objects': 363, 'num_object_clones': 0, 'num_object_copies': 1452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11949, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1815}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:28.847 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:52:28.847 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12247", 'reported_seq': 12350, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:25.739141+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:52:25.739141+0000', 'last_peered': '2024-09-16T04:52:25.739141+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:52:25.739141+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2888972, 'num_objects': 353, 'num_object_clones': 0, 'num_object_copies': 1412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 353, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12245, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1860}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:28.847 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:52:28.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12305", 'reported_seq': 12417, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:24.594581+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:52:24.594581+0000', 'last_peered': '2024-09-16T04:52:24.594581+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:52:24.594581+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3708172, 'num_objects': 453, 'num_object_clones': 0, 'num_object_copies': 1812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12303, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1736}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:28.848 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:52:28.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12113", 'reported_seq': 12251, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:26.261274+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:52:26.261274+0000', 'last_peered': '2024-09-16T04:52:26.261274+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:52:26.261274+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3092370, 'num_objects': 378, 'num_object_clones': 0, 'num_object_copies': 1512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12110, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1777}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:28.848 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:52:28.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11971", 'reported_seq': 12069, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:25.915763+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:52:25.915763+0000', 'last_peered': '2024-09-16T04:52:25.915763+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:52:25.915763+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2947718, 'num_objects': 360, 'num_object_clones': 0, 'num_object_copies': 1440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11970, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1745}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:28.848 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:52:28.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12101", 'reported_seq': 12195, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:24.462001+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:52:24.462001+0000', 'last_peered': '2024-09-16T04:52:24.462001+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:52:24.462001+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3537542, 'num_objects': 432, 'num_object_clones': 0, 'num_object_copies': 1728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 432, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12100, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1789}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:28.848 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:52:28.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12381", 'reported_seq': 12519, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:26.143837+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:52:26.143837+0000', 'last_peered': '2024-09-16T04:52:26.143837+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:52:26.143837+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3282188, 'num_objects': 401, 'num_object_clones': 0, 'num_object_copies': 1604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 401, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12379, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1797}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:28.848 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:52:28.848 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12046", 'reported_seq': 13750, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:24.684288+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:52:24.684288+0000', 'last_peered': '2024-09-16T04:52:24.684288+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:52:24.684288+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3298572, 'num_objects': 403, 'num_object_clones': 0, 'num_object_copies': 1612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1328, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12044, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 808, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -925}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:28.849 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:52:28.849 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11917", 'reported_seq': 12016, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:25.736993+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:52:25.736993+0000', 'last_peered': '2024-09-16T04:52:25.736993+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:52:25.736993+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3192076, 'num_objects': 390, 'num_object_clones': 0, 'num_object_copies': 1560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11915, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1838}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:28.849 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:52:29.573 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:29.573 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27689/25188 objects degraded (109.929%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":761,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":6297,"data_bytes":51537600,"bytes_used":2778648576,"bytes_avail":383768408064,"bytes_total":386547056640,"degraded_objects":27689,"degraded_total":25188,"degraded_ratio":1.0992933142766397,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":161},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:52:29.607 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:52:29.607 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:30.138 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:30.139 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:30.182 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-09-16T04:52:28.792240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":51537600,"num_objects":6297,"num_object_clones":0,"num_object_copies":25188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195190,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7893,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160817,"ondisk_log_size":160817,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2713524,"kb_used_data":105000,"kb_used_omap":14,"kb_used_meta":2608433,"kb_avail":374773836,"statfs":{"total":386547056640,"available":383768408064,"internally_reserved":0,"allocated":107520000,"data_stored":106725964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2671035953},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,5,14,9,16,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15835136,"num_objects":-1933,"num_object_clones":0,"num_object_copies":-7732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1933,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005299"},"pg_stats":[{"pgid":"1.f","version":"37'12278","reported_seq":12400,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.912110+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:25.912110+0000","last_peered":"2024-09-16T04:52:25.912110+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:25.912110+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3080192,"num_objects":376,"num_object_clones":0,"num_object_copies":1504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12278,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12298","reported_seq":12391,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.724534+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:25.724534+0000","last_peered":"2024-09-16T04:52:25.724534+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:25.724534+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3077388,"num_objects":376,"num_object_clones":0,"num_object_copies":1504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12296,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1827}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12281","reported_seq":12384,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:24.682729+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:24.682729+0000","last_peered":"2024-09-16T04:52:24.682729+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:24.682729+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3185310,"num_objects":390,"num_object_clones":0,"num_object_copies":1560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12280,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1829}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12258","reported_seq":16808,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:23.650397+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:23.650397+0000","last_peered":"2024-09-16T04:52:23.650397+0000","last_clean":"2024-09-16T04:52:23.650397+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:23.650397+0000","last_undegraded":"2024-09-16T04:52:23.650397+0000","last_fullsized":"2024-09-16T04:52:23.650397+0000","mapping_epoch":36,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:24.782344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3372300,"num_objects":412,"num_object_clones":0,"num_object_copies":1648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12256,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12364","reported_seq":12502,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.999464+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:25.999464+0000","last_peered":"2024-09-16T04:52:25.999464+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:25.999464+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3085580,"num_objects":377,"num_object_clones":0,"num_object_copies":1508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12362,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1871}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11949","reported_seq":12065,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:26.261092+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:26.261092+0000","last_peered":"2024-09-16T04:52:26.261092+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:26.261092+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2973696,"num_objects":363,"num_object_clones":0,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1815}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12247","reported_seq":12350,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.739141+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:25.739141+0000","last_peered":"2024-09-16T04:52:25.739141+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:25.739141+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2888972,"num_objects":353,"num_object_clones":0,"num_object_copies":1412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12245,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1860}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12359","reported_seq":16926,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:23.298929+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:23.298929+0000","last_peered":"2024-09-16T04:52:23.298929+0000","last_clean":"2024-09-16T04:52:23.298929+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:23.298929+0000","last_undegraded":"2024-09-16T04:52:23.298929+0000","last_fullsized":"2024-09-16T04:52:23.298929+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:24.782316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3486988,"num_objects":426,"num_object_clones":0,"num_object_copies":1704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12357,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12349","reported_seq":17112,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:26.259270+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:26.259270+0000","last_peered":"2024-09-16T04:52:26.259270+0000","last_clean":"2024-09-16T04:52:26.259270+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:26.259270+0000","last_undegraded":"2024-09-16T04:52:26.259270+0000","last_fullsized":"2024-09-16T04:52:26.259270+0000","mapping_epoch":34,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:30.950331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3328536,"num_objects":407,"num_object_clones":0,"num_object_copies":1628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12346,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12305","reported_seq":12417,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:24.594581+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:24.594581+0000","last_peered":"2024-09-16T04:52:24.594581+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:24.594581+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3708172,"num_objects":453,"num_object_clones":0,"num_object_copies":1812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1736}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12113","reported_seq":12251,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:26.261274+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:26.261274+0000","last_peered":"2024-09-16T04:52:26.261274+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:26.261274+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092370,"num_objects":378,"num_object_clones":0,"num_object_copies":1512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12110,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1777}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11971","reported_seq":12069,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.915763+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:25.915763+0000","last_peered":"2024-09-16T04:52:25.915763+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:25.915763+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2947718,"num_objects":360,"num_object_clones":0,"num_object_copies":1440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11970,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1745}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12101","reported_seq":12195,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:24.462001+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:24.462001+0000","last_peered":"2024-09-16T04:52:24.462001+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:24.462001+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3537542,"num_objects":432,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12100,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1789}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12381","reported_seq":12519,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:26.143837+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:26.143837+0000","last_peered":"2024-09-16T04:52:26.143837+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:26.143837+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3282188,"num_objects":401,"num_object_clones":0,"num_object_copies":1604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12379,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1797}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12046","reported_seq":13750,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:24.684288+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:24.684288+0000","last_peered":"2024-09-16T04:52:24.684288+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:24.684288+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3298572,"num_objects":403,"num_object_clones":0,"num_object_copies":1612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12044,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":808,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-925}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11917","reported_seq":12016,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:25.736993+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:25.736993+0000","last_peered":"2024-09-16T04:52:25.736993+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:25.736993+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3192076,"num_objects":390,"num_object_clones":0,"num_object_copies":1560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11915,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1838}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":51537600,"num_objects":6297,"num_object_clones":0,"num_object_copies":25188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195190,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7893,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104677376,"data_stored":104677376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160817,"ondisk_log_size":160817,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702388,"kb_used_data":25116,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93669452,"statfs":{"total":96636764160,"available":95917518848,"internally_reserved":0,"allocated":25718784,"data_stored":25520275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":693497551},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,5,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729256,"kb_used_data":26136,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93642584,"statfs":{"total":96636764160,"available":95890006016,"internally_reserved":0,"allocated":26763264,"data_stored":26564755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578932,"kb_used_data":28000,"kb_used_omap":4,"kb_used_meta":550907,"kb_avail":93792908,"statfs":{"total":96636764160,"available":96043937792,"internally_reserved":0,"allocated":28672000,"data_stored":28473491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":564129487},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702948,"kb_used_data":25748,"kb_used_omap":4,"kb_used_meta":677179,"kb_avail":93668892,"statfs":{"total":96636764160,"available":95916945408,"internally_reserved":0,"allocated":26365952,"data_stored":26167443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":693432014},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1,1,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25468928,"data_stored":25468928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27713536,"data_stored":27713536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":26419200,"data_stored":26419200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25075712,"data_stored":25075712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:33.184 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:33.810 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:33.810 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:33.846 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2024-09-16T04:52:32.794786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":43853504,"num_objects":5359,"num_object_clones":0,"num_object_copies":21436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196128,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7951,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160755,"ondisk_log_size":160755,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2714244,"kb_used_data":89088,"kb_used_omap":14,"kb_used_meta":2625073,"kb_avail":374773116,"statfs":{"total":386547056640,"available":383767670784,"internally_reserved":0,"allocated":91226112,"data_stored":90432076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2688075313},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,6,10,11,14,6,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18989056,"num_objects":-2318,"num_object_clones":0,"num_object_copies":-9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2318,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006077"},"pg_stats":[{"pgid":"1.f","version":"37'12328","reported_seq":12450,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.082841+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:31.082841+0000","last_peered":"2024-09-16T04:52:31.082841+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:31.082841+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2670592,"num_objects":326,"num_object_clones":0,"num_object_copies":1304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12328,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1860}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12365","reported_seq":12459,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:30.852214+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:30.852214+0000","last_peered":"2024-09-16T04:52:30.852214+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:30.852214+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2528524,"num_objects":309,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12363,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1894}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12342","reported_seq":12446,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:29.640845+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:29.640845+0000","last_peered":"2024-09-16T04:52:29.640845+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:29.640845+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2685598,"num_objects":329,"num_object_clones":0,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12341,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1890}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12328","reported_seq":16875,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:28.740613+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:28.740613+0000","last_peered":"2024-09-16T04:52:28.740613+0000","last_clean":"2024-09-16T04:52:28.740613+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:28.740613+0000","last_undegraded":"2024-09-16T04:52:28.740613+0000","last_fullsized":"2024-09-16T04:52:28.740613+0000","mapping_epoch":36,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:29.802644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2798860,"num_objects":342,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12326,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12418","reported_seq":12556,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.361950+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:31.361950+0000","last_peered":"2024-09-16T04:52:31.361950+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:31.361950+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2643212,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12416,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1925}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12007","reported_seq":12123,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.309410+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:31.309410+0000","last_peered":"2024-09-16T04:52:31.309410+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:31.309410+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2498560,"num_objects":305,"num_object_clones":0,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12007,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12288","reported_seq":12391,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:30.587420+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:30.587420+0000","last_peered":"2024-09-16T04:52:30.587420+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:30.587420+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2553100,"num_objects":312,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12286,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1901}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12412","reported_seq":16977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:28.660835+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:28.660835+0000","last_peered":"2024-09-16T04:52:28.660835+0000","last_clean":"2024-09-16T04:52:28.660835+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:28.660835+0000","last_undegraded":"2024-09-16T04:52:28.660835+0000","last_fullsized":"2024-09-16T04:52:28.660835+0000","mapping_epoch":32,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:29.802615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3052812,"num_objects":373,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12410,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12411","reported_seq":17174,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:31.178936+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:31.178936+0000","last_peered":"2024-09-16T04:52:31.178936+0000","last_clean":"2024-09-16T04:52:31.178936+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:31.178936+0000","last_undegraded":"2024-09-16T04:52:31.178936+0000","last_fullsized":"2024-09-16T04:52:31.178936+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2820632,"num_objects":345,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12408,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12375","reported_seq":12487,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:29.613066+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:29.613066+0000","last_peered":"2024-09-16T04:52:29.613066+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:29.613066+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3134732,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12373,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1806}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12160","reported_seq":12298,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.252224+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:31.252224+0000","last_peered":"2024-09-16T04:52:31.252224+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:31.252224+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2707346,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12157,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12028","reported_seq":12126,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.354437+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:31.354437+0000","last_peered":"2024-09-16T04:52:31.354437+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:31.354437+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2480774,"num_objects":303,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12027,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1802}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12170","reported_seq":12264,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:29.571361+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:29.571361+0000","last_peered":"2024-09-16T04:52:29.571361+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:29.571361+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2972294,"num_objects":363,"num_object_clones":0,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12169,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1858}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12442","reported_seq":12579,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.362737+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:31.362737+0000","last_peered":"2024-09-16T04:52:31.362737+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:31.362737+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2782476,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12440,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1858}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12099","reported_seq":13919,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:29.648643+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:29.648643+0000","last_peered":"2024-09-16T04:52:29.648643+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:29.648643+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2864396,"num_objects":350,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12097,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":866,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-920}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11982","reported_seq":12082,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:30.853667+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:30.853667+0000","last_peered":"2024-09-16T04:52:30.853667+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:30.853667+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2659596,"num_objects":325,"num_object_clones":0,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11980,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1903}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":43853504,"num_objects":5359,"num_object_clones":0,"num_object_copies":21436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196128,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7951,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":88875008,"data_stored":88875008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160755,"ondisk_log_size":160755,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702812,"kb_used_data":21388,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93669028,"statfs":{"total":96636764160,"available":95917084672,"internally_reserved":0,"allocated":21901312,"data_stored":21702803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697757391},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729532,"kb_used_data":22252,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93642308,"statfs":{"total":96636764160,"available":95889723392,"internally_reserved":0,"allocated":22786048,"data_stored":22587539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578628,"kb_used_data":23536,"kb_used_omap":4,"kb_used_meta":555067,"kb_avail":93793212,"statfs":{"total":96636764160,"available":96044249088,"internally_reserved":0,"allocated":24100864,"data_stored":23902355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":568389327},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,3,1,3,1,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703272,"kb_used_data":21912,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93668568,"statfs":{"total":96636764160,"available":95916613632,"internally_reserved":0,"allocated":22437888,"data_stored":22239379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":697691854},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21590016,"data_stored":21590016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":23658496,"data_stored":23658496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":22466560,"data_stored":22466560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":21159936,"data_stored":21159936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:33.847 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:34.491 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:34.492 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:34.516 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2024-09-16T04:52:32.794786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":43853504,"num_objects":5359,"num_object_clones":0,"num_object_copies":21436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196128,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7951,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160755,"ondisk_log_size":160755,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2714244,"kb_used_data":89088,"kb_used_omap":14,"kb_used_meta":2625073,"kb_avail":374773116,"statfs":{"total":386547056640,"available":383767670784,"internally_reserved":0,"allocated":91226112,"data_stored":90432076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2688075313},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,6,10,11,14,6,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18989056,"num_objects":-2318,"num_object_clones":0,"num_object_copies":-9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2318,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006077"},"pg_stats":[{"pgid":"1.f","version":"37'12328","reported_seq":12450,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.082841+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:31.082841+0000","last_peered":"2024-09-16T04:52:31.082841+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:31.082841+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2670592,"num_objects":326,"num_object_clones":0,"num_object_copies":1304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12328,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1860}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12365","reported_seq":12459,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:30.852214+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:30.852214+0000","last_peered":"2024-09-16T04:52:30.852214+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:30.852214+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2528524,"num_objects":309,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12363,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1894}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12342","reported_seq":12446,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:29.640845+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:29.640845+0000","last_peered":"2024-09-16T04:52:29.640845+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:29.640845+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2685598,"num_objects":329,"num_object_clones":0,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12341,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1890}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12328","reported_seq":16875,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:28.740613+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:28.740613+0000","last_peered":"2024-09-16T04:52:28.740613+0000","last_clean":"2024-09-16T04:52:28.740613+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:28.740613+0000","last_undegraded":"2024-09-16T04:52:28.740613+0000","last_fullsized":"2024-09-16T04:52:28.740613+0000","mapping_epoch":36,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:29.802644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2798860,"num_objects":342,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12326,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12418","reported_seq":12556,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.361950+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:31.361950+0000","last_peered":"2024-09-16T04:52:31.361950+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:31.361950+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2643212,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12416,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1925}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12007","reported_seq":12123,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.309410+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:31.309410+0000","last_peered":"2024-09-16T04:52:31.309410+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:31.309410+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2498560,"num_objects":305,"num_object_clones":0,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12007,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12288","reported_seq":12391,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:30.587420+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:30.587420+0000","last_peered":"2024-09-16T04:52:30.587420+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:30.587420+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2553100,"num_objects":312,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12286,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1901}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12412","reported_seq":16977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:28.660835+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:28.660835+0000","last_peered":"2024-09-16T04:52:28.660835+0000","last_clean":"2024-09-16T04:52:28.660835+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:28.660835+0000","last_undegraded":"2024-09-16T04:52:28.660835+0000","last_fullsized":"2024-09-16T04:52:28.660835+0000","mapping_epoch":32,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:29.802615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3052812,"num_objects":373,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12410,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12411","reported_seq":17174,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:31.178936+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:31.178936+0000","last_peered":"2024-09-16T04:52:31.178936+0000","last_clean":"2024-09-16T04:52:31.178936+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:31.178936+0000","last_undegraded":"2024-09-16T04:52:31.178936+0000","last_fullsized":"2024-09-16T04:52:31.178936+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2820632,"num_objects":345,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12408,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12375","reported_seq":12487,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:29.613066+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:29.613066+0000","last_peered":"2024-09-16T04:52:29.613066+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:29.613066+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3134732,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12373,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1806}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12160","reported_seq":12298,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.252224+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:31.252224+0000","last_peered":"2024-09-16T04:52:31.252224+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:31.252224+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2707346,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12157,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12028","reported_seq":12126,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.354437+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:31.354437+0000","last_peered":"2024-09-16T04:52:31.354437+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:31.354437+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2480774,"num_objects":303,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12027,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1802}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12170","reported_seq":12264,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:29.571361+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:29.571361+0000","last_peered":"2024-09-16T04:52:29.571361+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:29.571361+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2972294,"num_objects":363,"num_object_clones":0,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12169,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1858}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12442","reported_seq":12579,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.362737+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:31.362737+0000","last_peered":"2024-09-16T04:52:31.362737+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:31.362737+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2782476,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12440,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1858}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12099","reported_seq":13919,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:29.648643+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:29.648643+0000","last_peered":"2024-09-16T04:52:29.648643+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:29.648643+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2864396,"num_objects":350,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12097,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":866,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-920}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11982","reported_seq":12082,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:30.853667+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:30.853667+0000","last_peered":"2024-09-16T04:52:30.853667+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:30.853667+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2659596,"num_objects":325,"num_object_clones":0,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11980,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1903}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":43853504,"num_objects":5359,"num_object_clones":0,"num_object_copies":21436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196128,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7951,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":88875008,"data_stored":88875008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160755,"ondisk_log_size":160755,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702812,"kb_used_data":21388,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93669028,"statfs":{"total":96636764160,"available":95917084672,"internally_reserved":0,"allocated":21901312,"data_stored":21702803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697757391},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729532,"kb_used_data":22252,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93642308,"statfs":{"total":96636764160,"available":95889723392,"internally_reserved":0,"allocated":22786048,"data_stored":22587539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578628,"kb_used_data":23536,"kb_used_omap":4,"kb_used_meta":555067,"kb_avail":93793212,"statfs":{"total":96636764160,"available":96044249088,"internally_reserved":0,"allocated":24100864,"data_stored":23902355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":568389327},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,3,1,3,1,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703272,"kb_used_data":21912,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93668568,"statfs":{"total":96636764160,"available":95916613632,"internally_reserved":0,"allocated":22437888,"data_stored":22239379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":697691854},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21590016,"data_stored":21590016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":23658496,"data_stored":23658496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":22466560,"data_stored":22466560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":21159936,"data_stored":21159936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:34.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:52:34.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12328", 'reported_seq': 12450, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:31.082841+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:52:31.082841+0000', 'last_peered': '2024-09-16T04:52:31.082841+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:52:31.082841+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2670592, 'num_objects': 326, 'num_object_clones': 0, 'num_object_copies': 1304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 326, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12328, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1860}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:34.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:52:34.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12365", 'reported_seq': 12459, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:30.852214+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:52:30.852214+0000', 'last_peered': '2024-09-16T04:52:30.852214+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:52:30.852214+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2528524, 'num_objects': 309, 'num_object_clones': 0, 'num_object_copies': 1236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12363, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1894}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:34.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:52:34.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12342", 'reported_seq': 12446, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:29.640845+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:52:29.640845+0000', 'last_peered': '2024-09-16T04:52:29.640845+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:52:29.640845+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2685598, 'num_objects': 329, 'num_object_clones': 0, 'num_object_copies': 1316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 329, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12341, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1890}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:34.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:52:34.518 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12418", 'reported_seq': 12556, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:31.361950+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:52:31.361950+0000', 'last_peered': '2024-09-16T04:52:31.361950+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:52:31.361950+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2643212, 'num_objects': 323, 'num_object_clones': 0, 'num_object_copies': 1292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 323, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12416, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1925}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:34.518 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:52:34.518 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12007", 'reported_seq': 12123, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:31.309410+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:52:31.309410+0000', 'last_peered': '2024-09-16T04:52:31.309410+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:52:31.309410+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2498560, 'num_objects': 305, 'num_object_clones': 0, 'num_object_copies': 1220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12007, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1873}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:34.518 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:52:34.518 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12288", 'reported_seq': 12391, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:30.587420+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:52:30.587420+0000', 'last_peered': '2024-09-16T04:52:30.587420+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:52:30.587420+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2553100, 'num_objects': 312, 'num_object_clones': 0, 'num_object_copies': 1248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 312, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12286, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1901}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:34.518 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:52:34.518 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12375", 'reported_seq': 12487, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:29.613066+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:52:29.613066+0000', 'last_peered': '2024-09-16T04:52:29.613066+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:52:29.613066+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3134732, 'num_objects': 383, 'num_object_clones': 0, 'num_object_copies': 1532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12373, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1806}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:34.518 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:52:34.518 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12160", 'reported_seq': 12298, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:31.252224+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:52:31.252224+0000', 'last_peered': '2024-09-16T04:52:31.252224+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:52:31.252224+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2707346, 'num_objects': 331, 'num_object_clones': 0, 'num_object_copies': 1324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 331, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12157, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1824}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:34.518 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:52:34.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12028", 'reported_seq': 12126, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:31.354437+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:52:31.354437+0000', 'last_peered': '2024-09-16T04:52:31.354437+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:52:31.354437+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2480774, 'num_objects': 303, 'num_object_clones': 0, 'num_object_copies': 1212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 303, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12027, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1802}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:34.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:52:34.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12170", 'reported_seq': 12264, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:29.571361+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:52:29.571361+0000', 'last_peered': '2024-09-16T04:52:29.571361+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:52:29.571361+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2972294, 'num_objects': 363, 'num_object_clones': 0, 'num_object_copies': 1452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12169, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1858}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:34.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:52:34.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12442", 'reported_seq': 12579, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:31.362737+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:52:31.362737+0000', 'last_peered': '2024-09-16T04:52:31.362737+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:52:31.362737+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2782476, 'num_objects': 340, 'num_object_clones': 0, 'num_object_copies': 1360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12440, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1858}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:34.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:52:34.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12099", 'reported_seq': 13919, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:29.648643+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:52:29.648643+0000', 'last_peered': '2024-09-16T04:52:29.648643+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:52:29.648643+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2864396, 'num_objects': 350, 'num_object_clones': 0, 'num_object_copies': 1400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 350, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12097, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 866, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -920}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:34.519 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:52:34.519 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11982", 'reported_seq': 12082, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:30.853667+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:52:30.853667+0000', 'last_peered': '2024-09-16T04:52:30.853667+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:52:30.853667+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2659596, 'num_objects': 325, 'num_object_clones': 0, 'num_object_copies': 1300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 325, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11980, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1903}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:34.520 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:52:35.261 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:35.261 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27631/21436 objects degraded (128.900%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":767,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":5359,"data_bytes":43853504,"bytes_used":2779385856,"bytes_avail":383767670784,"bytes_total":386547056640,"degraded_objects":27631,"degraded_total":21436,"degraded_ratio":1.2889998133980221,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":193},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:52:35.289 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:52:35.289 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:35.893 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:35.894 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:35.912 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-16T04:52:34.795580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":42870464,"num_objects":5239,"num_object_clones":0,"num_object_copies":20956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196248,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7951,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160875,"ondisk_log_size":160875,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2732456,"kb_used_data":85080,"kb_used_omap":14,"kb_used_meta":2647281,"kb_avail":374754904,"statfs":{"total":386547056640,"available":383749021696,"internally_reserved":0,"allocated":87121920,"data_stored":86327884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2710816305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,5,9,13,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16646144,"num_objects":-2032,"num_object_clones":0,"num_object_copies":-8128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2032,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006030"},"pg_stats":[{"pgid":"1.f","version":"37'12328","reported_seq":12450,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.082841+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:31.082841+0000","last_peered":"2024-09-16T04:52:31.082841+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:31.082841+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2670592,"num_objects":326,"num_object_clones":0,"num_object_copies":1304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12328,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1860}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12365","reported_seq":12459,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:30.852214+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:30.852214+0000","last_peered":"2024-09-16T04:52:30.852214+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:30.852214+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2528524,"num_objects":309,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12363,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1894}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12342","reported_seq":12446,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:29.640845+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:29.640845+0000","last_peered":"2024-09-16T04:52:29.640845+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:29.640845+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2685598,"num_objects":329,"num_object_clones":0,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12341,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1890}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12387","reported_seq":16937,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:34.075825+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:34.075825+0000","last_peered":"2024-09-16T04:52:34.075825+0000","last_clean":"2024-09-16T04:52:34.075825+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:34.075825+0000","last_undegraded":"2024-09-16T04:52:34.075825+0000","last_fullsized":"2024-09-16T04:52:34.075825+0000","mapping_epoch":36,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:34.965464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2315532,"num_objects":283,"num_object_clones":0,"num_object_copies":1132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12385,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12418","reported_seq":12556,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.361950+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:31.361950+0000","last_peered":"2024-09-16T04:52:31.361950+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:31.361950+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2643212,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12416,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1925}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12007","reported_seq":12123,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.309410+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:31.309410+0000","last_peered":"2024-09-16T04:52:31.309410+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:31.309410+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2498560,"num_objects":305,"num_object_clones":0,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12007,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1873}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12288","reported_seq":12391,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:30.587420+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:30.587420+0000","last_peered":"2024-09-16T04:52:30.587420+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:30.587420+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2553100,"num_objects":312,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12286,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1901}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12473","reported_seq":17040,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:33.891896+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:33.891896+0000","last_peered":"2024-09-16T04:52:33.891896+0000","last_clean":"2024-09-16T04:52:33.891896+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:33.891896+0000","last_undegraded":"2024-09-16T04:52:33.891896+0000","last_fullsized":"2024-09-16T04:52:33.891896+0000","mapping_epoch":32,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:34.965442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2553100,"num_objects":312,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12471,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12411","reported_seq":17174,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:31.178936+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:31.178936+0000","last_peered":"2024-09-16T04:52:31.178936+0000","last_clean":"2024-09-16T04:52:31.178936+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:31.178936+0000","last_undegraded":"2024-09-16T04:52:31.178936+0000","last_fullsized":"2024-09-16T04:52:31.178936+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2820632,"num_objects":345,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12408,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12375","reported_seq":12487,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:29.613066+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:29.613066+0000","last_peered":"2024-09-16T04:52:29.613066+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:29.613066+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3134732,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12373,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1806}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12160","reported_seq":12298,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.252224+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:31.252224+0000","last_peered":"2024-09-16T04:52:31.252224+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:31.252224+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2707346,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12157,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12028","reported_seq":12126,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.354437+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:31.354437+0000","last_peered":"2024-09-16T04:52:31.354437+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:31.354437+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2480774,"num_objects":303,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12027,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1802}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12170","reported_seq":12264,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:29.571361+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:29.571361+0000","last_peered":"2024-09-16T04:52:29.571361+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:29.571361+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2972294,"num_objects":363,"num_object_clones":0,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12169,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1858}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12442","reported_seq":12579,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:31.362737+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:31.362737+0000","last_peered":"2024-09-16T04:52:31.362737+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:31.362737+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2782476,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12440,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1858}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12099","reported_seq":13919,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:29.648643+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:29.648643+0000","last_peered":"2024-09-16T04:52:29.648643+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:29.648643+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2864396,"num_objects":350,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12097,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":866,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-920}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11982","reported_seq":12082,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:30.853667+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:30.853667+0000","last_peered":"2024-09-16T04:52:30.853667+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:30.853667+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2659596,"num_objects":325,"num_object_clones":0,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11980,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1903}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":42870464,"num_objects":5239,"num_object_clones":0,"num_object_copies":20956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196248,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7951,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":84865024,"data_stored":84865024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160875,"ondisk_log_size":160875,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702812,"kb_used_data":21388,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93669028,"statfs":{"total":96636764160,"available":95917084672,"internally_reserved":0,"allocated":21901312,"data_stored":21702803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":697757391},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729532,"kb_used_data":22252,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93642308,"statfs":{"total":96636764160,"available":95889723392,"internally_reserved":0,"allocated":22786048,"data_stored":22587539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596840,"kb_used_data":19528,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93775000,"statfs":{"total":96636764160,"available":96025600000,"internally_reserved":0,"allocated":19996672,"data_stored":19798163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591130319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703272,"kb_used_data":21912,"kb_used_omap":4,"kb_used_meta":681339,"kb_avail":93668568,"statfs":{"total":96636764160,"available":95916613632,"internally_reserved":0,"allocated":22437888,"data_stored":22239379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":697691854},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21590016,"data_stored":21590016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19648512,"data_stored":19648512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":22466560,"data_stored":22466560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":21159936,"data_stored":21159936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:38.914 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:39.524 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:39.525 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:39.556 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-09-16T04:52:38.797356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":35841728,"num_objects":4381,"num_object_clones":0,"num_object_copies":17524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4381,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197106,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8006,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160833,"ondisk_log_size":160833,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2733640,"kb_used_data":73724,"kb_used_omap":14,"kb_used_meta":2659761,"kb_avail":374753720,"statfs":{"total":386547056640,"available":383747809280,"internally_reserved":0,"allocated":75493376,"data_stored":74699340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2723595825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,15,7,18,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15695872,"num_objects":-1916,"num_object_clones":0,"num_object_copies":-7664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1916,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006037"},"pg_stats":[{"pgid":"1.f","version":"37'12385","reported_seq":12507,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:35.998008+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:35.998008+0000","last_peered":"2024-09-16T04:52:35.998008+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:35.998008+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2203648,"num_objects":269,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12385,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12421","reported_seq":12515,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:35.831822+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:35.831822+0000","last_peered":"2024-09-16T04:52:35.831822+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:35.831822+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2069772,"num_objects":253,"num_object_clones":0,"num_object_copies":1012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12419,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12402","reported_seq":12506,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:34.415056+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:34.415056+0000","last_peered":"2024-09-16T04:52:34.415056+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:34.415056+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2194078,"num_objects":269,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12401,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1950}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12387","reported_seq":16937,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:34.075825+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:34.075825+0000","last_peered":"2024-09-16T04:52:34.075825+0000","last_clean":"2024-09-16T04:52:34.075825+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:34.075825+0000","last_undegraded":"2024-09-16T04:52:34.075825+0000","last_fullsized":"2024-09-16T04:52:34.075825+0000","mapping_epoch":36,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:34.965464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2315532,"num_objects":283,"num_object_clones":0,"num_object_copies":1132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12385,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12481","reported_seq":12617,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:36.329332+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:36.329332+0000","last_peered":"2024-09-16T04:52:36.329332+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:36.329332+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2127116,"num_objects":260,"num_object_clones":0,"num_object_copies":1040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12479,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12069","reported_seq":12185,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:36.193193+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:36.193193+0000","last_peered":"2024-09-16T04:52:36.193193+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:36.193193+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1990656,"num_objects":243,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12069,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12337","reported_seq":12440,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:35.789778+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:35.789778+0000","last_peered":"2024-09-16T04:52:35.789778+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:35.789778+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151692,"num_objects":263,"num_object_clones":0,"num_object_copies":1052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12335,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1950}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12473","reported_seq":17040,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:33.891896+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:33.891896+0000","last_peered":"2024-09-16T04:52:33.891896+0000","last_clean":"2024-09-16T04:52:33.891896+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:33.891896+0000","last_undegraded":"2024-09-16T04:52:33.891896+0000","last_fullsized":"2024-09-16T04:52:33.891896+0000","mapping_epoch":32,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:34.965442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2553100,"num_objects":312,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12471,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12473","reported_seq":17236,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:36.260434+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:36.260434+0000","last_peered":"2024-09-16T04:52:36.260434+0000","last_clean":"2024-09-16T04:52:36.260434+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:36.260434+0000","last_undegraded":"2024-09-16T04:52:36.260434+0000","last_fullsized":"2024-09-16T04:52:36.260434+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:36.897740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2312728,"num_objects":283,"num_object_clones":0,"num_object_copies":1132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12470,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12442","reported_seq":12551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:34.743922+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:34.743922+0000","last_peered":"2024-09-16T04:52:34.743922+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:34.743922+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2585868,"num_objects":316,"num_object_clones":0,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12440,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1873}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12222","reported_seq":12360,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:36.283364+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:36.283364+0000","last_peered":"2024-09-16T04:52:36.283364+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:36.283364+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2199442,"num_objects":269,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12219,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1886}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12095","reported_seq":12193,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:36.282941+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:36.282941+0000","last_peered":"2024-09-16T04:52:36.282941+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:36.282941+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1931910,"num_objects":236,"num_object_clones":0,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12094,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1869}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12225","reported_seq":12319,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:34.729982+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:34.729982+0000","last_peered":"2024-09-16T04:52:34.729982+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:34.729982+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521734,"num_objects":308,"num_object_clones":0,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12224,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1913}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12508","reported_seq":12646,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:36.197987+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:36.197987+0000","last_peered":"2024-09-16T04:52:36.197987+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:36.197987+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2241804,"num_objects":274,"num_object_clones":0,"num_object_copies":1096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12506,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1924}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12168","reported_seq":14098,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:34.751796+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:34.751796+0000","last_peered":"2024-09-16T04:52:34.751796+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:34.751796+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2299148,"num_objects":281,"num_object_clones":0,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12166,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-934}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12045","reported_seq":12145,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:35.950710+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:35.950710+0000","last_peered":"2024-09-16T04:52:35.950710+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:35.950710+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2143500,"num_objects":262,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12043,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":35841728,"num_objects":4381,"num_object_clones":0,"num_object_copies":17524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4381,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197106,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8006,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73170944,"data_stored":73170944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160833,"ondisk_log_size":160833,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703204,"kb_used_data":17620,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93668636,"statfs":{"total":96636764160,"available":95916683264,"internally_reserved":0,"allocated":18042880,"data_stored":17844371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":702017231},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,2,1,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729972,"kb_used_data":18504,"kb_used_omap":1,"kb_used_meta":711422,"kb_avail":93641868,"statfs":{"total":96636764160,"available":95889272832,"internally_reserved":0,"allocated":18948096,"data_stored":18749587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":728496581},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,1,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596840,"kb_used_data":19528,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93775000,"statfs":{"total":96636764160,"available":96025600000,"internally_reserved":0,"allocated":19996672,"data_stored":19798163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591130319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247956,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703624,"kb_used_data":18072,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93668216,"statfs":{"total":96636764160,"available":95916253184,"internally_reserved":0,"allocated":18505728,"data_stored":18307219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":701951694},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17682432,"data_stored":17682432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19648512,"data_stored":19648512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18575360,"data_stored":18575360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":17264640,"data_stored":17264640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:39.558 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:40.155 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:40.156 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:40.194 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-09-16T04:52:38.797356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":35841728,"num_objects":4381,"num_object_clones":0,"num_object_copies":17524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4381,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197106,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8006,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160833,"ondisk_log_size":160833,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2733640,"kb_used_data":73724,"kb_used_omap":14,"kb_used_meta":2659761,"kb_avail":374753720,"statfs":{"total":386547056640,"available":383747809280,"internally_reserved":0,"allocated":75493376,"data_stored":74699340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2723595825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,15,7,18,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15695872,"num_objects":-1916,"num_object_clones":0,"num_object_copies":-7664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1916,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006037"},"pg_stats":[{"pgid":"1.f","version":"37'12385","reported_seq":12507,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:35.998008+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:35.998008+0000","last_peered":"2024-09-16T04:52:35.998008+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:35.998008+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2203648,"num_objects":269,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12385,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12421","reported_seq":12515,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:35.831822+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:35.831822+0000","last_peered":"2024-09-16T04:52:35.831822+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:35.831822+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2069772,"num_objects":253,"num_object_clones":0,"num_object_copies":1012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12419,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12402","reported_seq":12506,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:34.415056+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:34.415056+0000","last_peered":"2024-09-16T04:52:34.415056+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:34.415056+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2194078,"num_objects":269,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12401,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1950}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12387","reported_seq":16937,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:34.075825+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:34.075825+0000","last_peered":"2024-09-16T04:52:34.075825+0000","last_clean":"2024-09-16T04:52:34.075825+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:34.075825+0000","last_undegraded":"2024-09-16T04:52:34.075825+0000","last_fullsized":"2024-09-16T04:52:34.075825+0000","mapping_epoch":36,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:34.965464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2315532,"num_objects":283,"num_object_clones":0,"num_object_copies":1132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12385,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12481","reported_seq":12617,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:36.329332+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:36.329332+0000","last_peered":"2024-09-16T04:52:36.329332+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:36.329332+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2127116,"num_objects":260,"num_object_clones":0,"num_object_copies":1040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12479,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12069","reported_seq":12185,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:36.193193+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:36.193193+0000","last_peered":"2024-09-16T04:52:36.193193+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:36.193193+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1990656,"num_objects":243,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12069,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12337","reported_seq":12440,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:35.789778+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:35.789778+0000","last_peered":"2024-09-16T04:52:35.789778+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:35.789778+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151692,"num_objects":263,"num_object_clones":0,"num_object_copies":1052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12335,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1950}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12473","reported_seq":17040,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:33.891896+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:33.891896+0000","last_peered":"2024-09-16T04:52:33.891896+0000","last_clean":"2024-09-16T04:52:33.891896+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:33.891896+0000","last_undegraded":"2024-09-16T04:52:33.891896+0000","last_fullsized":"2024-09-16T04:52:33.891896+0000","mapping_epoch":32,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:34.965442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2553100,"num_objects":312,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12471,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12473","reported_seq":17236,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:36.260434+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:36.260434+0000","last_peered":"2024-09-16T04:52:36.260434+0000","last_clean":"2024-09-16T04:52:36.260434+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:36.260434+0000","last_undegraded":"2024-09-16T04:52:36.260434+0000","last_fullsized":"2024-09-16T04:52:36.260434+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:36.897740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2312728,"num_objects":283,"num_object_clones":0,"num_object_copies":1132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12470,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12442","reported_seq":12551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:34.743922+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:34.743922+0000","last_peered":"2024-09-16T04:52:34.743922+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:34.743922+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2585868,"num_objects":316,"num_object_clones":0,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12440,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1873}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12222","reported_seq":12360,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:36.283364+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:36.283364+0000","last_peered":"2024-09-16T04:52:36.283364+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:36.283364+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2199442,"num_objects":269,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12219,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1886}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12095","reported_seq":12193,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:36.282941+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:36.282941+0000","last_peered":"2024-09-16T04:52:36.282941+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:36.282941+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1931910,"num_objects":236,"num_object_clones":0,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12094,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1869}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12225","reported_seq":12319,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:34.729982+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:34.729982+0000","last_peered":"2024-09-16T04:52:34.729982+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:34.729982+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521734,"num_objects":308,"num_object_clones":0,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12224,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1913}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12508","reported_seq":12646,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:36.197987+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:36.197987+0000","last_peered":"2024-09-16T04:52:36.197987+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:36.197987+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2241804,"num_objects":274,"num_object_clones":0,"num_object_copies":1096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12506,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1924}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12168","reported_seq":14098,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:34.751796+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:34.751796+0000","last_peered":"2024-09-16T04:52:34.751796+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:34.751796+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2299148,"num_objects":281,"num_object_clones":0,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12166,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-934}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12045","reported_seq":12145,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:35.950710+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:35.950710+0000","last_peered":"2024-09-16T04:52:35.950710+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:35.950710+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2143500,"num_objects":262,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12043,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":35841728,"num_objects":4381,"num_object_clones":0,"num_object_copies":17524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4381,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197106,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8006,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73170944,"data_stored":73170944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160833,"ondisk_log_size":160833,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703204,"kb_used_data":17620,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93668636,"statfs":{"total":96636764160,"available":95916683264,"internally_reserved":0,"allocated":18042880,"data_stored":17844371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":702017231},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,2,1,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729972,"kb_used_data":18504,"kb_used_omap":1,"kb_used_meta":711422,"kb_avail":93641868,"statfs":{"total":96636764160,"available":95889272832,"internally_reserved":0,"allocated":18948096,"data_stored":18749587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":728496581},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,1,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596840,"kb_used_data":19528,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93775000,"statfs":{"total":96636764160,"available":96025600000,"internally_reserved":0,"allocated":19996672,"data_stored":19798163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591130319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247956,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703624,"kb_used_data":18072,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93668216,"statfs":{"total":96636764160,"available":95916253184,"internally_reserved":0,"allocated":18505728,"data_stored":18307219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":701951694},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17682432,"data_stored":17682432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19648512,"data_stored":19648512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18575360,"data_stored":18575360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":17264640,"data_stored":17264640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:40.195 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:52:40.195 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12385", 'reported_seq': 12507, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:35.998008+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:52:35.998008+0000', 'last_peered': '2024-09-16T04:52:35.998008+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:52:35.998008+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2203648, 'num_objects': 269, 'num_object_clones': 0, 'num_object_copies': 1076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12385, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1917}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:40.195 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:52:40.195 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12421", 'reported_seq': 12515, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:35.831822+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:52:35.831822+0000', 'last_peered': '2024-09-16T04:52:35.831822+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:52:35.831822+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2069772, 'num_objects': 253, 'num_object_clones': 0, 'num_object_copies': 1012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 253, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12419, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:40.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:52:40.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12402", 'reported_seq': 12506, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:34.415056+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:52:34.415056+0000', 'last_peered': '2024-09-16T04:52:34.415056+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:52:34.415056+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2194078, 'num_objects': 269, 'num_object_clones': 0, 'num_object_copies': 1076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 269, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12401, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:40.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:52:40.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12481", 'reported_seq': 12617, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:36.329332+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:52:36.329332+0000', 'last_peered': '2024-09-16T04:52:36.329332+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:52:36.329332+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2127116, 'num_objects': 260, 'num_object_clones': 0, 'num_object_copies': 1040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 260, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12479, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1988}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:40.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:52:40.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12069", 'reported_seq': 12185, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:36.193193+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:52:36.193193+0000', 'last_peered': '2024-09-16T04:52:36.193193+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:52:36.193193+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1990656, 'num_objects': 243, 'num_object_clones': 0, 'num_object_copies': 972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 243, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12069, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1935}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:40.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:52:40.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12337", 'reported_seq': 12440, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:35.789778+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:52:35.789778+0000', 'last_peered': '2024-09-16T04:52:35.789778+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:52:35.789778+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2151692, 'num_objects': 263, 'num_object_clones': 0, 'num_object_copies': 1052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12335, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:40.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:52:40.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12442", 'reported_seq': 12551, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:34.743922+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:52:34.743922+0000', 'last_peered': '2024-09-16T04:52:34.743922+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:52:34.743922+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2585868, 'num_objects': 316, 'num_object_clones': 0, 'num_object_copies': 1264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12440, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1873}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:40.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:52:40.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12222", 'reported_seq': 12360, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:36.283364+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:52:36.283364+0000', 'last_peered': '2024-09-16T04:52:36.283364+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:52:36.283364+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2199442, 'num_objects': 269, 'num_object_clones': 0, 'num_object_copies': 1076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12219, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1886}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:40.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:52:40.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12095", 'reported_seq': 12193, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:36.282941+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:52:36.282941+0000', 'last_peered': '2024-09-16T04:52:36.282941+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:52:36.282941+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1931910, 'num_objects': 236, 'num_object_clones': 0, 'num_object_copies': 944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12094, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1869}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:40.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:52:40.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12225", 'reported_seq': 12319, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:34.729982+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:52:34.729982+0000', 'last_peered': '2024-09-16T04:52:34.729982+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:52:34.729982+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2521734, 'num_objects': 308, 'num_object_clones': 0, 'num_object_copies': 1232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12224, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1913}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:40.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:52:40.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12508", 'reported_seq': 12646, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:36.197987+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:52:36.197987+0000', 'last_peered': '2024-09-16T04:52:36.197987+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:52:36.197987+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2241804, 'num_objects': 274, 'num_object_clones': 0, 'num_object_copies': 1096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12506, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1924}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:40.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:52:40.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12168", 'reported_seq': 14098, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:34.751796+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:52:34.751796+0000', 'last_peered': '2024-09-16T04:52:34.751796+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:52:34.751796+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2299148, 'num_objects': 281, 'num_object_clones': 0, 'num_object_copies': 1124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12166, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 921, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -934}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:40.198 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:52:40.198 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12045", 'reported_seq': 12145, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:35.950710+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:52:35.950710+0000', 'last_peered': '2024-09-16T04:52:35.950710+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:52:35.950710+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2143500, 'num_objects': 262, 'num_object_clones': 0, 'num_object_copies': 1048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12043, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:40.198 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:52:40.934 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:40.934 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27576/17524 objects degraded (157.361%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":773,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4381,"data_bytes":35841728,"bytes_used":2799247360,"bytes_avail":383747809280,"bytes_total":386547056640,"degraded_objects":27576,"degraded_total":17524,"degraded_ratio":1.5736133302898885,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":159},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:52:40.955 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:52:40.955 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:41.544 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:41.545 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:41.566 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-09-16T04:52:40.798216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":32614080,"num_objects":3987,"num_object_clones":0,"num_object_copies":15948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3987,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197500,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8053,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2736320,"kb_used_data":66716,"kb_used_omap":14,"kb_used_meta":2669489,"kb_avail":374751040,"statfs":{"total":386547056640,"available":383745064960,"internally_reserved":0,"allocated":68317184,"data_stored":67523148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2733557297},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,9,11,15,15,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18923520,"num_objects":-2310,"num_object_clones":0,"num_object_copies":-9240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2310,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005976"},"pg_stats":[{"pgid":"1.f","version":"37'12385","reported_seq":12507,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:35.998008+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:35.998008+0000","last_peered":"2024-09-16T04:52:35.998008+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:35.998008+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2203648,"num_objects":269,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12385,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1917}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12421","reported_seq":12515,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:35.831822+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:35.831822+0000","last_peered":"2024-09-16T04:52:35.831822+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:35.831822+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2069772,"num_objects":253,"num_object_clones":0,"num_object_copies":1012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12419,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12454","reported_seq":12558,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:39.580455+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:39.580455+0000","last_peered":"2024-09-16T04:52:39.580455+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:39.580455+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1768094,"num_objects":217,"num_object_clones":0,"num_object_copies":868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12453,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2002}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12445","reported_seq":16995,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:38.831836+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:38.831836+0000","last_peered":"2024-09-16T04:52:38.831836+0000","last_clean":"2024-09-16T04:52:38.831836+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:38.831836+0000","last_undegraded":"2024-09-16T04:52:38.831836+0000","last_fullsized":"2024-09-16T04:52:38.831836+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:40.131218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1840396,"num_objects":225,"num_object_clones":0,"num_object_copies":900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12443,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12481","reported_seq":12617,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:36.329332+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:36.329332+0000","last_peered":"2024-09-16T04:52:36.329332+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:36.329332+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2127116,"num_objects":260,"num_object_clones":0,"num_object_copies":1040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12479,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12069","reported_seq":12185,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:36.193193+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:36.193193+0000","last_peered":"2024-09-16T04:52:36.193193+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:36.193193+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1990656,"num_objects":243,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12069,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12337","reported_seq":12440,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:35.789778+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:35.789778+0000","last_peered":"2024-09-16T04:52:35.789778+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:35.789778+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151692,"num_objects":263,"num_object_clones":0,"num_object_copies":1052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12335,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1950}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12534","reported_seq":17099,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:38.955531+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:38.955531+0000","last_peered":"2024-09-16T04:52:38.955531+0000","last_clean":"2024-09-16T04:52:38.955531+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:38.955531+0000","last_undegraded":"2024-09-16T04:52:38.955531+0000","last_fullsized":"2024-09-16T04:52:38.955531+0000","mapping_epoch":32,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:40.130466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053388,"num_objects":251,"num_object_clones":0,"num_object_copies":1004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12532,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12473","reported_seq":17236,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:36.260434+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:36.260434+0000","last_peered":"2024-09-16T04:52:36.260434+0000","last_clean":"2024-09-16T04:52:36.260434+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:36.260434+0000","last_undegraded":"2024-09-16T04:52:36.260434+0000","last_fullsized":"2024-09-16T04:52:36.260434+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:36.897740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2312728,"num_objects":283,"num_object_clones":0,"num_object_copies":1132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12470,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12637,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:39.605237+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:39.605237+0000","last_peered":"2024-09-16T04:52:39.605237+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:39.605237+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1897740,"num_objects":232,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1957}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12222","reported_seq":12360,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:36.283364+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:36.283364+0000","last_peered":"2024-09-16T04:52:36.283364+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:36.283364+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2199442,"num_objects":269,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12219,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1886}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12095","reported_seq":12193,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:36.282941+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:36.282941+0000","last_peered":"2024-09-16T04:52:36.282941+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:36.282941+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1931910,"num_objects":236,"num_object_clones":0,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12094,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1869}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12291","reported_seq":12384,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:39.745865+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:39.745865+0000","last_peered":"2024-09-16T04:52:39.745865+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:39.745865+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1981062,"num_objects":242,"num_object_clones":0,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1979}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12508","reported_seq":12646,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:36.197987+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:36.197987+0000","last_peered":"2024-09-16T04:52:36.197987+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:36.197987+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2241804,"num_objects":274,"num_object_clones":0,"num_object_copies":1096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12506,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1924}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12241","reported_seq":14265,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:39.745109+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:39.745109+0000","last_peered":"2024-09-16T04:52:39.745109+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:39.745109+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1701132,"num_objects":208,"num_object_clones":0,"num_object_copies":832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12239,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-960}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12045","reported_seq":12145,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:35.950710+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:35.950710+0000","last_peered":"2024-09-16T04:52:35.950710+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:35.950710+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2143500,"num_objects":262,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12043,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":32614080,"num_objects":3987,"num_object_clones":0,"num_object_copies":15948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3987,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197500,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8053,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65200128,"data_stored":65200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703204,"kb_used_data":17620,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93668636,"statfs":{"total":96636764160,"available":95916683264,"internally_reserved":0,"allocated":18042880,"data_stored":17844371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":702017231},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,2,1,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731692,"kb_used_data":14680,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93640148,"statfs":{"total":96636764160,"available":95887511552,"internally_reserved":0,"allocated":15032320,"data_stored":14833811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,0,2,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597800,"kb_used_data":16344,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93774040,"statfs":{"total":96636764160,"available":96024616960,"internally_reserved":0,"allocated":16736256,"data_stored":16537747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595390159},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,10,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247956,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703624,"kb_used_data":18072,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93668216,"statfs":{"total":96636764160,"available":95916253184,"internally_reserved":0,"allocated":18505728,"data_stored":18307219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":701951694},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17682432,"data_stored":17682432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15683584,"data_stored":15683584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14569472,"data_stored":14569472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":17264640,"data_stored":17264640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:44.568 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:45.215 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:45.216 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:45.248 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-09-16T04:52:44.799765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26977984,"num_objects":3299,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198188,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8053,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160915,"ondisk_log_size":160915,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2737936,"kb_used_data":54440,"kb_used_omap":14,"kb_used_meta":2683377,"kb_avail":374749424,"statfs":{"total":386547056640,"available":383743410176,"internally_reserved":0,"allocated":55746560,"data_stored":54952524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2747778609},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,13,5,10,18,9,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16875520,"num_objects":-2060,"num_object_clones":0,"num_object_copies":-8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2060,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004978"},"pg_stats":[{"pgid":"1.f","version":"37'12427","reported_seq":12549,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:41.275522+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:41.275522+0000","last_peered":"2024-09-16T04:52:41.275522+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:41.275522+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1859584,"num_objects":227,"num_object_clones":0,"num_object_copies":908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12427,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1959}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12475","reported_seq":12568,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:41.034606+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:41.034606+0000","last_peered":"2024-09-16T04:52:41.034606+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:41.034606+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1627404,"num_objects":199,"num_object_clones":0,"num_object_copies":796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12473,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12454","reported_seq":12558,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:39.580455+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:39.580455+0000","last_peered":"2024-09-16T04:52:39.580455+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:39.580455+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1768094,"num_objects":217,"num_object_clones":0,"num_object_copies":868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12453,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2002}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12496","reported_seq":17046,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:44.147825+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:44.147825+0000","last_peered":"2024-09-16T04:52:44.147825+0000","last_clean":"2024-09-16T04:52:44.147825+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:44.147825+0000","last_undegraded":"2024-09-16T04:52:44.147825+0000","last_fullsized":"2024-09-16T04:52:44.147825+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:45.243479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1422604,"num_objects":174,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12494,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12540","reported_seq":12678,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:41.359869+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:41.359869+0000","last_peered":"2024-09-16T04:52:41.359869+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:41.359869+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1643788,"num_objects":201,"num_object_clones":0,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12538,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12119","reported_seq":12235,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:41.373255+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:41.373255+0000","last_peered":"2024-09-16T04:52:41.373255+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:41.373255+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1581056,"num_objects":193,"num_object_clones":0,"num_object_copies":772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12119,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12395","reported_seq":12498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:40.949510+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:40.949510+0000","last_peered":"2024-09-16T04:52:40.949510+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:40.949510+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1676556,"num_objects":205,"num_object_clones":0,"num_object_copies":820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12393,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2008}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12586","reported_seq":17154,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:44.028935+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:44.028935+0000","last_peered":"2024-09-16T04:52:44.028935+0000","last_clean":"2024-09-16T04:52:44.028935+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:44.028935+0000","last_undegraded":"2024-09-16T04:52:44.028935+0000","last_fullsized":"2024-09-16T04:52:44.028935+0000","mapping_epoch":32,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:45.241508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1627404,"num_objects":199,"num_object_clones":0,"num_object_copies":796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12584,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12543","reported_seq":17306,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:41.482831+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:41.482831+0000","last_peered":"2024-09-16T04:52:41.482831+0000","last_clean":"2024-09-16T04:52:41.482831+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:41.482831+0000","last_undegraded":"2024-09-16T04:52:41.482831+0000","last_fullsized":"2024-09-16T04:52:41.482831+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:41.933068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1739288,"num_objects":213,"num_object_clones":0,"num_object_copies":852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12540,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12637,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:39.605237+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:39.605237+0000","last_peered":"2024-09-16T04:52:39.605237+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:39.605237+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1897740,"num_objects":232,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1957}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12279","reported_seq":12417,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:41.453856+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:41.453856+0000","last_peered":"2024-09-16T04:52:41.453856+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:41.453856+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1732498,"num_objects":212,"num_object_clones":0,"num_object_copies":848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12276,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1943}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12155","reported_seq":12253,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:41.501157+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:41.501157+0000","last_peered":"2024-09-16T04:52:41.501157+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:41.501157+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1440390,"num_objects":176,"num_object_clones":0,"num_object_copies":704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12154,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1929}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12291","reported_seq":12384,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:39.745865+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:39.745865+0000","last_peered":"2024-09-16T04:52:39.745865+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:39.745865+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1981062,"num_objects":242,"num_object_clones":0,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1979}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12572","reported_seq":12706,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:41.544214+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:41.544214+0000","last_peered":"2024-09-16T04:52:41.544214+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:41.544214+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1717516,"num_objects":210,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12570,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1988}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12241","reported_seq":14265,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:39.745109+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:39.745109+0000","last_peered":"2024-09-16T04:52:39.745109+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:39.745109+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1701132,"num_objects":208,"num_object_clones":0,"num_object_copies":832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12239,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-960}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12116","reported_seq":12216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:40.902116+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:40.902116+0000","last_peered":"2024-09-16T04:52:40.902116+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:40.902116+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561868,"num_objects":191,"num_object_clones":0,"num_object_copies":764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12114,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26977984,"num_objects":3299,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198188,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8053,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53252096,"data_stored":53252096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160915,"ondisk_log_size":160915,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721596,"kb_used_data":13792,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93650244,"statfs":{"total":96636764160,"available":95897849856,"internally_reserved":0,"allocated":14123008,"data_stored":13924499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":724758223},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,7,5,4,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731692,"kb_used_data":14680,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93640148,"statfs":{"total":96636764160,"available":95887511552,"internally_reserved":0,"allocated":15032320,"data_stored":14833811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,0,2,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580704,"kb_used_data":11716,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93791136,"statfs":{"total":96636764160,"available":96042123264,"internally_reserved":0,"allocated":11997184,"data_stored":11798675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582610639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,3,0,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703944,"kb_used_data":14252,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93667896,"statfs":{"total":96636764160,"available":95915925504,"internally_reserved":0,"allocated":14594048,"data_stored":14395539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":706211534},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,1,8,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13697024,"data_stored":13697024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11685888,"data_stored":11685888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14569472,"data_stored":14569472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13299712,"data_stored":13299712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:45.249 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:45.844 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:45.844 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:45.867 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-09-16T04:52:44.799765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26977984,"num_objects":3299,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198188,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8053,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160915,"ondisk_log_size":160915,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2737936,"kb_used_data":54440,"kb_used_omap":14,"kb_used_meta":2683377,"kb_avail":374749424,"statfs":{"total":386547056640,"available":383743410176,"internally_reserved":0,"allocated":55746560,"data_stored":54952524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2747778609},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,13,5,10,18,9,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16875520,"num_objects":-2060,"num_object_clones":0,"num_object_copies":-8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2060,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004978"},"pg_stats":[{"pgid":"1.f","version":"37'12427","reported_seq":12549,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:41.275522+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:41.275522+0000","last_peered":"2024-09-16T04:52:41.275522+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:41.275522+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1859584,"num_objects":227,"num_object_clones":0,"num_object_copies":908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12427,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1959}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12475","reported_seq":12568,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:41.034606+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:41.034606+0000","last_peered":"2024-09-16T04:52:41.034606+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:41.034606+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1627404,"num_objects":199,"num_object_clones":0,"num_object_copies":796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12473,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12454","reported_seq":12558,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:39.580455+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:39.580455+0000","last_peered":"2024-09-16T04:52:39.580455+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:39.580455+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1768094,"num_objects":217,"num_object_clones":0,"num_object_copies":868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12453,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2002}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12496","reported_seq":17046,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:44.147825+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:44.147825+0000","last_peered":"2024-09-16T04:52:44.147825+0000","last_clean":"2024-09-16T04:52:44.147825+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:44.147825+0000","last_undegraded":"2024-09-16T04:52:44.147825+0000","last_fullsized":"2024-09-16T04:52:44.147825+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:45.243479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1422604,"num_objects":174,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12494,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12540","reported_seq":12678,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:41.359869+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:41.359869+0000","last_peered":"2024-09-16T04:52:41.359869+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:41.359869+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1643788,"num_objects":201,"num_object_clones":0,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12538,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12119","reported_seq":12235,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:41.373255+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:41.373255+0000","last_peered":"2024-09-16T04:52:41.373255+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:41.373255+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1581056,"num_objects":193,"num_object_clones":0,"num_object_copies":772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12119,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12395","reported_seq":12498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:40.949510+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:40.949510+0000","last_peered":"2024-09-16T04:52:40.949510+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:40.949510+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1676556,"num_objects":205,"num_object_clones":0,"num_object_copies":820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12393,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2008}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12586","reported_seq":17154,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:44.028935+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:44.028935+0000","last_peered":"2024-09-16T04:52:44.028935+0000","last_clean":"2024-09-16T04:52:44.028935+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:44.028935+0000","last_undegraded":"2024-09-16T04:52:44.028935+0000","last_fullsized":"2024-09-16T04:52:44.028935+0000","mapping_epoch":32,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:45.241508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1627404,"num_objects":199,"num_object_clones":0,"num_object_copies":796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12584,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12543","reported_seq":17306,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:41.482831+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:41.482831+0000","last_peered":"2024-09-16T04:52:41.482831+0000","last_clean":"2024-09-16T04:52:41.482831+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:41.482831+0000","last_undegraded":"2024-09-16T04:52:41.482831+0000","last_fullsized":"2024-09-16T04:52:41.482831+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:41.933068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1739288,"num_objects":213,"num_object_clones":0,"num_object_copies":852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12540,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12526","reported_seq":12637,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:39.605237+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:39.605237+0000","last_peered":"2024-09-16T04:52:39.605237+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:39.605237+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1897740,"num_objects":232,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1957}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12279","reported_seq":12417,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:41.453856+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:41.453856+0000","last_peered":"2024-09-16T04:52:41.453856+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:41.453856+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1732498,"num_objects":212,"num_object_clones":0,"num_object_copies":848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12276,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1943}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12155","reported_seq":12253,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:41.501157+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:41.501157+0000","last_peered":"2024-09-16T04:52:41.501157+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:41.501157+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1440390,"num_objects":176,"num_object_clones":0,"num_object_copies":704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12154,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1929}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12291","reported_seq":12384,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:39.745865+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:39.745865+0000","last_peered":"2024-09-16T04:52:39.745865+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:39.745865+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1981062,"num_objects":242,"num_object_clones":0,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1979}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12572","reported_seq":12706,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:41.544214+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:41.544214+0000","last_peered":"2024-09-16T04:52:41.544214+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:41.544214+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1717516,"num_objects":210,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12570,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1988}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12241","reported_seq":14265,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:39.745109+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:39.745109+0000","last_peered":"2024-09-16T04:52:39.745109+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:39.745109+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1701132,"num_objects":208,"num_object_clones":0,"num_object_copies":832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12239,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-960}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12116","reported_seq":12216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:40.902116+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:40.902116+0000","last_peered":"2024-09-16T04:52:40.902116+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:40.902116+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561868,"num_objects":191,"num_object_clones":0,"num_object_copies":764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12114,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":26977984,"num_objects":3299,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198188,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8053,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53252096,"data_stored":53252096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160915,"ondisk_log_size":160915,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721596,"kb_used_data":13792,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93650244,"statfs":{"total":96636764160,"available":95897849856,"internally_reserved":0,"allocated":14123008,"data_stored":13924499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":724758223},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,7,5,4,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731692,"kb_used_data":14680,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93640148,"statfs":{"total":96636764160,"available":95887511552,"internally_reserved":0,"allocated":15032320,"data_stored":14833811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,0,2,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580704,"kb_used_data":11716,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93791136,"statfs":{"total":96636764160,"available":96042123264,"internally_reserved":0,"allocated":11997184,"data_stored":11798675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582610639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,3,0,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703944,"kb_used_data":14252,"kb_used_omap":4,"kb_used_meta":689659,"kb_avail":93667896,"statfs":{"total":96636764160,"available":95915925504,"internally_reserved":0,"allocated":14594048,"data_stored":14395539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":706211534},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,1,8,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13697024,"data_stored":13697024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11685888,"data_stored":11685888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14569472,"data_stored":14569472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13299712,"data_stored":13299712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:45.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:52:45.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12427", 'reported_seq': 12549, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:41.275522+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:52:41.275522+0000', 'last_peered': '2024-09-16T04:52:41.275522+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:52:41.275522+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1859584, 'num_objects': 227, 'num_object_clones': 0, 'num_object_copies': 908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12427, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1959}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:45.869 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:52:45.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12475", 'reported_seq': 12568, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:41.034606+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:52:41.034606+0000', 'last_peered': '2024-09-16T04:52:41.034606+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:52:41.034606+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1627404, 'num_objects': 199, 'num_object_clones': 0, 'num_object_copies': 796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12473, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2004}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:45.869 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:52:45.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12454", 'reported_seq': 12558, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:39.580455+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:52:39.580455+0000', 'last_peered': '2024-09-16T04:52:39.580455+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:52:39.580455+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1768094, 'num_objects': 217, 'num_object_clones': 0, 'num_object_copies': 868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 217, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12453, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2002}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:45.869 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:52:45.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12540", 'reported_seq': 12678, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:41.359869+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:52:41.359869+0000', 'last_peered': '2024-09-16T04:52:41.359869+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:52:41.359869+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1643788, 'num_objects': 201, 'num_object_clones': 0, 'num_object_copies': 804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12538, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2047}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:45.869 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:52:45.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12119", 'reported_seq': 12235, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:41.373255+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:52:41.373255+0000', 'last_peered': '2024-09-16T04:52:41.373255+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:52:41.373255+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1581056, 'num_objects': 193, 'num_object_clones': 0, 'num_object_copies': 772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12119, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1985}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:45.870 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:52:45.870 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12395", 'reported_seq': 12498, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:40.949510+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:52:40.949510+0000', 'last_peered': '2024-09-16T04:52:40.949510+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:52:40.949510+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1676556, 'num_objects': 205, 'num_object_clones': 0, 'num_object_copies': 820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12393, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2008}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:45.870 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:52:45.870 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12526", 'reported_seq': 12637, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:39.605237+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:52:39.605237+0000', 'last_peered': '2024-09-16T04:52:39.605237+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:52:39.605237+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1897740, 'num_objects': 232, 'num_object_clones': 0, 'num_object_copies': 928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12524, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1957}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:45.870 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:52:45.870 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12279", 'reported_seq': 12417, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:41.453856+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:52:41.453856+0000', 'last_peered': '2024-09-16T04:52:41.453856+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:52:41.453856+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1732498, 'num_objects': 212, 'num_object_clones': 0, 'num_object_copies': 848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 212, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12276, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1943}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:45.870 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:52:45.870 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12155", 'reported_seq': 12253, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:41.501157+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:52:41.501157+0000', 'last_peered': '2024-09-16T04:52:41.501157+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:52:41.501157+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1440390, 'num_objects': 176, 'num_object_clones': 0, 'num_object_copies': 704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 176, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12154, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1929}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:45.870 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:52:45.870 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12291", 'reported_seq': 12384, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:39.745865+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:52:39.745865+0000', 'last_peered': '2024-09-16T04:52:39.745865+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:52:39.745865+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1981062, 'num_objects': 242, 'num_object_clones': 0, 'num_object_copies': 968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 242, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12290, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1979}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:45.870 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:52:45.871 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12572", 'reported_seq': 12706, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:41.544214+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:52:41.544214+0000', 'last_peered': '2024-09-16T04:52:41.544214+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:52:41.544214+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1717516, 'num_objects': 210, 'num_object_clones': 0, 'num_object_copies': 840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12570, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1988}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:45.871 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:52:45.871 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12241", 'reported_seq': 14265, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:39.745109+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:52:39.745109+0000', 'last_peered': '2024-09-16T04:52:39.745109+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:52:39.745109+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1701132, 'num_objects': 208, 'num_object_clones': 0, 'num_object_copies': 832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1168, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 208, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12239, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 968, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -960}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:45.871 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:52:45.871 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12116", 'reported_seq': 12216, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:40.902116+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:52:40.902116+0000', 'last_peered': '2024-09-16T04:52:40.902116+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:52:40.902116+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1561868, 'num_objects': 191, 'num_object_clones': 0, 'num_object_copies': 764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12114, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:45.871 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:52:46.619 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:46.619 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27529/13196 objects degraded (208.616%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":778,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":3299,"data_bytes":26977984,"bytes_used":2803646464,"bytes_avail":383743410176,"bytes_total":386547056640,"degraded_objects":27529,"degraded_total":13196,"degraded_ratio":2.0861624734768109,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":171},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:52:46.637 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:52:46.637 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:47.201 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:47.202 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:47.225 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2024-09-16T04:52:46.800377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20014784,"num_objects":2449,"num_object_clones":0,"num_object_copies":9796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":199038,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8114,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160765,"ondisk_log_size":160765,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2723472,"kb_used_data":42736,"kb_used_omap":14,"kb_used_meta":2680625,"kb_avail":374763888,"statfs":{"total":386547056640,"available":383758221312,"internally_reserved":0,"allocated":43761664,"data_stored":42967628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2744960561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,12,0,10,15,13,10,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22855680,"num_objects":-2790,"num_object_clones":0,"num_object_copies":-11160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2790,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004797"},"pg_stats":[{"pgid":"1.f","version":"37'12504","reported_seq":12626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.511647+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:46.511647+0000","last_peered":"2024-09-16T04:52:46.511647+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:46.511647+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1228800,"num_objects":150,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12504,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12530","reported_seq":12624,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:45.937623+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:45.937623+0000","last_peered":"2024-09-16T04:52:45.937623+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:45.937623+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1176844,"num_objects":144,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12528,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12512","reported_seq":12616,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:44.783329+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:44.783329+0000","last_peered":"2024-09-16T04:52:44.783329+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:44.783329+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1292958,"num_objects":159,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12511,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2060}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12496","reported_seq":17046,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:44.147825+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:44.147825+0000","last_peered":"2024-09-16T04:52:44.147825+0000","last_clean":"2024-09-16T04:52:44.147825+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:44.147825+0000","last_undegraded":"2024-09-16T04:52:44.147825+0000","last_fullsized":"2024-09-16T04:52:44.147825+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:45.243479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1422604,"num_objects":174,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12494,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12607","reported_seq":12742,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.433854+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:46.433854+0000","last_peered":"2024-09-16T04:52:46.433854+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:46.433854+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1094924,"num_objects":134,"num_object_clones":0,"num_object_copies":536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12605,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2114}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12173","reported_seq":12289,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.058851+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:46.058851+0000","last_peered":"2024-09-16T04:52:46.058851+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:46.058851+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1138688,"num_objects":139,"num_object_clones":0,"num_object_copies":556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12173,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12449","reported_seq":12552,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:45.988213+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:45.988213+0000","last_peered":"2024-09-16T04:52:45.988213+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:45.988213+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234188,"num_objects":151,"num_object_clones":0,"num_object_copies":604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12447,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2062}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12586","reported_seq":17154,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:44.028935+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:44.028935+0000","last_peered":"2024-09-16T04:52:44.028935+0000","last_clean":"2024-09-16T04:52:44.028935+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:44.028935+0000","last_undegraded":"2024-09-16T04:52:44.028935+0000","last_fullsized":"2024-09-16T04:52:44.028935+0000","mapping_epoch":32,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:45.241508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1627404,"num_objects":199,"num_object_clones":0,"num_object_copies":796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12584,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12604","reported_seq":17367,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:46.488659+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:46.488659+0000","last_peered":"2024-09-16T04:52:46.488659+0000","last_clean":"2024-09-16T04:52:46.488659+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:46.488659+0000","last_undegraded":"2024-09-16T04:52:46.488659+0000","last_fullsized":"2024-09-16T04:52:46.488659+0000","mapping_epoch":34,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:46.934778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1239576,"num_objects":152,"num_object_clones":0,"num_object_copies":608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12601,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12579","reported_seq":12691,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:44.781895+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:44.781895+0000","last_peered":"2024-09-16T04:52:44.781895+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:44.781895+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1463564,"num_objects":179,"num_object_clones":0,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12577,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2010}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12341","reported_seq":12479,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.494280+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:46.494280+0000","last_peered":"2024-09-16T04:52:46.494280+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:46.494280+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1224594,"num_objects":150,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12338,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12218","reported_seq":12316,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.138515+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:46.138515+0000","last_peered":"2024-09-16T04:52:46.138515+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:46.138515+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":924294,"num_objects":113,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12217,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1992}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12367","reported_seq":12461,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:44.656308+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:44.656308+0000","last_peered":"2024-09-16T04:52:44.656308+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:44.656308+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1358470,"num_objects":166,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12366,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2055}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12635","reported_seq":12773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.330994+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:46.330994+0000","last_peered":"2024-09-16T04:52:46.330994+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:46.330994+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1201420,"num_objects":147,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12633,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2051}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12296","reported_seq":14440,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:44.867306+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:44.867306+0000","last_peered":"2024-09-16T04:52:44.867306+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:44.867306+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1250572,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12294,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1029,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12168","reported_seq":12268,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:45.886121+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:45.886121+0000","last_peered":"2024-09-16T04:52:45.886121+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:45.886121+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1135884,"num_objects":139,"num_object_clones":0,"num_object_copies":556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12166,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2089}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":20014784,"num_objects":2449,"num_object_clones":0,"num_object_copies":9796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":199038,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8114,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41332736,"data_stored":41332736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160765,"ondisk_log_size":160765,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705208,"kb_used_data":9896,"kb_used_omap":4,"kb_used_meta":695291,"kb_avail":93666632,"statfs":{"total":96636764160,"available":95914631168,"internally_reserved":0,"allocated":10133504,"data_stored":9934995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":711978703},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,7,5,4,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731984,"kb_used_data":10804,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93639856,"statfs":{"total":96636764160,"available":95887212544,"internally_reserved":0,"allocated":11063296,"data_stored":10864787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,0,5,6,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580704,"kb_used_data":11716,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93791136,"statfs":{"total":96636764160,"available":96042123264,"internally_reserved":0,"allocated":11997184,"data_stored":11798675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582610639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,3,0,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705576,"kb_used_data":10320,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93666264,"statfs":{"total":96636764160,"available":95914254336,"internally_reserved":0,"allocated":10567680,"data_stored":10369171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":711913166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,1,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9707520,"data_stored":9707520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11685888,"data_stored":11685888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10600448,"data_stored":10600448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9338880,"data_stored":9338880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:50.228 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:50.842 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:50.842 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:50.861 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-16T04:52:50.804443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":16901824,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":199418,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8171,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160645,"ondisk_log_size":160645,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2719952,"kb_used_data":35068,"kb_used_omap":14,"kb_used_meta":2684785,"kb_avail":374767408,"statfs":{"total":386547056640,"available":383761825792,"internally_reserved":0,"allocated":35909632,"data_stored":35115596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2749220401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,3,7,18,24,15,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18939904,"num_objects":-2312,"num_object_clones":0,"num_object_copies":-9248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2312,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007087"},"pg_stats":[{"pgid":"1.f","version":"37'12504","reported_seq":12626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.511647+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:46.511647+0000","last_peered":"2024-09-16T04:52:46.511647+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:46.511647+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1228800,"num_objects":150,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12504,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12530","reported_seq":12624,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:45.937623+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:45.937623+0000","last_peered":"2024-09-16T04:52:45.937623+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:45.937623+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1176844,"num_objects":144,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12528,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12567","reported_seq":12671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.831917+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:49.831917+0000","last_peered":"2024-09-16T04:52:49.831917+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:49.831917+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":842398,"num_objects":104,"num_object_clones":0,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12566,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2115}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12556","reported_seq":17106,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:48.944334+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:48.944334+0000","last_peered":"2024-09-16T04:52:48.944334+0000","last_clean":"2024-09-16T04:52:48.944334+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:48.944334+0000","last_undegraded":"2024-09-16T04:52:48.944334+0000","last_fullsized":"2024-09-16T04:52:48.944334+0000","mapping_epoch":36,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:50.289804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":931084,"num_objects":114,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12554,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12607","reported_seq":12742,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.433854+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:46.433854+0000","last_peered":"2024-09-16T04:52:46.433854+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:46.433854+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1094924,"num_objects":134,"num_object_clones":0,"num_object_copies":536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12605,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2114}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12173","reported_seq":12289,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.058851+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:46.058851+0000","last_peered":"2024-09-16T04:52:46.058851+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:46.058851+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1138688,"num_objects":139,"num_object_clones":0,"num_object_copies":556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12173,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12449","reported_seq":12552,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:45.988213+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:45.988213+0000","last_peered":"2024-09-16T04:52:45.988213+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:45.988213+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234188,"num_objects":151,"num_object_clones":0,"num_object_copies":604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12447,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2062}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12650","reported_seq":17218,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:49.209134+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:49.209134+0000","last_peered":"2024-09-16T04:52:49.209134+0000","last_clean":"2024-09-16T04:52:49.209134+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:49.209134+0000","last_undegraded":"2024-09-16T04:52:49.209134+0000","last_fullsized":"2024-09-16T04:52:49.209134+0000","mapping_epoch":32,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:50.281409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1103116,"num_objects":135,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12648,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12604","reported_seq":17367,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:46.488659+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:46.488659+0000","last_peered":"2024-09-16T04:52:46.488659+0000","last_clean":"2024-09-16T04:52:46.488659+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:46.488659+0000","last_undegraded":"2024-09-16T04:52:46.488659+0000","last_fullsized":"2024-09-16T04:52:46.488659+0000","mapping_epoch":34,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:46.934778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1239576,"num_objects":152,"num_object_clones":0,"num_object_copies":608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12601,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12666","reported_seq":12777,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.830310+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:49.830310+0000","last_peered":"2024-09-16T04:52:49.830310+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:49.830310+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":750860,"num_objects":92,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12664,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2097}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12341","reported_seq":12479,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.494280+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:46.494280+0000","last_peered":"2024-09-16T04:52:46.494280+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:46.494280+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1224594,"num_objects":150,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12338,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12218","reported_seq":12316,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.138515+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:46.138515+0000","last_peered":"2024-09-16T04:52:46.138515+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:46.138515+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":924294,"num_objects":113,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12217,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1992}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12424","reported_seq":12518,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.906026+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:49.906026+0000","last_peered":"2024-09-16T04:52:49.906026+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:49.906026+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":891526,"num_objects":109,"num_object_clones":0,"num_object_copies":436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12423,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2112}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12635","reported_seq":12773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.330994+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:46.330994+0000","last_peered":"2024-09-16T04:52:46.330994+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:46.330994+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1201420,"num_objects":147,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12633,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2051}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12353","reported_seq":14613,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.908992+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:49.908992+0000","last_peered":"2024-09-16T04:52:49.908992+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:49.908992+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":783628,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12351,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12168","reported_seq":12268,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:45.886121+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:45.886121+0000","last_peered":"2024-09-16T04:52:45.886121+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:45.886121+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1135884,"num_objects":139,"num_object_clones":0,"num_object_copies":556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12166,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2089}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":16901824,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":199418,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8171,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33386496,"data_stored":33386496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160645,"ondisk_log_size":160645,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705208,"kb_used_data":9896,"kb_used_omap":4,"kb_used_meta":695291,"kb_avail":93666632,"statfs":{"total":96636764160,"available":95914631168,"internally_reserved":0,"allocated":10133504,"data_stored":9934995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":711978703},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,7,5,4,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728168,"kb_used_data":7000,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93643672,"statfs":{"total":96636764160,"available":95891120128,"internally_reserved":0,"allocated":7168000,"data_stored":6969491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581000,"kb_used_data":7852,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93790840,"statfs":{"total":96636764160,"available":96041820160,"internally_reserved":0,"allocated":8040448,"data_stored":7841939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586870479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,5,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705576,"kb_used_data":10320,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93666264,"statfs":{"total":96636764160,"available":95914254336,"internally_reserved":0,"allocated":10567680,"data_stored":10369171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":711913166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,1,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9707520,"data_stored":9707520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7716864,"data_stored":7716864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6623232,"data_stored":6623232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9338880,"data_stored":9338880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:50.862 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:51.505 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:51.506 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:51.552 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-16T04:52:50.804443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":16901824,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":199418,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8171,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160645,"ondisk_log_size":160645,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2719952,"kb_used_data":35068,"kb_used_omap":14,"kb_used_meta":2684785,"kb_avail":374767408,"statfs":{"total":386547056640,"available":383761825792,"internally_reserved":0,"allocated":35909632,"data_stored":35115596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2749220401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,3,7,18,24,15,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18939904,"num_objects":-2312,"num_object_clones":0,"num_object_copies":-9248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2312,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007087"},"pg_stats":[{"pgid":"1.f","version":"37'12504","reported_seq":12626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.511647+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:46.511647+0000","last_peered":"2024-09-16T04:52:46.511647+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:46.511647+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1228800,"num_objects":150,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12504,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12530","reported_seq":12624,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:45.937623+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:45.937623+0000","last_peered":"2024-09-16T04:52:45.937623+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:45.937623+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1176844,"num_objects":144,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12528,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12567","reported_seq":12671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.831917+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:49.831917+0000","last_peered":"2024-09-16T04:52:49.831917+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:49.831917+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":842398,"num_objects":104,"num_object_clones":0,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12566,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2115}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12556","reported_seq":17106,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:48.944334+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:48.944334+0000","last_peered":"2024-09-16T04:52:48.944334+0000","last_clean":"2024-09-16T04:52:48.944334+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:48.944334+0000","last_undegraded":"2024-09-16T04:52:48.944334+0000","last_fullsized":"2024-09-16T04:52:48.944334+0000","mapping_epoch":36,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:50.289804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":931084,"num_objects":114,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12554,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12607","reported_seq":12742,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.433854+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:46.433854+0000","last_peered":"2024-09-16T04:52:46.433854+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:46.433854+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1094924,"num_objects":134,"num_object_clones":0,"num_object_copies":536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12605,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2114}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12173","reported_seq":12289,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.058851+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:46.058851+0000","last_peered":"2024-09-16T04:52:46.058851+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:46.058851+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1138688,"num_objects":139,"num_object_clones":0,"num_object_copies":556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12173,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12449","reported_seq":12552,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:45.988213+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:45.988213+0000","last_peered":"2024-09-16T04:52:45.988213+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:45.988213+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234188,"num_objects":151,"num_object_clones":0,"num_object_copies":604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12447,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2062}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12650","reported_seq":17218,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:49.209134+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:49.209134+0000","last_peered":"2024-09-16T04:52:49.209134+0000","last_clean":"2024-09-16T04:52:49.209134+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:49.209134+0000","last_undegraded":"2024-09-16T04:52:49.209134+0000","last_fullsized":"2024-09-16T04:52:49.209134+0000","mapping_epoch":32,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:50.281409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1103116,"num_objects":135,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12648,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12604","reported_seq":17367,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:46.488659+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:46.488659+0000","last_peered":"2024-09-16T04:52:46.488659+0000","last_clean":"2024-09-16T04:52:46.488659+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:46.488659+0000","last_undegraded":"2024-09-16T04:52:46.488659+0000","last_fullsized":"2024-09-16T04:52:46.488659+0000","mapping_epoch":34,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:46.934778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1239576,"num_objects":152,"num_object_clones":0,"num_object_copies":608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12601,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12666","reported_seq":12777,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.830310+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:49.830310+0000","last_peered":"2024-09-16T04:52:49.830310+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:49.830310+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":750860,"num_objects":92,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12664,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2097}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12341","reported_seq":12479,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.494280+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:46.494280+0000","last_peered":"2024-09-16T04:52:46.494280+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:46.494280+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1224594,"num_objects":150,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12338,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12218","reported_seq":12316,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.138515+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:46.138515+0000","last_peered":"2024-09-16T04:52:46.138515+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:46.138515+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":924294,"num_objects":113,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12217,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1992}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12424","reported_seq":12518,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.906026+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:49.906026+0000","last_peered":"2024-09-16T04:52:49.906026+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:49.906026+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":891526,"num_objects":109,"num_object_clones":0,"num_object_copies":436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12423,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2112}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12635","reported_seq":12773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:46.330994+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:46.330994+0000","last_peered":"2024-09-16T04:52:46.330994+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:46.330994+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1201420,"num_objects":147,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12633,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2051}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12353","reported_seq":14613,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.908992+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:49.908992+0000","last_peered":"2024-09-16T04:52:49.908992+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:49.908992+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":783628,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12351,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12168","reported_seq":12268,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:45.886121+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:45.886121+0000","last_peered":"2024-09-16T04:52:45.886121+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:45.886121+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1135884,"num_objects":139,"num_object_clones":0,"num_object_copies":556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12166,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2089}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":16901824,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":199418,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8171,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33386496,"data_stored":33386496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160645,"ondisk_log_size":160645,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705208,"kb_used_data":9896,"kb_used_omap":4,"kb_used_meta":695291,"kb_avail":93666632,"statfs":{"total":96636764160,"available":95914631168,"internally_reserved":0,"allocated":10133504,"data_stored":9934995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":711978703},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,7,5,4,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728168,"kb_used_data":7000,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93643672,"statfs":{"total":96636764160,"available":95891120128,"internally_reserved":0,"allocated":7168000,"data_stored":6969491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581000,"kb_used_data":7852,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93790840,"statfs":{"total":96636764160,"available":96041820160,"internally_reserved":0,"allocated":8040448,"data_stored":7841939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586870479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,5,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705576,"kb_used_data":10320,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93666264,"statfs":{"total":96636764160,"available":95914254336,"internally_reserved":0,"allocated":10567680,"data_stored":10369171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":711913166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,1,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9707520,"data_stored":9707520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7716864,"data_stored":7716864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6623232,"data_stored":6623232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9338880,"data_stored":9338880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:51.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:52:51.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12504", 'reported_seq': 12626, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:46.511647+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:52:46.511647+0000', 'last_peered': '2024-09-16T04:52:46.511647+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:52:46.511647+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1228800, 'num_objects': 150, 'num_object_clones': 0, 'num_object_copies': 600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 150, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12504, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2036}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:51.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:52:51.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12530", 'reported_seq': 12624, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:45.937623+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:52:45.937623+0000', 'last_peered': '2024-09-16T04:52:45.937623+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:52:45.937623+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1176844, 'num_objects': 144, 'num_object_clones': 0, 'num_object_copies': 576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 144, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12528, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:51.553 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:52:51.553 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12567", 'reported_seq': 12671, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:49.831917+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:52:49.831917+0000', 'last_peered': '2024-09-16T04:52:49.831917+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:52:49.831917+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 842398, 'num_objects': 104, 'num_object_clones': 0, 'num_object_copies': 416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 104, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12566, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2115}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:51.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:52:51.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12607", 'reported_seq': 12742, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:46.433854+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:52:46.433854+0000', 'last_peered': '2024-09-16T04:52:46.433854+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:52:46.433854+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1094924, 'num_objects': 134, 'num_object_clones': 0, 'num_object_copies': 536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12605, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2114}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:51.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:52:51.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12173", 'reported_seq': 12289, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:46.058851+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:52:46.058851+0000', 'last_peered': '2024-09-16T04:52:46.058851+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:52:46.058851+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1138688, 'num_objects': 139, 'num_object_clones': 0, 'num_object_copies': 556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 139, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12173, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2039}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:51.554 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:52:51.554 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12449", 'reported_seq': 12552, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:45.988213+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:52:45.988213+0000', 'last_peered': '2024-09-16T04:52:45.988213+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:52:45.988213+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234188, 'num_objects': 151, 'num_object_clones': 0, 'num_object_copies': 604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12447, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2062}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:51.556 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:52:51.556 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12666", 'reported_seq': 12777, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:49.830310+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:52:49.830310+0000', 'last_peered': '2024-09-16T04:52:49.830310+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:52:49.830310+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 750860, 'num_objects': 92, 'num_object_clones': 0, 'num_object_copies': 368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 92, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12664, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2097}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:51.556 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:52:51.556 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12341", 'reported_seq': 12479, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:46.494280+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:52:46.494280+0000', 'last_peered': '2024-09-16T04:52:46.494280+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:52:46.494280+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1224594, 'num_objects': 150, 'num_object_clones': 0, 'num_object_copies': 600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 150, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12338, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2005}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:51.556 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:52:51.556 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12218", 'reported_seq': 12316, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:46.138515+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:52:46.138515+0000', 'last_peered': '2024-09-16T04:52:46.138515+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:52:46.138515+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 924294, 'num_objects': 113, 'num_object_clones': 0, 'num_object_copies': 452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12217, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1992}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:51.556 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:52:51.556 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12424", 'reported_seq': 12518, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:49.906026+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:52:49.906026+0000', 'last_peered': '2024-09-16T04:52:49.906026+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:52:49.906026+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 891526, 'num_objects': 109, 'num_object_clones': 0, 'num_object_copies': 436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 109, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12423, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2112}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:51.556 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:52:51.556 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12635", 'reported_seq': 12773, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:46.330994+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:52:46.330994+0000', 'last_peered': '2024-09-16T04:52:46.330994+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:52:46.330994+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1201420, 'num_objects': 147, 'num_object_clones': 0, 'num_object_copies': 588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 147, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12633, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2051}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:51.557 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:52:51.557 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12353", 'reported_seq': 14613, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:49.908992+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:52:49.908992+0000', 'last_peered': '2024-09-16T04:52:49.908992+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:52:49.908992+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 783628, 'num_objects': 96, 'num_object_clones': 0, 'num_object_copies': 384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 96, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12351, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1086, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:51.557 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:52:51.557 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12168", 'reported_seq': 12268, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:45.886121+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:52:45.886121+0000', 'last_peered': '2024-09-16T04:52:45.886121+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:52:45.886121+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1135884, 'num_objects': 139, 'num_object_clones': 0, 'num_object_copies': 556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 139, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12166, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2089}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:51.557 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:52:52.249 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:52.249 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27411/8276 objects degraded (331.211%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":784,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":2069,"data_bytes":16901824,"bytes_used":2785230848,"bytes_avail":383761825792,"bytes_total":386547056640,"degraded_objects":27411,"degraded_total":8276,"degraded_ratio":3.3121072982116964,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":192},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:52:52.279 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:52:52.279 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:52.903 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:52.903 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:52.934 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-09-16T04:52:52.805304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12134080,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200000,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8171,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2720660,"kb_used_data":27416,"kb_used_omap":14,"kb_used_meta":2693105,"kb_avail":374766700,"statfs":{"total":386547056640,"available":383761100800,"internally_reserved":0,"allocated":28073984,"data_stored":27279948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2757740081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,13,12,16,16,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20480000,"num_objects":-2500,"num_object_clones":0,"num_object_copies":-10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2500,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007088"},"pg_stats":[{"pgid":"1.f","version":"37'12566","reported_seq":12688,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.540439+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:51.540439+0000","last_peered":"2024-09-16T04:52:51.540439+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:51.540439+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":88,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12566,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12579","reported_seq":12673,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:50.848467+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:50.848467+0000","last_peered":"2024-09-16T04:52:50.848467+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:50.848467+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":775436,"num_objects":95,"num_object_clones":0,"num_object_copies":380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12577,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12567","reported_seq":12671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.831917+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:49.831917+0000","last_peered":"2024-09-16T04:52:49.831917+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:49.831917+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":842398,"num_objects":104,"num_object_clones":0,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12566,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2115}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12556","reported_seq":17106,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:48.944334+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:48.944334+0000","last_peered":"2024-09-16T04:52:48.944334+0000","last_clean":"2024-09-16T04:52:48.944334+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:48.944334+0000","last_undegraded":"2024-09-16T04:52:48.944334+0000","last_fullsized":"2024-09-16T04:52:48.944334+0000","mapping_epoch":36,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:50.289804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":931084,"num_objects":114,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12554,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12660","reported_seq":12798,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.486105+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:51.486105+0000","last_peered":"2024-09-16T04:52:51.486105+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:51.486105+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":660748,"num_objects":81,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12228","reported_seq":12343,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.657434+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:51.657434+0000","last_peered":"2024-09-16T04:52:51.657434+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:51.657434+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":688128,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12506","reported_seq":12609,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.033976+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:51.033976+0000","last_peered":"2024-09-16T04:52:51.033976+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:51.033976+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":767244,"num_objects":94,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12504,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2119}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12650","reported_seq":17218,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:49.209134+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:49.209134+0000","last_peered":"2024-09-16T04:52:49.209134+0000","last_clean":"2024-09-16T04:52:49.209134+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:49.209134+0000","last_undegraded":"2024-09-16T04:52:49.209134+0000","last_fullsized":"2024-09-16T04:52:49.209134+0000","mapping_epoch":32,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:50.281409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1103116,"num_objects":135,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12648,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12670","reported_seq":17433,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:51.512365+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:51.512365+0000","last_peered":"2024-09-16T04:52:51.512365+0000","last_clean":"2024-09-16T04:52:51.512365+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:51.512365+0000","last_undegraded":"2024-09-16T04:52:51.512365+0000","last_fullsized":"2024-09-16T04:52:51.512365+0000","mapping_epoch":34,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:52.030420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":698904,"num_objects":86,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12667,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12666","reported_seq":12777,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.830310+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:49.830310+0000","last_peered":"2024-09-16T04:52:49.830310+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:49.830310+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":750860,"num_objects":92,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12664,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2097}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12402","reported_seq":12540,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.580313+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:51.580313+0000","last_peered":"2024-09-16T04:52:51.580313+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:51.580313+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724882,"num_objects":89,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12399,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12273","reported_seq":12371,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.557349+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:51.557349+0000","last_peered":"2024-09-16T04:52:51.557349+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:51.557349+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473734,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12272,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2047}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12424","reported_seq":12518,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.906026+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:49.906026+0000","last_peered":"2024-09-16T04:52:49.906026+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:49.906026+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":891526,"num_objects":109,"num_object_clones":0,"num_object_copies":436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12423,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2112}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12700","reported_seq":12837,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.630273+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:51.630273+0000","last_peered":"2024-09-16T04:52:51.630273+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:51.630273+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":668940,"num_objects":82,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12698,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2116}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12353","reported_seq":14613,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.908992+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:49.908992+0000","last_peered":"2024-09-16T04:52:49.908992+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:49.908992+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":783628,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12351,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12227","reported_seq":12327,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.122432+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:51.122432+0000","last_peered":"2024-09-16T04:52:51.122432+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:51.122432+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":652556,"num_objects":80,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12225,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2148}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":12134080,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200000,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8171,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25407488,"data_stored":25407488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705524,"kb_used_data":6028,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93666316,"statfs":{"total":96636764160,"available":95914307584,"internally_reserved":0,"allocated":6172672,"data_stored":5974163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716238543},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,1,0,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728168,"kb_used_data":7000,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93643672,"statfs":{"total":96636764160,"available":95891120128,"internally_reserved":0,"allocated":7168000,"data_stored":6969491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581000,"kb_used_data":7852,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93790840,"statfs":{"total":96636764160,"available":96041820160,"internally_reserved":0,"allocated":8040448,"data_stored":7841939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586870479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,5,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705968,"kb_used_data":6536,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93665872,"statfs":{"total":96636764160,"available":95913852928,"internally_reserved":0,"allocated":6692864,"data_stored":6494355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":716173006},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,2,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5767168,"data_stored":5767168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7716864,"data_stored":7716864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6623232,"data_stored":6623232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5300224,"data_stored":5300224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:55.937 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:56.525 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:56.526 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:56.547 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-16T04:52:54.806031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10921664,"num_objects":1339,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200148,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8171,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160775,"ondisk_log_size":160775,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2721048,"kb_used_data":23652,"kb_used_omap":14,"kb_used_meta":2697265,"kb_avail":374766312,"statfs":{"total":386547056640,"available":383760703488,"internally_reserved":0,"allocated":24219648,"data_stored":23425612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2761999921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,8,9,11,12,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16900096,"num_objects":-2063,"num_object_clones":0,"num_object_copies":-8252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2063,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006878"},"pg_stats":[{"pgid":"1.f","version":"37'12566","reported_seq":12688,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.540439+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:51.540439+0000","last_peered":"2024-09-16T04:52:51.540439+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:51.540439+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":88,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12566,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12579","reported_seq":12673,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:50.848467+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:50.848467+0000","last_peered":"2024-09-16T04:52:50.848467+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:50.848467+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":775436,"num_objects":95,"num_object_clones":0,"num_object_copies":380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12577,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12567","reported_seq":12671,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.831917+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:49.831917+0000","last_peered":"2024-09-16T04:52:49.831917+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:49.831917+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":842398,"num_objects":104,"num_object_clones":0,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12566,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2115}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12623","reported_seq":17173,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:54.300516+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:54.300516+0000","last_peered":"2024-09-16T04:52:54.300516+0000","last_clean":"2024-09-16T04:52:54.300516+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:54.300516+0000","last_undegraded":"2024-09-16T04:52:54.300516+0000","last_fullsized":"2024-09-16T04:52:54.300516+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:56.296838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382220,"num_objects":47,"num_object_clones":0,"num_object_copies":188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12621,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12660","reported_seq":12798,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.486105+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:51.486105+0000","last_peered":"2024-09-16T04:52:51.486105+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:51.486105+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":660748,"num_objects":81,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12228","reported_seq":12343,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.657434+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:51.657434+0000","last_peered":"2024-09-16T04:52:51.657434+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:51.657434+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":688128,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12506","reported_seq":12609,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.033976+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:51.033976+0000","last_peered":"2024-09-16T04:52:51.033976+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:51.033976+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":767244,"num_objects":94,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12504,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2119}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12731","reported_seq":17298,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:54.390608+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:54.390608+0000","last_peered":"2024-09-16T04:52:54.390608+0000","last_clean":"2024-09-16T04:52:54.390608+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:54.390608+0000","last_undegraded":"2024-09-16T04:52:54.390608+0000","last_fullsized":"2024-09-16T04:52:54.390608+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:56.296814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439564,"num_objects":54,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12729,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12670","reported_seq":17433,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:51.512365+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:51.512365+0000","last_peered":"2024-09-16T04:52:51.512365+0000","last_clean":"2024-09-16T04:52:51.512365+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:51.512365+0000","last_undegraded":"2024-09-16T04:52:51.512365+0000","last_fullsized":"2024-09-16T04:52:51.512365+0000","mapping_epoch":34,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:52.030420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":698904,"num_objects":86,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12667,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12666","reported_seq":12777,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.830310+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:49.830310+0000","last_peered":"2024-09-16T04:52:49.830310+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:49.830310+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":750860,"num_objects":92,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12664,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2097}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12402","reported_seq":12540,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.580313+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:51.580313+0000","last_peered":"2024-09-16T04:52:51.580313+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:51.580313+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724882,"num_objects":89,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12399,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12273","reported_seq":12371,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.557349+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:51.557349+0000","last_peered":"2024-09-16T04:52:51.557349+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:51.557349+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473734,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12272,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2047}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12424","reported_seq":12518,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.906026+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:49.906026+0000","last_peered":"2024-09-16T04:52:49.906026+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:49.906026+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":891526,"num_objects":109,"num_object_clones":0,"num_object_copies":436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12423,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2112}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12700","reported_seq":12837,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.630273+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:51.630273+0000","last_peered":"2024-09-16T04:52:51.630273+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:51.630273+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":668940,"num_objects":82,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12698,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2116}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12353","reported_seq":14613,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:49.908992+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:49.908992+0000","last_peered":"2024-09-16T04:52:49.908992+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:49.908992+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":783628,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12351,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12227","reported_seq":12327,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.122432+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:51.122432+0000","last_peered":"2024-09-16T04:52:51.122432+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:51.122432+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":652556,"num_objects":80,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12225,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2148}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":10921664,"num_objects":1339,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200148,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8171,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21467136,"data_stored":21467136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160775,"ondisk_log_size":160775,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705524,"kb_used_data":6028,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93666316,"statfs":{"total":96636764160,"available":95914307584,"internally_reserved":0,"allocated":6172672,"data_stored":5974163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716238543},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,1,0,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728168,"kb_used_data":7000,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93643672,"statfs":{"total":96636764160,"available":95891120128,"internally_reserved":0,"allocated":7168000,"data_stored":6969491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581388,"kb_used_data":4088,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93790452,"statfs":{"total":96636764160,"available":96041422848,"internally_reserved":0,"allocated":4186112,"data_stored":3987603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591130319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705968,"kb_used_data":6536,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93665872,"statfs":{"total":96636764160,"available":95913852928,"internally_reserved":0,"allocated":6692864,"data_stored":6494355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":716173006},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,2,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5767168,"data_stored":5767168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3776512,"data_stored":3776512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6623232,"data_stored":6623232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5300224,"data_stored":5300224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:56.548 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:57.229 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:57.230 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:57.252 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-09-16T04:52:56.806600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7562944,"num_objects":929,"num_object_clones":0,"num_object_copies":3716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200558,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8242,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2757896,"kb_used_data":16092,"kb_used_omap":14,"kb_used_meta":2741681,"kb_avail":374729464,"statfs":{"total":386547056640,"available":383722971136,"internally_reserved":0,"allocated":16478208,"data_stored":15684172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2807481905},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,7,14,5,4,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19415040,"num_objects":-2370,"num_object_clones":0,"num_object_copies":-9480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2370,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006834"},"pg_stats":[{"pgid":"1.f","version":"37'12566","reported_seq":12688,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.540439+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:51.540439+0000","last_peered":"2024-09-16T04:52:51.540439+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:51.540439+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":88,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12566,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12647","reported_seq":12741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.118612+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:56.118612+0000","last_peered":"2024-09-16T04:52:56.118612+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:56.118612+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218380,"num_objects":27,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12645,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12627","reported_seq":12731,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:54.695407+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:54.695407+0000","last_peered":"2024-09-16T04:52:54.695407+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:54.695407+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350878,"num_objects":44,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12626,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2175}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12623","reported_seq":17173,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:54.300516+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:54.300516+0000","last_peered":"2024-09-16T04:52:54.300516+0000","last_clean":"2024-09-16T04:52:54.300516+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:54.300516+0000","last_undegraded":"2024-09-16T04:52:54.300516+0000","last_fullsized":"2024-09-16T04:52:54.300516+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:56.296838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382220,"num_objects":47,"num_object_clones":0,"num_object_copies":188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12621,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12660","reported_seq":12798,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.486105+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:51.486105+0000","last_peered":"2024-09-16T04:52:51.486105+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:51.486105+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":660748,"num_objects":81,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12228","reported_seq":12343,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.657434+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:51.657434+0000","last_peered":"2024-09-16T04:52:51.657434+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:51.657434+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":688128,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12564","reported_seq":12667,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:55.907009+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:55.907009+0000","last_peered":"2024-09-16T04:52:55.907009+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:55.907009+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292108,"num_objects":36,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12562,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2177}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12731","reported_seq":17298,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:54.390608+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:54.390608+0000","last_peered":"2024-09-16T04:52:54.390608+0000","last_clean":"2024-09-16T04:52:54.390608+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:54.390608+0000","last_undegraded":"2024-09-16T04:52:54.390608+0000","last_fullsized":"2024-09-16T04:52:54.390608+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:56.296814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439564,"num_objects":54,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12729,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12670","reported_seq":17433,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:51.512365+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:51.512365+0000","last_peered":"2024-09-16T04:52:51.512365+0000","last_clean":"2024-09-16T04:52:51.512365+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:51.512365+0000","last_undegraded":"2024-09-16T04:52:51.512365+0000","last_fullsized":"2024-09-16T04:52:51.512365+0000","mapping_epoch":34,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:52.030420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":698904,"num_objects":86,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12667,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12720","reported_seq":12833,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:54.991145+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:54.991145+0000","last_peered":"2024-09-16T04:52:54.991145+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:54.991145+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308492,"num_objects":38,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12718,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2151}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12402","reported_seq":12540,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.580313+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:51.580313+0000","last_peered":"2024-09-16T04:52:51.580313+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:51.580313+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724882,"num_objects":89,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12399,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12273","reported_seq":12371,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.557349+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:51.557349+0000","last_peered":"2024-09-16T04:52:51.557349+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:51.557349+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473734,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12272,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2047}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12491","reported_seq":12585,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:54.792307+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:54.792307+0000","last_peered":"2024-09-16T04:52:54.792307+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:54.792307+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342662,"num_objects":42,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12490,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2179}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12700","reported_seq":12837,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.630273+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:51.630273+0000","last_peered":"2024-09-16T04:52:51.630273+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:51.630273+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":668940,"num_objects":82,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12698,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2116}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12403","reported_seq":14805,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:54.979496+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:54.979496+0000","last_peered":"2024-09-16T04:52:54.979496+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:54.979496+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374028,"num_objects":46,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12401,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1157,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-933}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12280","reported_seq":12380,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:55.932733+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:55.932733+0000","last_peered":"2024-09-16T04:52:55.932733+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:55.932733+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218380,"num_objects":27,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12278,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":7562944,"num_objects":929,"num_object_clones":0,"num_object_copies":3716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200558,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8242,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13508608,"data_stored":13508608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705524,"kb_used_data":6028,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93666316,"statfs":{"total":96636764160,"available":95914307584,"internally_reserved":0,"allocated":6172672,"data_stored":5974163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716238543},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,1,0,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746652,"kb_used_data":3272,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93625188,"statfs":{"total":96636764160,"available":95872192512,"internally_reserved":0,"allocated":3350528,"data_stored":3152019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,0,1,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581388,"kb_used_data":4088,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93790452,"statfs":{"total":96636764160,"available":96041422848,"internally_reserved":0,"allocated":4186112,"data_stored":3987603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591130319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724332,"kb_used_data":2704,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93647508,"statfs":{"total":96636764160,"available":95895048192,"internally_reserved":0,"allocated":2768896,"data_stored":2570387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1728512,"data_stored":1728512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3776512,"data_stored":3776512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2703360,"data_stored":2703360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5300224,"data_stored":5300224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:52:57.253 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:52:57.253 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12566", 'reported_seq': 12688, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:51.540439+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:52:51.540439+0000', 'last_peered': '2024-09-16T04:52:51.540439+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:52:51.540439+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 720896, 'num_objects': 88, 'num_object_clones': 0, 'num_object_copies': 352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 88, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12566, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2098}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:57.253 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:52:57.253 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12647", 'reported_seq': 12741, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:56.118612+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:52:56.118612+0000', 'last_peered': '2024-09-16T04:52:56.118612+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:52:56.118612+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 218380, 'num_objects': 27, 'num_object_clones': 0, 'num_object_copies': 108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 27, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12645, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2176}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:57.253 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:52:57.254 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12627", 'reported_seq': 12731, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:54.695407+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:52:54.695407+0000', 'last_peered': '2024-09-16T04:52:54.695407+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:52:54.695407+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 350878, 'num_objects': 44, 'num_object_clones': 0, 'num_object_copies': 176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 44, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12626, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2175}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:57.254 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:52:57.254 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12660", 'reported_seq': 12798, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:51.486105+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:52:51.486105+0000', 'last_peered': '2024-09-16T04:52:51.486105+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:52:51.486105+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 660748, 'num_objects': 81, 'num_object_clones': 0, 'num_object_copies': 324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 81, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12658, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2167}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:57.254 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:52:57.254 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12228", 'reported_seq': 12343, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:51.657434+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:52:51.657434+0000', 'last_peered': '2024-09-16T04:52:51.657434+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:52:51.657434+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 688128, 'num_objects': 84, 'num_object_clones': 0, 'num_object_copies': 336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 84, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12228, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2094}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:57.254 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:52:57.254 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12564", 'reported_seq': 12667, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:55.907009+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:52:55.907009+0000', 'last_peered': '2024-09-16T04:52:55.907009+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:52:55.907009+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 292108, 'num_objects': 36, 'num_object_clones': 0, 'num_object_copies': 144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 36, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12562, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2177}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:57.254 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:52:57.254 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12720", 'reported_seq': 12833, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:54.991145+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:52:54.991145+0000', 'last_peered': '2024-09-16T04:52:54.991145+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:52:54.991145+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 308492, 'num_objects': 38, 'num_object_clones': 0, 'num_object_copies': 152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 38, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12718, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2151}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:57.254 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:52:57.254 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12402", 'reported_seq': 12540, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:51.580313+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:52:51.580313+0000', 'last_peered': '2024-09-16T04:52:51.580313+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:52:51.580313+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 724882, 'num_objects': 89, 'num_object_clones': 0, 'num_object_copies': 356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 89, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12399, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2066}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:57.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:52:57.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12273", 'reported_seq': 12371, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:51.557349+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:52:51.557349+0000', 'last_peered': '2024-09-16T04:52:51.557349+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:52:51.557349+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 473734, 'num_objects': 58, 'num_object_clones': 0, 'num_object_copies': 232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 58, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12272, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2047}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:57.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:52:57.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12491", 'reported_seq': 12585, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:54.792307+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:52:54.792307+0000', 'last_peered': '2024-09-16T04:52:54.792307+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:52:54.792307+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 342662, 'num_objects': 42, 'num_object_clones': 0, 'num_object_copies': 168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 42, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12490, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2179}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:57.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:52:57.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12700", 'reported_seq': 12837, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:51.630273+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:52:51.630273+0000', 'last_peered': '2024-09-16T04:52:51.630273+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:52:51.630273+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 668940, 'num_objects': 82, 'num_object_clones': 0, 'num_object_copies': 328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 82, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12698, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2116}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:52:57.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:52:57.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12403", 'reported_seq': 14805, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:54.979496+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:52:54.979496+0000', 'last_peered': '2024-09-16T04:52:54.979496+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:52:54.979496+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 374028, 'num_objects': 46, 'num_object_clones': 0, 'num_object_copies': 184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 46, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12401, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1157, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -933}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:52:57.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:52:57.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12280", 'reported_seq': 12380, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:55.932733+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:52:55.932733+0000', 'last_peered': '2024-09-16T04:52:55.932733+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:52:55.932733+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 218380, 'num_objects': 27, 'num_object_clones': 0, 'num_object_copies': 108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 27, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12278, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:52:57.256 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:52:57.944 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:57.944 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27340/3716 objects degraded (735.737%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":790,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":929,"data_bytes":7562944,"bytes_used":2824085504,"bytes_avail":383722971136,"bytes_total":386547056640,"degraded_objects":27340,"degraded_total":3716,"degraded_ratio":7.357373519913887,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":189,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":197},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:52:57.972 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:52:57.972 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:52:58.462 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Removed 100701 objects 2024-09-16T04:52:58.462 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi152.stdout:Clean up completed and total clean up time :484.128 2024-09-16T04:52:58.525 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:52:58.526 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:52:58.538 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-09-16T04:52:56.806600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7562944,"num_objects":929,"num_object_clones":0,"num_object_copies":3716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200558,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8242,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2757896,"kb_used_data":16092,"kb_used_omap":14,"kb_used_meta":2741681,"kb_avail":374729464,"statfs":{"total":386547056640,"available":383722971136,"internally_reserved":0,"allocated":16478208,"data_stored":15684172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2807481905},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,7,14,5,4,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19415040,"num_objects":-2370,"num_object_clones":0,"num_object_copies":-9480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2370,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006834"},"pg_stats":[{"pgid":"1.f","version":"37'12566","reported_seq":12688,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.540439+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:51.540439+0000","last_peered":"2024-09-16T04:52:51.540439+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:51.540439+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":88,"num_object_clones":0,"num_object_copies":352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12566,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12647","reported_seq":12741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.118612+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:56.118612+0000","last_peered":"2024-09-16T04:52:56.118612+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:56.118612+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218380,"num_objects":27,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12645,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12627","reported_seq":12731,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:54.695407+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:54.695407+0000","last_peered":"2024-09-16T04:52:54.695407+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:54.695407+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350878,"num_objects":44,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12626,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2175}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12623","reported_seq":17173,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:54.300516+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:54.300516+0000","last_peered":"2024-09-16T04:52:54.300516+0000","last_clean":"2024-09-16T04:52:54.300516+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:54.300516+0000","last_undegraded":"2024-09-16T04:52:54.300516+0000","last_fullsized":"2024-09-16T04:52:54.300516+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:56.296838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382220,"num_objects":47,"num_object_clones":0,"num_object_copies":188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12621,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12660","reported_seq":12798,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.486105+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:51.486105+0000","last_peered":"2024-09-16T04:52:51.486105+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:51.486105+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":660748,"num_objects":81,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12228","reported_seq":12343,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.657434+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:51.657434+0000","last_peered":"2024-09-16T04:52:51.657434+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:51.657434+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":688128,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12564","reported_seq":12667,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:55.907009+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:55.907009+0000","last_peered":"2024-09-16T04:52:55.907009+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:55.907009+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292108,"num_objects":36,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12562,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2177}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12731","reported_seq":17298,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:54.390608+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:54.390608+0000","last_peered":"2024-09-16T04:52:54.390608+0000","last_clean":"2024-09-16T04:52:54.390608+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:54.390608+0000","last_undegraded":"2024-09-16T04:52:54.390608+0000","last_fullsized":"2024-09-16T04:52:54.390608+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:56.296814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439564,"num_objects":54,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12729,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12670","reported_seq":17433,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:51.512365+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:51.512365+0000","last_peered":"2024-09-16T04:52:51.512365+0000","last_clean":"2024-09-16T04:52:51.512365+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:51.512365+0000","last_undegraded":"2024-09-16T04:52:51.512365+0000","last_fullsized":"2024-09-16T04:52:51.512365+0000","mapping_epoch":34,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:52.030420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":698904,"num_objects":86,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12667,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12720","reported_seq":12833,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:54.991145+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:54.991145+0000","last_peered":"2024-09-16T04:52:54.991145+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:54.991145+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308492,"num_objects":38,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12718,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2151}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12402","reported_seq":12540,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.580313+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:51.580313+0000","last_peered":"2024-09-16T04:52:51.580313+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:51.580313+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724882,"num_objects":89,"num_object_clones":0,"num_object_copies":356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12399,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12273","reported_seq":12371,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.557349+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:51.557349+0000","last_peered":"2024-09-16T04:52:51.557349+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:51.557349+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473734,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12272,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2047}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12491","reported_seq":12585,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:54.792307+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:54.792307+0000","last_peered":"2024-09-16T04:52:54.792307+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:54.792307+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342662,"num_objects":42,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12490,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2179}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12700","reported_seq":12837,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:51.630273+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:51.630273+0000","last_peered":"2024-09-16T04:52:51.630273+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:51.630273+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":668940,"num_objects":82,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12698,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2116}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12403","reported_seq":14805,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:54.979496+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:52:54.979496+0000","last_peered":"2024-09-16T04:52:54.979496+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:52:54.979496+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374028,"num_objects":46,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12401,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1157,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-933}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12280","reported_seq":12380,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:55.932733+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:55.932733+0000","last_peered":"2024-09-16T04:52:55.932733+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:55.932733+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218380,"num_objects":27,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12278,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":7562944,"num_objects":929,"num_object_clones":0,"num_object_copies":3716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200558,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8242,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13508608,"data_stored":13508608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705524,"kb_used_data":6028,"kb_used_omap":4,"kb_used_meta":699451,"kb_avail":93666316,"statfs":{"total":96636764160,"available":95914307584,"internally_reserved":0,"allocated":6172672,"data_stored":5974163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":716238543},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,1,0,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746652,"kb_used_data":3272,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93625188,"statfs":{"total":96636764160,"available":95872192512,"internally_reserved":0,"allocated":3350528,"data_stored":3152019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,0,1,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581388,"kb_used_data":4088,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93790452,"statfs":{"total":96636764160,"available":96041422848,"internally_reserved":0,"allocated":4186112,"data_stored":3987603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591130319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724332,"kb_used_data":2704,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93647508,"statfs":{"total":96636764160,"available":95895048192,"internally_reserved":0,"allocated":2768896,"data_stored":2570387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1728512,"data_stored":1728512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3776512,"data_stored":3776512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2703360,"data_stored":2703360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5300224,"data_stored":5300224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:01.540 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:01.860 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:01.861 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:01.875 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-09-16T04:53:00.807718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1861288,"num_objects":232,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201255,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8600,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2761620,"kb_used_data":5956,"kb_used_omap":14,"kb_used_meta":2755569,"kb_avail":374725740,"statfs":{"total":386547056640,"available":383719157760,"internally_reserved":0,"allocated":6098944,"data_stored":5304908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2821703217},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,6,12,8,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18153496,"num_objects":-2217,"num_object_clones":0,"num_object_copies":-8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2217,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003729"},"pg_stats":[{"pgid":"1.f","version":"37'12638","reported_seq":12760,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.854825+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:56.854825+0000","last_peered":"2024-09-16T04:52:56.854825+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:56.854825+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12638,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2170}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12647","reported_seq":12741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.118612+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:56.118612+0000","last_peered":"2024-09-16T04:52:56.118612+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:56.118612+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218380,"num_objects":27,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12645,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12774,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.304644+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:58.304644+0000","last_peered":"2024-09-16T04:52:58.304644+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:58.304644+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17218,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:58.390829+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:58.390829+0000","last_peered":"2024-09-16T04:52:58.390829+0000","last_clean":"2024-09-16T04:52:58.390829+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:58.390829+0000","last_undegraded":"2024-09-16T04:52:58.390829+0000","last_fullsized":"2024-09-16T04:52:58.390829+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:02.260546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12717","reported_seq":12855,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.715490+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:56.715490+0000","last_peered":"2024-09-16T04:52:56.715490+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:56.715490+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193804,"num_objects":24,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12715,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2224}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12298","reported_seq":12414,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.742396+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:56.742396+0000","last_peered":"2024-09-16T04:52:56.742396+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:56.742396+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114688,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12298,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2164}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12564","reported_seq":12667,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:55.907009+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:55.907009+0000","last_peered":"2024-09-16T04:52:55.907009+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:55.907009+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292108,"num_objects":36,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12562,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2177}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17351,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:58.321583+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:58.321583+0000","last_peered":"2024-09-16T04:52:58.321583+0000","last_clean":"2024-09-16T04:52:58.321583+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:58.321583+0000","last_undegraded":"2024-09-16T04:52:58.321583+0000","last_fullsized":"2024-09-16T04:52:58.321583+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:02.260522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12736","reported_seq":17499,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:56.744394+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:56.744394+0000","last_peered":"2024-09-16T04:52:56.744394+0000","last_clean":"2024-09-16T04:52:56.744394+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:56.744394+0000","last_undegraded":"2024-09-16T04:52:56.744394+0000","last_fullsized":"2024-09-16T04:52:56.744394+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:58.054142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158232,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12733,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12869,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.405966+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:58.405966+0000","last_peered":"2024-09-16T04:52:58.405966+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:58.405966+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12464","reported_seq":12602,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.695130+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:56.695130+0000","last_peered":"2024-09-16T04:52:56.695130+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:56.695130+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216978,"num_objects":27,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12320","reported_seq":12418,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.840929+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:56.840929+0000","last_peered":"2024-09-16T04:52:56.840929+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:56.840929+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88710,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12319,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2094}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.412387+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:58.412387+0000","last_peered":"2024-09-16T04:52:58.412387+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:58.412387+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12762","reported_seq":12900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.208186+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:56.208186+0000","last_peered":"2024-09-16T04:52:56.208186+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:56.208186+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161036,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12760,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2178}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":15565,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:00.003661+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:53:00.003661+0000","last_peered":"2024-09-16T04:53:00.003661+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:53:00.003661+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1515,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-619}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12280","reported_seq":12380,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:55.932733+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:55.932733+0000","last_peered":"2024-09-16T04:52:55.932733+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:55.932733+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218380,"num_objects":27,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12278,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":1861288,"num_objects":232,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201255,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8600,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3493888,"data_stored":3493888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723764,"kb_used_data":2080,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93648076,"statfs":{"total":96636764160,"available":95895629824,"internally_reserved":0,"allocated":2129920,"data_stored":1931411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,5,0,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731204,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640636,"statfs":{"total":96636764160,"available":95888011264,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582320,"kb_used_data":864,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93789520,"statfs":{"total":96636764160,"available":96040468480,"internally_reserved":0,"allocated":884736,"data_stored":686227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595390159},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,7,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724332,"kb_used_data":2704,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93647508,"statfs":{"total":96636764160,"available":95895048192,"internally_reserved":0,"allocated":2768896,"data_stored":2570387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1728512,"data_stored":1728512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":573440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1187840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:01.876 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:02.177 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:02.178 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:02.190 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-09-16T04:53:00.807718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1861288,"num_objects":232,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201255,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8600,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2761620,"kb_used_data":5956,"kb_used_omap":14,"kb_used_meta":2755569,"kb_avail":374725740,"statfs":{"total":386547056640,"available":383719157760,"internally_reserved":0,"allocated":6098944,"data_stored":5304908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2821703217},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,6,12,8,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18153496,"num_objects":-2217,"num_object_clones":0,"num_object_copies":-8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2217,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003729"},"pg_stats":[{"pgid":"1.f","version":"37'12638","reported_seq":12760,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.854825+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:56.854825+0000","last_peered":"2024-09-16T04:52:56.854825+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:56.854825+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12638,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2170}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12647","reported_seq":12741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.118612+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:56.118612+0000","last_peered":"2024-09-16T04:52:56.118612+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:56.118612+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218380,"num_objects":27,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12645,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12774,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.304644+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:58.304644+0000","last_peered":"2024-09-16T04:52:58.304644+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:58.304644+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17218,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:58.390829+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:58.390829+0000","last_peered":"2024-09-16T04:52:58.390829+0000","last_clean":"2024-09-16T04:52:58.390829+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:58.390829+0000","last_undegraded":"2024-09-16T04:52:58.390829+0000","last_fullsized":"2024-09-16T04:52:58.390829+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:02.260546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12717","reported_seq":12855,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.715490+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:56.715490+0000","last_peered":"2024-09-16T04:52:56.715490+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:56.715490+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193804,"num_objects":24,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12715,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2224}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12298","reported_seq":12414,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.742396+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:56.742396+0000","last_peered":"2024-09-16T04:52:56.742396+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:56.742396+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114688,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12298,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2164}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12564","reported_seq":12667,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:55.907009+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:55.907009+0000","last_peered":"2024-09-16T04:52:55.907009+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:55.907009+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292108,"num_objects":36,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12562,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2177}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17351,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:58.321583+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:58.321583+0000","last_peered":"2024-09-16T04:52:58.321583+0000","last_clean":"2024-09-16T04:52:58.321583+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:58.321583+0000","last_undegraded":"2024-09-16T04:52:58.321583+0000","last_fullsized":"2024-09-16T04:52:58.321583+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:02.260522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12736","reported_seq":17499,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:56.744394+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:56.744394+0000","last_peered":"2024-09-16T04:52:56.744394+0000","last_clean":"2024-09-16T04:52:56.744394+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:56.744394+0000","last_undegraded":"2024-09-16T04:52:56.744394+0000","last_fullsized":"2024-09-16T04:52:56.744394+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:52:58.054142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158232,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12733,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12869,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.405966+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:58.405966+0000","last_peered":"2024-09-16T04:52:58.405966+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:58.405966+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12464","reported_seq":12602,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.695130+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:56.695130+0000","last_peered":"2024-09-16T04:52:56.695130+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:56.695130+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216978,"num_objects":27,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12320","reported_seq":12418,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.840929+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:56.840929+0000","last_peered":"2024-09-16T04:52:56.840929+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:56.840929+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88710,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12319,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2094}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.412387+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:58.412387+0000","last_peered":"2024-09-16T04:52:58.412387+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:58.412387+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12762","reported_seq":12900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:56.208186+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:56.208186+0000","last_peered":"2024-09-16T04:52:56.208186+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:56.208186+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161036,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12760,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2178}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":15565,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:00.003661+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:53:00.003661+0000","last_peered":"2024-09-16T04:53:00.003661+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:53:00.003661+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1515,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-619}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12280","reported_seq":12380,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:55.932733+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:55.932733+0000","last_peered":"2024-09-16T04:52:55.932733+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:55.932733+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"28'2200","ondisk_log_start":"28'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218380,"num_objects":27,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12278,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":1861288,"num_objects":232,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201255,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8600,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3493888,"data_stored":3493888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723764,"kb_used_data":2080,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93648076,"statfs":{"total":96636764160,"available":95895629824,"internally_reserved":0,"allocated":2129920,"data_stored":1931411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,5,0,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731204,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640636,"statfs":{"total":96636764160,"available":95888011264,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582320,"kb_used_data":864,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93789520,"statfs":{"total":96636764160,"available":96040468480,"internally_reserved":0,"allocated":884736,"data_stored":686227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595390159},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,7,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724332,"kb_used_data":2704,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93647508,"statfs":{"total":96636764160,"available":95895048192,"internally_reserved":0,"allocated":2768896,"data_stored":2570387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1728512,"data_stored":1728512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":573440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1187840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:02.191 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:53:02.191 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12638", 'reported_seq': 12760, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:56.854825+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:52:56.854825+0000', 'last_peered': '2024-09-16T04:52:56.854825+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:52:56.854825+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 131072, 'num_objects': 16, 'num_object_clones': 0, 'num_object_copies': 64, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 16, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12638, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2170}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:02.192 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:53:02.192 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12647", 'reported_seq': 12741, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:56.118612+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:52:56.118612+0000', 'last_peered': '2024-09-16T04:52:56.118612+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:52:56.118612+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 218380, 'num_objects': 27, 'num_object_clones': 0, 'num_object_copies': 108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 27, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12645, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2176}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:02.192 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:53:02.192 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12774, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:58.304644+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:52:58.304644+0000', 'last_peered': '2024-09-16T04:52:58.304644+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:52:58.304644+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:02.192 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:53:02.192 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12717", 'reported_seq': 12855, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:56.715490+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:52:56.715490+0000', 'last_peered': '2024-09-16T04:52:56.715490+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:52:56.715490+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 193804, 'num_objects': 24, 'num_object_clones': 0, 'num_object_copies': 96, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 24, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12715, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2224}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:02.192 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:53:02.192 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12298", 'reported_seq': 12414, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:56.742396+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:52:56.742396+0000', 'last_peered': '2024-09-16T04:52:56.742396+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:52:56.742396+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 114688, 'num_objects': 14, 'num_object_clones': 0, 'num_object_copies': 56, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 14, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12298, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2164}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:02.192 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:53:02.192 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12564", 'reported_seq': 12667, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:55.907009+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:52:55.907009+0000', 'last_peered': '2024-09-16T04:52:55.907009+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:52:55.907009+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 292108, 'num_objects': 36, 'num_object_clones': 0, 'num_object_copies': 144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 36, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12562, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2177}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:02.193 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:53:02.193 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12869, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:58.405966+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:52:58.405966+0000', 'last_peered': '2024-09-16T04:52:58.405966+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:52:58.405966+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:02.193 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:53:02.193 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12464", 'reported_seq': 12602, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:56.695130+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:52:56.695130+0000', 'last_peered': '2024-09-16T04:52:56.695130+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:52:56.695130+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 216978, 'num_objects': 27, 'num_object_clones': 0, 'num_object_copies': 108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 27, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12461, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2128}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:02.193 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:53:02.193 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12320", 'reported_seq': 12418, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:56.840929+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:52:56.840929+0000', 'last_peered': '2024-09-16T04:52:56.840929+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:52:56.840929+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 88710, 'num_objects': 11, 'num_object_clones': 0, 'num_object_copies': 44, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 11, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12319, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2094}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:02.193 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:53:02.193 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12532", 'reported_seq': 12626, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:58.412387+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:52:58.412387+0000', 'last_peered': '2024-09-16T04:52:58.412387+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:52:58.412387+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12531, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:02.193 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:53:02.193 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12762", 'reported_seq': 12900, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:56.208186+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:52:56.208186+0000', 'last_peered': '2024-09-16T04:52:56.208186+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:52:56.208186+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 161036, 'num_objects': 20, 'num_object_clones': 0, 'num_object_copies': 80, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 20, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12760, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2178}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:02.194 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-16T04:53:02.194 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12447", 'reported_seq': 15565, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:00.003661+0000', 'last_change': '2024-09-16T04:51:03.096842+0000', 'last_active': '2024-09-16T04:53:00.003661+0000', 'last_peered': '2024-09-16T04:53:00.003661+0000', 'last_clean': '2024-09-16T04:40:23.326429+0000', 'last_became_active': '2024-09-16T04:41:27.771262+0000', 'last_became_peered': '2024-09-16T04:41:27.771262+0000', 'last_unstale': '2024-09-16T04:53:00.003661+0000', 'last_undegraded': '2024-09-16T04:41:27.539214+0000', 'last_fullsized': '2024-09-16T04:41:27.487122+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 621, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49789, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1515, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -619}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:02.194 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:53:02.194 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12280", 'reported_seq': 12380, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:55.932733+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:52:55.932733+0000', 'last_peered': '2024-09-16T04:52:55.932733+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:52:55.932733+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "28'2200", 'ondisk_log_start': "28'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 218380, 'num_objects': 27, 'num_object_clones': 0, 'num_object_copies': 108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 27, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12278, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:02.194 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:53:02.596 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:02.597 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26982/928 objects degraded (2907.543%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":794,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":232,"data_bytes":1861288,"bytes_used":2827898880,"bytes_avail":383719157760,"bytes_total":386547056640,"degraded_objects":26982,"degraded_total":928,"degraded_ratio":29.075431034482758,"recovering_objects_per_sec":40,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":486,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":184},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:53:02.613 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:53:02.613 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:02.930 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:02.931 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:02.949 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2024-09-16T04:53:02.808161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8600,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2757436,"kb_used_data":1788,"kb_used_omap":14,"kb_used_meta":2755569,"kb_avail":374729924,"statfs":{"total":386547056640,"available":383723442176,"internally_reserved":0,"allocated":1830912,"data_stored":1036876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2821703217},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,12,11,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16711704,"num_objects":-2041,"num_object_clones":0,"num_object_copies":-8164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2041,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003717"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12776,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.385498+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:58.385498+0000","last_peered":"2024-09-16T04:52:58.385498+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:58.385498+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12766,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.449312+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:58.449312+0000","last_peered":"2024-09-16T04:52:58.449312+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:58.449312+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12774,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.304644+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:58.304644+0000","last_peered":"2024-09-16T04:52:58.304644+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:58.304644+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17218,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:58.390829+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:52:58.390829+0000","last_peered":"2024-09-16T04:52:58.390829+0000","last_clean":"2024-09-16T04:52:58.390829+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:52:58.390829+0000","last_undegraded":"2024-09-16T04:52:58.390829+0000","last_fullsized":"2024-09-16T04:52:58.390829+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:02.260546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12878,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.327851+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:58.327851+0000","last_peered":"2024-09-16T04:52:58.327851+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:58.327851+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":12428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.232120+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:58.232120+0000","last_peered":"2024-09-16T04:52:58.232120+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:58.232120+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":12701,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.411158+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:58.411158+0000","last_peered":"2024-09-16T04:52:58.411158+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:58.411158+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2211}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17351,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:58.321583+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:52:58.321583+0000","last_peered":"2024-09-16T04:52:58.321583+0000","last_clean":"2024-09-16T04:52:58.321583+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:52:58.321583+0000","last_undegraded":"2024-09-16T04:52:58.321583+0000","last_fullsized":"2024-09-16T04:52:58.321583+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:02.260522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17515,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:58.244148+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:58.244148+0000","last_peered":"2024-09-16T04:52:58.244148+0000","last_clean":"2024-09-16T04:52:58.244148+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:58.244148+0000","last_undegraded":"2024-09-16T04:52:58.244148+0000","last_fullsized":"2024-09-16T04:52:58.244148+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12869,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.405966+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:58.405966+0000","last_peered":"2024-09-16T04:52:58.405966+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:58.405966+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.352619+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:58.352619+0000","last_peered":"2024-09-16T04:52:58.352619+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:58.352619+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.186515+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:58.186515+0000","last_peered":"2024-09-16T04:52:58.186515+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:58.186515+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.412387+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:58.412387+0000","last_peered":"2024-09-16T04:52:58.412387+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:58.412387+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12918,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.290930+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:58.290930+0000","last_peered":"2024-09-16T04:52:58.290930+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:58.290930+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":15565,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:00.003661+0000","last_change":"2024-09-16T04:51:03.096842+0000","last_active":"2024-09-16T04:53:00.003661+0000","last_peered":"2024-09-16T04:53:00.003661+0000","last_clean":"2024-09-16T04:40:23.326429+0000","last_became_active":"2024-09-16T04:41:27.771262+0000","last_became_peered":"2024-09-16T04:41:27.771262+0000","last_unstale":"2024-09-16T04:53:00.003661+0000","last_undegraded":"2024-09-16T04:41:27.539214+0000","last_fullsized":"2024-09-16T04:41:27.487122+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1515,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-619}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12405,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.360030+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:58.360030+0000","last_peered":"2024-09-16T04:52:58.360030+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:58.360030+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8600,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":585728,"data_stored":585728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721988,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649852,"statfs":{"total":96636764160,"available":95897448448,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731204,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640636,"statfs":{"total":96636764160,"available":95888011264,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582320,"kb_used_data":864,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93789520,"statfs":{"total":96636764160,"available":96040468480,"internally_reserved":0,"allocated":884736,"data_stored":686227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595390159},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,7,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721924,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649916,"statfs":{"total":96636764160,"available":95897513984,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,5,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":573440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:05.951 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:06.247 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:06.248 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:06.261 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-16T04:53:04.808569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8275,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749132,"kb_used_data":1796,"kb_used_omap":14,"kb_used_meta":2747249,"kb_avail":374738228,"statfs":{"total":386547056640,"available":383731945472,"internally_reserved":0,"allocated":1839104,"data_stored":1035687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2813183537},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,5,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7372824,"num_objects":-901,"num_object_clones":0,"num_object_copies":-3604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":901,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001969"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12776,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.385498+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:58.385498+0000","last_peered":"2024-09-16T04:52:58.385498+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:58.385498+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12766,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.449312+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:58.449312+0000","last_peered":"2024-09-16T04:52:58.449312+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:58.449312+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12774,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.304644+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:58.304644+0000","last_peered":"2024-09-16T04:52:58.304644+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:58.304644+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17222,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.945924+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:03.945924+0000","last_peered":"2024-09-16T04:53:03.945924+0000","last_clean":"2024-09-16T04:53:03.945924+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:03.945924+0000","last_undegraded":"2024-09-16T04:53:03.945924+0000","last_fullsized":"2024-09-16T04:53:03.945924+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12878,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.327851+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:58.327851+0000","last_peered":"2024-09-16T04:52:58.327851+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:58.327851+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":12428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.232120+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:58.232120+0000","last_peered":"2024-09-16T04:52:58.232120+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:58.232120+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":12701,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.411158+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:58.411158+0000","last_peered":"2024-09-16T04:52:58.411158+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:58.411158+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2211}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.946220+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:03.946220+0000","last_peered":"2024-09-16T04:53:03.946220+0000","last_clean":"2024-09-16T04:53:03.946220+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:03.946220+0000","last_undegraded":"2024-09-16T04:53:03.946220+0000","last_fullsized":"2024-09-16T04:53:03.946220+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17515,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:58.244148+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:58.244148+0000","last_peered":"2024-09-16T04:52:58.244148+0000","last_clean":"2024-09-16T04:52:58.244148+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:58.244148+0000","last_undegraded":"2024-09-16T04:52:58.244148+0000","last_fullsized":"2024-09-16T04:52:58.244148+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12869,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.405966+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:58.405966+0000","last_peered":"2024-09-16T04:52:58.405966+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:58.405966+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.352619+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:58.352619+0000","last_peered":"2024-09-16T04:52:58.352619+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:58.352619+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.186515+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:58.186515+0000","last_peered":"2024-09-16T04:52:58.186515+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:58.186515+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.412387+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:58.412387+0000","last_peered":"2024-09-16T04:52:58.412387+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:58.412387+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12918,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.290930+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:58.290930+0000","last_peered":"2024-09-16T04:52:58.290930+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:58.290930+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14929,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:04.014146+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:04.014146+0000","last_peered":"2024-09-16T04:53:04.014146+0000","last_clean":"2024-09-16T04:53:04.014146+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:04.014146+0000","last_undegraded":"2024-09-16T04:53:04.014146+0000","last_fullsized":"2024-09-16T04:53:04.014146+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:09.394708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12405,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.360030+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:58.360030+0000","last_peered":"2024-09-16T04:52:58.360030+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:58.360030+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8275,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721988,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649852,"statfs":{"total":96636764160,"available":95897448448,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731204,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640636,"statfs":{"total":96636764160,"available":95888011264,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574016,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93797824,"statfs":{"total":96636764160,"available":96048971776,"internally_reserved":0,"allocated":892928,"data_stored":685038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586870479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721924,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649916,"statfs":{"total":96636764160,"available":95897513984,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,5,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:06.262 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:06.566 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:06.567 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:06.579 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-16T04:53:04.808569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8275,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749132,"kb_used_data":1796,"kb_used_omap":14,"kb_used_meta":2747249,"kb_avail":374738228,"statfs":{"total":386547056640,"available":383731945472,"internally_reserved":0,"allocated":1839104,"data_stored":1035687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2813183537},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,5,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7372824,"num_objects":-901,"num_object_clones":0,"num_object_copies":-3604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":901,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001969"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12776,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.385498+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:58.385498+0000","last_peered":"2024-09-16T04:52:58.385498+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:58.385498+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12766,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.449312+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:52:58.449312+0000","last_peered":"2024-09-16T04:52:58.449312+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:52:58.449312+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12774,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.304644+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:52:58.304644+0000","last_peered":"2024-09-16T04:52:58.304644+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:52:58.304644+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17222,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.945924+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:03.945924+0000","last_peered":"2024-09-16T04:53:03.945924+0000","last_clean":"2024-09-16T04:53:03.945924+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:03.945924+0000","last_undegraded":"2024-09-16T04:53:03.945924+0000","last_fullsized":"2024-09-16T04:53:03.945924+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12878,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.327851+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:58.327851+0000","last_peered":"2024-09-16T04:52:58.327851+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:58.327851+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":12428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.232120+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:58.232120+0000","last_peered":"2024-09-16T04:52:58.232120+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:58.232120+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":12701,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.411158+0000","last_change":"2024-09-16T04:41:27.777636+0000","last_active":"2024-09-16T04:52:58.411158+0000","last_peered":"2024-09-16T04:52:58.411158+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:52:58.411158+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2211}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.946220+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:03.946220+0000","last_peered":"2024-09-16T04:53:03.946220+0000","last_clean":"2024-09-16T04:53:03.946220+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:03.946220+0000","last_undegraded":"2024-09-16T04:53:03.946220+0000","last_fullsized":"2024-09-16T04:53:03.946220+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17515,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:58.244148+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:58.244148+0000","last_peered":"2024-09-16T04:52:58.244148+0000","last_clean":"2024-09-16T04:52:58.244148+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:58.244148+0000","last_undegraded":"2024-09-16T04:52:58.244148+0000","last_fullsized":"2024-09-16T04:52:58.244148+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12869,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.405966+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:52:58.405966+0000","last_peered":"2024-09-16T04:52:58.405966+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:52:58.405966+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.352619+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:58.352619+0000","last_peered":"2024-09-16T04:52:58.352619+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:58.352619+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.186515+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:58.186515+0000","last_peered":"2024-09-16T04:52:58.186515+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:58.186515+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.412387+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:52:58.412387+0000","last_peered":"2024-09-16T04:52:58.412387+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:52:58.412387+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12918,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.290930+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:58.290930+0000","last_peered":"2024-09-16T04:52:58.290930+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:58.290930+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14929,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:04.014146+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:04.014146+0000","last_peered":"2024-09-16T04:53:04.014146+0000","last_clean":"2024-09-16T04:53:04.014146+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:04.014146+0000","last_undegraded":"2024-09-16T04:53:04.014146+0000","last_fullsized":"2024-09-16T04:53:04.014146+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:09.394708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12405,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.360030+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:52:58.360030+0000","last_peered":"2024-09-16T04:52:58.360030+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:52:58.360030+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8275,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721988,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649852,"statfs":{"total":96636764160,"available":95897448448,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731204,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640636,"statfs":{"total":96636764160,"available":95888011264,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574016,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93797824,"statfs":{"total":96636764160,"available":96048971776,"internally_reserved":0,"allocated":892928,"data_stored":685038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586870479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721924,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649916,"statfs":{"total":96636764160,"available":95897513984,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,5,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:06.581 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:53:06.581 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 12776, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:58.385498+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:52:58.385498+0000', 'last_peered': '2024-09-16T04:52:58.385498+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:52:58.385498+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:06.581 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:53:06.582 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12672", 'reported_seq': 12766, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:58.449312+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:52:58.449312+0000', 'last_peered': '2024-09-16T04:52:58.449312+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:52:58.449312+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12670, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:06.582 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:53:06.582 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12774, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:58.304644+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:52:58.304644+0000', 'last_peered': '2024-09-16T04:52:58.304644+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:52:58.304644+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:06.582 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:53:06.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12878, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:58.327851+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:52:58.327851+0000', 'last_peered': '2024-09-16T04:52:58.327851+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:52:58.327851+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:06.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:53:06.583 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12312", 'reported_seq': 12428, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:58.232120+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:52:58.232120+0000', 'last_peered': '2024-09-16T04:52:58.232120+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:52:58.232120+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12312, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2178}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:06.583 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:53:06.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12598", 'reported_seq': 12701, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:58.411158+0000', 'last_change': '2024-09-16T04:41:27.777636+0000', 'last_active': '2024-09-16T04:52:58.411158+0000', 'last_peered': '2024-09-16T04:52:58.411158+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:52:58.411158+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2215, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12596, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2211}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:06.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:53:06.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12869, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:58.405966+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:52:58.405966+0000', 'last_peered': '2024-09-16T04:52:58.405966+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:52:58.405966+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:06.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:53:06.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12626, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:58.352619+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:52:58.352619+0000', 'last_peered': '2024-09-16T04:52:58.352619+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:52:58.352619+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:06.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:53:06.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12428, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:58.186515+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:52:58.186515+0000', 'last_peered': '2024-09-16T04:52:58.186515+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:52:58.186515+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:06.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:53:06.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12532", 'reported_seq': 12626, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:58.412387+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:52:58.412387+0000', 'last_peered': '2024-09-16T04:52:58.412387+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:52:58.412387+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12531, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:06.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:53:06.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12918, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:58.290930+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:52:58.290930+0000', 'last_peered': '2024-09-16T04:52:58.290930+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:52:58.290930+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:06.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:53:06.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12305", 'reported_seq': 12405, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:52:58.360030+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:52:58.360030+0000', 'last_peered': '2024-09-16T04:52:58.360030+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:52:58.360030+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12303, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:06.587 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:53:06.972 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:06.972 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25822/112 objects degraded (23055.357%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":799,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2815160320,"bytes_avail":383731896320,"bytes_total":386547056640,"degraded_objects":25822,"degraded_total":112,"degraded_ratio":230.55357142857142,"recovering_objects_per_sec":71,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":572,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":58},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:53:06.986 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:53:06.986 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:07.283 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:07.284 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:07.296 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2024-09-16T04:53:06.808937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8814,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749180,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":2747249,"kb_avail":374738180,"statfs":{"total":386547056640,"available":383731896320,"internally_reserved":0,"allocated":1871872,"data_stored":1049693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2813183537},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3801112,"num_objects":-465,"num_object_clones":0,"num_object_copies":-1860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":465,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":572,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001614"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12776,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.385498+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:52:58.385498+0000","last_peered":"2024-09-16T04:52:58.385498+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:52:58.385498+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12770,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.948572+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:53:03.948572+0000","last_peered":"2024-09-16T04:53:03.948572+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:03.948572+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12778,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.945253+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:03.945253+0000","last_peered":"2024-09-16T04:53:03.945253+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:03.945253+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17222,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.945924+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:03.945924+0000","last_peered":"2024-09-16T04:53:03.945924+0000","last_clean":"2024-09-16T04:53:03.945924+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:03.945924+0000","last_undegraded":"2024-09-16T04:53:03.945924+0000","last_fullsized":"2024-09-16T04:53:03.945924+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12878,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.327851+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:52:58.327851+0000","last_peered":"2024-09-16T04:52:58.327851+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:52:58.327851+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":12428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.232120+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:52:58.232120+0000","last_peered":"2024-09-16T04:52:58.232120+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:52:58.232120+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":13785,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:06.226924+0000","last_change":"2024-09-16T04:53:02.647802+0000","last_active":"2024-09-16T04:53:06.226924+0000","last_peered":"2024-09-16T04:53:06.226924+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:53:06.226924+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1676,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1674}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.946220+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:03.946220+0000","last_peered":"2024-09-16T04:53:03.946220+0000","last_clean":"2024-09-16T04:53:03.946220+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:03.946220+0000","last_undegraded":"2024-09-16T04:53:03.946220+0000","last_fullsized":"2024-09-16T04:53:03.946220+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17515,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:52:58.244148+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:52:58.244148+0000","last_peered":"2024-09-16T04:52:58.244148+0000","last_clean":"2024-09-16T04:52:58.244148+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:52:58.244148+0000","last_undegraded":"2024-09-16T04:52:58.244148+0000","last_fullsized":"2024-09-16T04:52:58.244148+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12873,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.945290+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:03.945290+0000","last_peered":"2024-09-16T04:53:03.945290+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:03.945290+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.352619+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:52:58.352619+0000","last_peered":"2024-09-16T04:52:58.352619+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:52:58.352619+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.186515+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:52:58.186515+0000","last_peered":"2024-09-16T04:52:58.186515+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:52:58.186515+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943578+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:03.943578+0000","last_peered":"2024-09-16T04:53:03.943578+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:03.943578+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12918,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:52:58.290930+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:52:58.290930+0000","last_peered":"2024-09-16T04:52:58.290930+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:52:58.290930+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14929,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:04.014146+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:04.014146+0000","last_peered":"2024-09-16T04:53:04.014146+0000","last_clean":"2024-09-16T04:53:04.014146+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:04.014146+0000","last_undegraded":"2024-09-16T04:53:04.014146+0000","last_fullsized":"2024-09-16T04:53:04.014146+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:09.394708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12409,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.948535+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:03.948535+0000","last_peered":"2024-09-16T04:53:03.948535+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:03.948535+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8814,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721988,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649852,"statfs":{"total":96636764160,"available":95897448448,"internally_reserved":0,"allocated":315392,"data_stored":116883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731228,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640612,"statfs":{"total":96636764160,"available":95887986688,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574016,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93797824,"statfs":{"total":96636764160,"available":96048971776,"internally_reserved":0,"allocated":892928,"data_stored":685038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586870479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721948,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649892,"statfs":{"total":96636764160,"available":95897489408,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:10.298 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:10.590 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:10.591 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:10.604 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-16T04:53:08.809289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8814,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749200,"kb_used_data":1844,"kb_used_omap":14,"kb_used_meta":2747249,"kb_avail":374738160,"statfs":{"total":386547056640,"available":383731875840,"internally_reserved":0,"allocated":1888256,"data_stored":1056696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2813183537},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1671168,"num_objects":-204,"num_object_clones":0,"num_object_copies":-816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":204,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001571"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12780,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942729+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:03.942729+0000","last_peered":"2024-09-16T04:53:03.942729+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:03.942729+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12770,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.948572+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:53:03.948572+0000","last_peered":"2024-09-16T04:53:03.948572+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:03.948572+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12778,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.945253+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:03.945253+0000","last_peered":"2024-09-16T04:53:03.945253+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:03.945253+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17222,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.945924+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:03.945924+0000","last_peered":"2024-09-16T04:53:03.945924+0000","last_clean":"2024-09-16T04:53:03.945924+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:03.945924+0000","last_undegraded":"2024-09-16T04:53:03.945924+0000","last_fullsized":"2024-09-16T04:53:03.945924+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12882,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943377+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:03.943377+0000","last_peered":"2024-09-16T04:53:03.943377+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:03.943377+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":12432,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942781+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:53:03.942781+0000","last_peered":"2024-09-16T04:53:03.942781+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:03.942781+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":13785,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:06.226924+0000","last_change":"2024-09-16T04:53:02.647802+0000","last_active":"2024-09-16T04:53:06.226924+0000","last_peered":"2024-09-16T04:53:06.226924+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:53:06.226924+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1676,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1674}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.946220+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:03.946220+0000","last_peered":"2024-09-16T04:53:03.946220+0000","last_clean":"2024-09-16T04:53:03.946220+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:03.946220+0000","last_undegraded":"2024-09-16T04:53:03.946220+0000","last_fullsized":"2024-09-16T04:53:03.946220+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17519,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.942739+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:03.942739+0000","last_peered":"2024-09-16T04:53:03.942739+0000","last_clean":"2024-09-16T04:53:03.942739+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:03.942739+0000","last_undegraded":"2024-09-16T04:53:03.942739+0000","last_fullsized":"2024-09-16T04:53:03.942739+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:04.045489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12873,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.945290+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:03.945290+0000","last_peered":"2024-09-16T04:53:03.945290+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:03.945290+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942709+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:03.942709+0000","last_peered":"2024-09-16T04:53:03.942709+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:03.942709+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12432,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942792+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:03.942792+0000","last_peered":"2024-09-16T04:53:03.942792+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:03.942792+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943578+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:03.943578+0000","last_peered":"2024-09-16T04:53:03.943578+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:03.943578+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12922,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943981+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:03.943981+0000","last_peered":"2024-09-16T04:53:03.943981+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:03.943981+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14929,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:04.014146+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:04.014146+0000","last_peered":"2024-09-16T04:53:04.014146+0000","last_clean":"2024-09-16T04:53:04.014146+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:04.014146+0000","last_undegraded":"2024-09-16T04:53:04.014146+0000","last_fullsized":"2024-09-16T04:53:04.014146+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:09.394708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12409,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.948535+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:03.948535+0000","last_peered":"2024-09-16T04:53:03.948535+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:03.948535+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8814,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722008,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649832,"statfs":{"total":96636764160,"available":95897427968,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731228,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640612,"statfs":{"total":96636764160,"available":95887986688,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574016,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93797824,"statfs":{"total":96636764160,"available":96048971776,"internally_reserved":0,"allocated":892928,"data_stored":685038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586870479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721948,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649892,"statfs":{"total":96636764160,"available":95897489408,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:10.606 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:10.908 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:10.909 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:10.923 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-09-16T04:53:10.809642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8814,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2753352,"kb_used_data":1844,"kb_used_omap":14,"kb_used_meta":2751409,"kb_avail":374734008,"statfs":{"total":386547056640,"available":383727624192,"internally_reserved":0,"allocated":1888256,"data_stored":1056696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2817443377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001481"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12780,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942729+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:03.942729+0000","last_peered":"2024-09-16T04:53:03.942729+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:03.942729+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12770,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.948572+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:53:03.948572+0000","last_peered":"2024-09-16T04:53:03.948572+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:03.948572+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12778,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.945253+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:03.945253+0000","last_peered":"2024-09-16T04:53:03.945253+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:03.945253+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17223,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.945996+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:03.945996+0000","last_peered":"2024-09-16T04:53:03.945996+0000","last_clean":"2024-09-16T04:53:03.945996+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:03.945996+0000","last_undegraded":"2024-09-16T04:53:03.945996+0000","last_fullsized":"2024-09-16T04:53:03.945996+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:12.421461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12882,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943377+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:03.943377+0000","last_peered":"2024-09-16T04:53:03.943377+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:03.943377+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":12432,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942781+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:53:03.942781+0000","last_peered":"2024-09-16T04:53:03.942781+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:03.942781+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":13785,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:06.226924+0000","last_change":"2024-09-16T04:53:02.647802+0000","last_active":"2024-09-16T04:53:06.226924+0000","last_peered":"2024-09-16T04:53:06.226924+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:53:06.226924+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1676,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1674}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.946220+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:03.946220+0000","last_peered":"2024-09-16T04:53:03.946220+0000","last_clean":"2024-09-16T04:53:03.946220+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:03.946220+0000","last_undegraded":"2024-09-16T04:53:03.946220+0000","last_fullsized":"2024-09-16T04:53:03.946220+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17519,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.942739+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:03.942739+0000","last_peered":"2024-09-16T04:53:03.942739+0000","last_clean":"2024-09-16T04:53:03.942739+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:03.942739+0000","last_undegraded":"2024-09-16T04:53:03.942739+0000","last_fullsized":"2024-09-16T04:53:03.942739+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:04.045489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12873,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.945290+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:03.945290+0000","last_peered":"2024-09-16T04:53:03.945290+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:03.945290+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942709+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:03.942709+0000","last_peered":"2024-09-16T04:53:03.942709+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:03.942709+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12432,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942792+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:03.942792+0000","last_peered":"2024-09-16T04:53:03.942792+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:03.942792+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943578+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:03.943578+0000","last_peered":"2024-09-16T04:53:03.943578+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:03.943578+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12922,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943981+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:03.943981+0000","last_peered":"2024-09-16T04:53:03.943981+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:03.943981+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14929,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:04.014146+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:04.014146+0000","last_peered":"2024-09-16T04:53:04.014146+0000","last_clean":"2024-09-16T04:53:04.014146+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:04.014146+0000","last_undegraded":"2024-09-16T04:53:04.014146+0000","last_fullsized":"2024-09-16T04:53:04.014146+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:09.394708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12409,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.948535+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:03.948535+0000","last_peered":"2024-09-16T04:53:03.948535+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:03.948535+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8814,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722008,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649832,"statfs":{"total":96636764160,"available":95897427968,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731228,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640612,"statfs":{"total":96636764160,"available":95887986688,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578168,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93793672,"statfs":{"total":96636764160,"available":96044720128,"internally_reserved":0,"allocated":892928,"data_stored":685038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591130319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721948,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649892,"statfs":{"total":96636764160,"available":95897489408,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:10.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:53:10.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 12780, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.942729+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:53:03.942729+0000', 'last_peered': '2024-09-16T04:53:03.942729+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:53:03.942729+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:10.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:53:10.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12672", 'reported_seq': 12770, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.948572+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:53:03.948572+0000', 'last_peered': '2024-09-16T04:53:03.948572+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:53:03.948572+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12670, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:10.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:53:10.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12778, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.945253+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:53:03.945253+0000', 'last_peered': '2024-09-16T04:53:03.945253+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:53:03.945253+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:10.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:53:10.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12882, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.943377+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:53:03.943377+0000', 'last_peered': '2024-09-16T04:53:03.943377+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:53:03.943377+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:10.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:53:10.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12312", 'reported_seq': 12432, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.942781+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:53:03.942781+0000', 'last_peered': '2024-09-16T04:53:03.942781+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:53:03.942781+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12312, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2178}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:10.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:53:10.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12598", 'reported_seq': 13785, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:06.226924+0000', 'last_change': '2024-09-16T04:53:02.647802+0000', 'last_active': '2024-09-16T04:53:06.226924+0000', 'last_peered': '2024-09-16T04:53:06.226924+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:53:06.226924+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1676, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12596, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 554, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1674}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:10.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:53:10.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12873, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.945290+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:53:03.945290+0000', 'last_peered': '2024-09-16T04:53:03.945290+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:53:03.945290+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:10.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:53:10.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12630, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.942709+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:53:03.942709+0000', 'last_peered': '2024-09-16T04:53:03.942709+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:53:03.942709+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:10.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:53:10.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12432, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.942792+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:53:03.942792+0000', 'last_peered': '2024-09-16T04:53:03.942792+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:53:03.942792+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:10.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:53:10.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12532", 'reported_seq': 12630, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.943578+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:53:03.943578+0000', 'last_peered': '2024-09-16T04:53:03.943578+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:53:03.943578+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12531, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:10.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:53:10.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12922, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.943981+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:53:03.943981+0000', 'last_peered': '2024-09-16T04:53:03.943981+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:53:03.943981+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:10.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:53:10.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12305", 'reported_seq': 12409, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.948535+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:53:03.948535+0000', 'last_peered': '2024-09-16T04:53:03.948535+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:53:03.948535+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12303, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:10.926 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:53:11.312 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:11.312 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25822/112 objects degraded (23055.357%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":803,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2819432448,"bytes_avail":383727624192,"bytes_total":386547056640,"degraded_objects":25822,"degraded_total":112,"degraded_ratio":230.55357142857142,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":214,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:53:11.325 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:53:11.325 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:11.647 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:11.648 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:11.661 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-09-16T04:53:10.809642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8814,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2753352,"kb_used_data":1844,"kb_used_omap":14,"kb_used_meta":2751409,"kb_avail":374734008,"statfs":{"total":386547056640,"available":383727624192,"internally_reserved":0,"allocated":1888256,"data_stored":1056696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2817443377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001481"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12780,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942729+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:03.942729+0000","last_peered":"2024-09-16T04:53:03.942729+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:03.942729+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12770,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.948572+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:53:03.948572+0000","last_peered":"2024-09-16T04:53:03.948572+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:03.948572+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12778,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.945253+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:03.945253+0000","last_peered":"2024-09-16T04:53:03.945253+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:03.945253+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17223,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.945996+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:03.945996+0000","last_peered":"2024-09-16T04:53:03.945996+0000","last_clean":"2024-09-16T04:53:03.945996+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:03.945996+0000","last_undegraded":"2024-09-16T04:53:03.945996+0000","last_fullsized":"2024-09-16T04:53:03.945996+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:12.421461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12882,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943377+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:03.943377+0000","last_peered":"2024-09-16T04:53:03.943377+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:03.943377+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":12432,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942781+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:53:03.942781+0000","last_peered":"2024-09-16T04:53:03.942781+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:03.942781+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":13785,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:06.226924+0000","last_change":"2024-09-16T04:53:02.647802+0000","last_active":"2024-09-16T04:53:06.226924+0000","last_peered":"2024-09-16T04:53:06.226924+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:53:06.226924+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1676,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1674}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.946220+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:03.946220+0000","last_peered":"2024-09-16T04:53:03.946220+0000","last_clean":"2024-09-16T04:53:03.946220+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:03.946220+0000","last_undegraded":"2024-09-16T04:53:03.946220+0000","last_fullsized":"2024-09-16T04:53:03.946220+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17519,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.942739+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:03.942739+0000","last_peered":"2024-09-16T04:53:03.942739+0000","last_clean":"2024-09-16T04:53:03.942739+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:03.942739+0000","last_undegraded":"2024-09-16T04:53:03.942739+0000","last_fullsized":"2024-09-16T04:53:03.942739+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:04.045489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12873,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.945290+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:03.945290+0000","last_peered":"2024-09-16T04:53:03.945290+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:03.945290+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942709+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:03.942709+0000","last_peered":"2024-09-16T04:53:03.942709+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:03.942709+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12432,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942792+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:03.942792+0000","last_peered":"2024-09-16T04:53:03.942792+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:03.942792+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943578+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:03.943578+0000","last_peered":"2024-09-16T04:53:03.943578+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:03.943578+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12922,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943981+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:03.943981+0000","last_peered":"2024-09-16T04:53:03.943981+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:03.943981+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14929,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:04.014146+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:04.014146+0000","last_peered":"2024-09-16T04:53:04.014146+0000","last_clean":"2024-09-16T04:53:04.014146+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:04.014146+0000","last_undegraded":"2024-09-16T04:53:04.014146+0000","last_fullsized":"2024-09-16T04:53:04.014146+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:09.394708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12409,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.948535+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:03.948535+0000","last_peered":"2024-09-16T04:53:03.948535+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:03.948535+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8814,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722008,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649832,"statfs":{"total":96636764160,"available":95897427968,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731228,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640612,"statfs":{"total":96636764160,"available":95887986688,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578168,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93793672,"statfs":{"total":96636764160,"available":96044720128,"internally_reserved":0,"allocated":892928,"data_stored":685038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591130319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721948,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649892,"statfs":{"total":96636764160,"available":95897489408,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:14.663 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:14.982 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:14.982 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:14.996 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-16T04:53:14.971611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9667,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745032,"kb_used_data":1844,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742328,"statfs":{"total":386547056640,"available":383736143872,"internally_reserved":0,"allocated":1888256,"data_stored":1056696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2808923697},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1392,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.163041"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12780,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942729+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:03.942729+0000","last_peered":"2024-09-16T04:53:03.942729+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:03.942729+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12770,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.948572+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:53:03.948572+0000","last_peered":"2024-09-16T04:53:03.948572+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:03.948572+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12778,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.945253+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:03.945253+0000","last_peered":"2024-09-16T04:53:03.945253+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:03.945253+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17223,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.945996+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:03.945996+0000","last_peered":"2024-09-16T04:53:03.945996+0000","last_clean":"2024-09-16T04:53:03.945996+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:03.945996+0000","last_undegraded":"2024-09-16T04:53:03.945996+0000","last_fullsized":"2024-09-16T04:53:03.945996+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:12.421461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12882,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943377+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:03.943377+0000","last_peered":"2024-09-16T04:53:03.943377+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:03.943377+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":12432,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942781+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:53:03.942781+0000","last_peered":"2024-09-16T04:53:03.942781+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:03.942781+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":15491,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:11.231179+0000","last_change":"2024-09-16T04:53:02.647802+0000","last_active":"2024-09-16T04:53:11.231179+0000","last_peered":"2024-09-16T04:53:11.231179+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:53:11.231179+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1407,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-821}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.946220+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:03.946220+0000","last_peered":"2024-09-16T04:53:03.946220+0000","last_clean":"2024-09-16T04:53:03.946220+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:03.946220+0000","last_undegraded":"2024-09-16T04:53:03.946220+0000","last_fullsized":"2024-09-16T04:53:03.946220+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17519,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.942739+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:03.942739+0000","last_peered":"2024-09-16T04:53:03.942739+0000","last_clean":"2024-09-16T04:53:03.942739+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:03.942739+0000","last_undegraded":"2024-09-16T04:53:03.942739+0000","last_fullsized":"2024-09-16T04:53:03.942739+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:04.045489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12873,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.945290+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:03.945290+0000","last_peered":"2024-09-16T04:53:03.945290+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:03.945290+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942709+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:03.942709+0000","last_peered":"2024-09-16T04:53:03.942709+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:03.942709+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12432,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942792+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:03.942792+0000","last_peered":"2024-09-16T04:53:03.942792+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:03.942792+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943578+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:03.943578+0000","last_peered":"2024-09-16T04:53:03.943578+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:03.943578+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12922,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943981+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:03.943981+0000","last_peered":"2024-09-16T04:53:03.943981+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:03.943981+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14929,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:04.014146+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:04.014146+0000","last_peered":"2024-09-16T04:53:04.014146+0000","last_clean":"2024-09-16T04:53:04.014146+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:04.014146+0000","last_undegraded":"2024-09-16T04:53:04.014146+0000","last_fullsized":"2024-09-16T04:53:04.014146+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:09.394708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12409,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.948535+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:03.948535+0000","last_peered":"2024-09-16T04:53:03.948535+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:03.948535+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9667,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722008,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649832,"statfs":{"total":96636764160,"available":95897427968,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731228,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640612,"statfs":{"total":96636764160,"available":95887986688,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569848,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93801992,"statfs":{"total":96636764160,"available":96053239808,"internally_reserved":0,"allocated":892928,"data_stored":685038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582610639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721948,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649892,"statfs":{"total":96636764160,"available":95897489408,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:14.997 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:15.337 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:15.337 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:15.352 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-16T04:53:14.971611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9667,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745032,"kb_used_data":1844,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742328,"statfs":{"total":386547056640,"available":383736143872,"internally_reserved":0,"allocated":1888256,"data_stored":1056696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2808923697},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1392,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.163041"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12780,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942729+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:03.942729+0000","last_peered":"2024-09-16T04:53:03.942729+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:03.942729+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12770,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.948572+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:53:03.948572+0000","last_peered":"2024-09-16T04:53:03.948572+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:03.948572+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12778,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.945253+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:03.945253+0000","last_peered":"2024-09-16T04:53:03.945253+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:03.945253+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17223,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.945996+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:03.945996+0000","last_peered":"2024-09-16T04:53:03.945996+0000","last_clean":"2024-09-16T04:53:03.945996+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:03.945996+0000","last_undegraded":"2024-09-16T04:53:03.945996+0000","last_fullsized":"2024-09-16T04:53:03.945996+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:12.421461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12882,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943377+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:03.943377+0000","last_peered":"2024-09-16T04:53:03.943377+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:03.943377+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":12432,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942781+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:53:03.942781+0000","last_peered":"2024-09-16T04:53:03.942781+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:03.942781+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":15491,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:11.231179+0000","last_change":"2024-09-16T04:53:02.647802+0000","last_active":"2024-09-16T04:53:11.231179+0000","last_peered":"2024-09-16T04:53:11.231179+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:53:11.231179+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1407,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-821}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.946220+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:03.946220+0000","last_peered":"2024-09-16T04:53:03.946220+0000","last_clean":"2024-09-16T04:53:03.946220+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:03.946220+0000","last_undegraded":"2024-09-16T04:53:03.946220+0000","last_fullsized":"2024-09-16T04:53:03.946220+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17519,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.942739+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:03.942739+0000","last_peered":"2024-09-16T04:53:03.942739+0000","last_clean":"2024-09-16T04:53:03.942739+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:03.942739+0000","last_undegraded":"2024-09-16T04:53:03.942739+0000","last_fullsized":"2024-09-16T04:53:03.942739+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:04.045489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12873,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.945290+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:03.945290+0000","last_peered":"2024-09-16T04:53:03.945290+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:03.945290+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942709+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:03.942709+0000","last_peered":"2024-09-16T04:53:03.942709+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:03.942709+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12432,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942792+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:03.942792+0000","last_peered":"2024-09-16T04:53:03.942792+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:03.942792+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943578+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:03.943578+0000","last_peered":"2024-09-16T04:53:03.943578+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:03.943578+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12922,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943981+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:03.943981+0000","last_peered":"2024-09-16T04:53:03.943981+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:03.943981+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14929,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:04.014146+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:04.014146+0000","last_peered":"2024-09-16T04:53:04.014146+0000","last_clean":"2024-09-16T04:53:04.014146+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:04.014146+0000","last_undegraded":"2024-09-16T04:53:04.014146+0000","last_fullsized":"2024-09-16T04:53:04.014146+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:09.394708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12409,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.948535+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:03.948535+0000","last_peered":"2024-09-16T04:53:03.948535+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:03.948535+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9667,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722008,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649832,"statfs":{"total":96636764160,"available":95897427968,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731228,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640612,"statfs":{"total":96636764160,"available":95887986688,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569848,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93801992,"statfs":{"total":96636764160,"available":96053239808,"internally_reserved":0,"allocated":892928,"data_stored":685038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582610639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721948,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649892,"statfs":{"total":96636764160,"available":95897489408,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:15.353 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:53:15.353 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 12780, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.942729+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:53:03.942729+0000', 'last_peered': '2024-09-16T04:53:03.942729+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:53:03.942729+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:15.354 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:53:15.354 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12672", 'reported_seq': 12770, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.948572+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:53:03.948572+0000', 'last_peered': '2024-09-16T04:53:03.948572+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:53:03.948572+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12670, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:15.354 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:53:15.354 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12778, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.945253+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:53:03.945253+0000', 'last_peered': '2024-09-16T04:53:03.945253+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:53:03.945253+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:15.354 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:53:15.354 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12882, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.943377+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:53:03.943377+0000', 'last_peered': '2024-09-16T04:53:03.943377+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:53:03.943377+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:15.354 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:53:15.354 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12312", 'reported_seq': 12432, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.942781+0000', 'last_change': '2024-09-16T04:41:27.791991+0000', 'last_active': '2024-09-16T04:53:03.942781+0000', 'last_peered': '2024-09-16T04:53:03.942781+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:53:03.942781+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2178, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12312, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2178}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:15.354 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-16T04:53:15.354 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12598", 'reported_seq': 15491, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:11.231179+0000', 'last_change': '2024-09-16T04:53:02.647802+0000', 'last_active': '2024-09-16T04:53:11.231179+0000', 'last_peered': '2024-09-16T04:53:11.231179+0000', 'last_clean': '2024-09-16T04:40:23.427434+0000', 'last_became_active': '2024-09-16T04:41:27.770687+0000', 'last_became_peered': '2024-09-16T04:41:27.770687+0000', 'last_unstale': '2024-09-16T04:53:11.231179+0000', 'last_undegraded': '2024-09-16T04:41:27.542622+0000', 'last_fullsized': '2024-09-16T04:41:27.482507+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 823, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12596, 'num_write_kb': 50388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1407, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -821}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:15.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:53:15.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12873, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.945290+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:53:03.945290+0000', 'last_peered': '2024-09-16T04:53:03.945290+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:53:03.945290+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:15.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:53:15.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12630, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.942709+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:53:03.942709+0000', 'last_peered': '2024-09-16T04:53:03.942709+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:53:03.942709+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:15.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:53:15.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12432, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.942792+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:53:03.942792+0000', 'last_peered': '2024-09-16T04:53:03.942792+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:53:03.942792+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:15.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:53:15.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12532", 'reported_seq': 12630, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.943578+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:53:03.943578+0000', 'last_peered': '2024-09-16T04:53:03.943578+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:53:03.943578+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12531, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:15.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:53:15.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12922, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.943981+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:53:03.943981+0000', 'last_peered': '2024-09-16T04:53:03.943981+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:53:03.943981+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:15.356 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:53:15.356 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12305", 'reported_seq': 12409, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:03.948535+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:53:03.948535+0000', 'last_peered': '2024-09-16T04:53:03.948535+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:53:03.948535+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12303, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:15.356 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:53:15.777 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:15.777 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24969/112 objects degraded (22293.750%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":808,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2810912768,"bytes_avail":383736143872,"bytes_total":386547056640,"degraded_objects":24969,"degraded_total":112,"degraded_ratio":222.9375,"recovering_objects_per_sec":98,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1067,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:53:15.791 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:53:15.792 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:16.174 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:16.174 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:16.186 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":436,"stamp":"2024-09-16T04:53:15.973972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9667,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745032,"kb_used_data":1844,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742328,"statfs":{"total":386547056640,"available":383736143872,"internally_reserved":0,"allocated":1888256,"data_stored":1056696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2808923697},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.165035"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12780,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942729+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:03.942729+0000","last_peered":"2024-09-16T04:53:03.942729+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:03.942729+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12770,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.948572+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:53:03.948572+0000","last_peered":"2024-09-16T04:53:03.948572+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:03.948572+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12780,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:14.974152+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:14.974152+0000","last_peered":"2024-09-16T04:53:14.974152+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:14.974152+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17223,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.945996+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:03.945996+0000","last_peered":"2024-09-16T04:53:03.945996+0000","last_clean":"2024-09-16T04:53:03.945996+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:03.945996+0000","last_undegraded":"2024-09-16T04:53:03.945996+0000","last_fullsized":"2024-09-16T04:53:03.945996+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:12.421461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12882,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943377+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:03.943377+0000","last_peered":"2024-09-16T04:53:03.943377+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:03.943377+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":12432,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942781+0000","last_change":"2024-09-16T04:41:27.791991+0000","last_active":"2024-09-16T04:53:03.942781+0000","last_peered":"2024-09-16T04:53:03.942781+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:03.942781+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":15491,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:11.231179+0000","last_change":"2024-09-16T04:53:02.647802+0000","last_active":"2024-09-16T04:53:11.231179+0000","last_peered":"2024-09-16T04:53:11.231179+0000","last_clean":"2024-09-16T04:40:23.427434+0000","last_became_active":"2024-09-16T04:41:27.770687+0000","last_became_peered":"2024-09-16T04:41:27.770687+0000","last_unstale":"2024-09-16T04:53:11.231179+0000","last_undegraded":"2024-09-16T04:41:27.542622+0000","last_fullsized":"2024-09-16T04:41:27.482507+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1407,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-821}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.946220+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:03.946220+0000","last_peered":"2024-09-16T04:53:03.946220+0000","last_clean":"2024-09-16T04:53:03.946220+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:03.946220+0000","last_undegraded":"2024-09-16T04:53:03.946220+0000","last_fullsized":"2024-09-16T04:53:03.946220+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17519,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.942739+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:03.942739+0000","last_peered":"2024-09-16T04:53:03.942739+0000","last_clean":"2024-09-16T04:53:03.942739+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:03.942739+0000","last_undegraded":"2024-09-16T04:53:03.942739+0000","last_fullsized":"2024-09-16T04:53:03.942739+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:04.045489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12875,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:14.975433+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:14.975433+0000","last_peered":"2024-09-16T04:53:14.975433+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:14.975433+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942709+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:03.942709+0000","last_peered":"2024-09-16T04:53:03.942709+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:03.942709+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12432,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.942792+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:03.942792+0000","last_peered":"2024-09-16T04:53:03.942792+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:03.942792+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12632,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:14.974838+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:14.974838+0000","last_peered":"2024-09-16T04:53:14.974838+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:14.974838+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12922,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.943981+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:03.943981+0000","last_peered":"2024-09-16T04:53:03.943981+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:03.943981+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14929,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:04.014146+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:04.014146+0000","last_peered":"2024-09-16T04:53:04.014146+0000","last_clean":"2024-09-16T04:53:04.014146+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:04.014146+0000","last_undegraded":"2024-09-16T04:53:04.014146+0000","last_fullsized":"2024-09-16T04:53:04.014146+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:09.394708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12409,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:03.948535+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:03.948535+0000","last_peered":"2024-09-16T04:53:03.948535+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:03.948535+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9667,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722008,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649832,"statfs":{"total":96636764160,"available":95897427968,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731228,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640612,"statfs":{"total":96636764160,"available":95887986688,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569848,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93801992,"statfs":{"total":96636764160,"available":96053239808,"internally_reserved":0,"allocated":892928,"data_stored":685038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582610639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721948,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649892,"statfs":{"total":96636764160,"available":95897489408,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:19.189 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:19.477 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:19.477 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:19.489 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-09-16T04:53:18.811205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10856,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745044,"kb_used_data":1860,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742316,"statfs":{"total":386547056640,"available":383736131584,"internally_reserved":0,"allocated":1904640,"data_stored":1063646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2808923697},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2042,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001562"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12784,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.981003+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:15.981003+0000","last_peered":"2024-09-16T04:53:15.981003+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:15.981003+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12774,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975688+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:53:15.975688+0000","last_peered":"2024-09-16T04:53:15.975688+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:15.975688+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12780,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:14.974152+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:14.974152+0000","last_peered":"2024-09-16T04:53:14.974152+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:14.974152+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17223,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.945996+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:03.945996+0000","last_peered":"2024-09-16T04:53:03.945996+0000","last_clean":"2024-09-16T04:53:03.945996+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:03.945996+0000","last_undegraded":"2024-09-16T04:53:03.945996+0000","last_fullsized":"2024-09-16T04:53:03.945996+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:12.421461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12886,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978428+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:15.978428+0000","last_peered":"2024-09-16T04:53:15.978428+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:15.978428+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":13169,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:16.907717+0000","last_change":"2024-09-16T04:53:14.841453+0000","last_active":"2024-09-16T04:53:16.907717+0000","last_peered":"2024-09-16T04:53:16.907717+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:16.907717+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1812}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17149,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:16.100110+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:16.100110+0000","last_peered":"2024-09-16T04:53:16.100110+0000","last_clean":"2024-09-16T04:53:16.100110+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:16.100110+0000","last_undegraded":"2024-09-16T04:53:16.100110+0000","last_fullsized":"2024-09-16T04:53:16.100110+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.946220+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:03.946220+0000","last_peered":"2024-09-16T04:53:03.946220+0000","last_clean":"2024-09-16T04:53:03.946220+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:03.946220+0000","last_undegraded":"2024-09-16T04:53:03.946220+0000","last_fullsized":"2024-09-16T04:53:03.946220+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17523,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.979935+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:15.979935+0000","last_peered":"2024-09-16T04:53:15.979935+0000","last_clean":"2024-09-16T04:53:15.979935+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:15.979935+0000","last_undegraded":"2024-09-16T04:53:15.979935+0000","last_fullsized":"2024-09-16T04:53:15.979935+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:20.946135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12875,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:14.975433+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:14.975433+0000","last_peered":"2024-09-16T04:53:14.975433+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:14.975433+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12634,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978546+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:15.978546+0000","last_peered":"2024-09-16T04:53:15.978546+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:15.978546+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12436,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978725+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:15.978725+0000","last_peered":"2024-09-16T04:53:15.978725+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:15.978725+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12632,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:14.974838+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:14.974838+0000","last_peered":"2024-09-16T04:53:14.974838+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:14.974838+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12926,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978662+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:15.978662+0000","last_peered":"2024-09-16T04:53:15.978662+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:15.978662+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14929,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:04.014146+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:04.014146+0000","last_peered":"2024-09-16T04:53:04.014146+0000","last_clean":"2024-09-16T04:53:04.014146+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:04.014146+0000","last_undegraded":"2024-09-16T04:53:04.014146+0000","last_fullsized":"2024-09-16T04:53:04.014146+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:09.394708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12413,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975272+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:15.975272+0000","last_peered":"2024-09-16T04:53:15.975272+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:15.975272+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10856,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722016,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649824,"statfs":{"total":96636764160,"available":95897419776,"internally_reserved":0,"allocated":339968,"data_stored":127361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731228,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640612,"statfs":{"total":96636764160,"available":95887986688,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569848,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93801992,"statfs":{"total":96636764160,"available":96053239808,"internally_reserved":0,"allocated":892928,"data_stored":685038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582610639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721952,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649888,"statfs":{"total":96636764160,"available":95897485312,"internally_reserved":0,"allocated":339968,"data_stored":127361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:19.491 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:19.883 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:19.884 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:19.896 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-09-16T04:53:18.811205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10856,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745044,"kb_used_data":1860,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742316,"statfs":{"total":386547056640,"available":383736131584,"internally_reserved":0,"allocated":1904640,"data_stored":1063646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2808923697},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2042,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001562"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12784,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.981003+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:15.981003+0000","last_peered":"2024-09-16T04:53:15.981003+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:15.981003+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12774,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975688+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:53:15.975688+0000","last_peered":"2024-09-16T04:53:15.975688+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:15.975688+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12780,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:14.974152+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:14.974152+0000","last_peered":"2024-09-16T04:53:14.974152+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:14.974152+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17223,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.945996+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:03.945996+0000","last_peered":"2024-09-16T04:53:03.945996+0000","last_clean":"2024-09-16T04:53:03.945996+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:03.945996+0000","last_undegraded":"2024-09-16T04:53:03.945996+0000","last_fullsized":"2024-09-16T04:53:03.945996+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:12.421461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12886,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978428+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:15.978428+0000","last_peered":"2024-09-16T04:53:15.978428+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:15.978428+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":13169,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:16.907717+0000","last_change":"2024-09-16T04:53:14.841453+0000","last_active":"2024-09-16T04:53:16.907717+0000","last_peered":"2024-09-16T04:53:16.907717+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:16.907717+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1812}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17149,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:16.100110+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:16.100110+0000","last_peered":"2024-09-16T04:53:16.100110+0000","last_clean":"2024-09-16T04:53:16.100110+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:16.100110+0000","last_undegraded":"2024-09-16T04:53:16.100110+0000","last_fullsized":"2024-09-16T04:53:16.100110+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.946220+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:03.946220+0000","last_peered":"2024-09-16T04:53:03.946220+0000","last_clean":"2024-09-16T04:53:03.946220+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:03.946220+0000","last_undegraded":"2024-09-16T04:53:03.946220+0000","last_fullsized":"2024-09-16T04:53:03.946220+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17523,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.979935+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:15.979935+0000","last_peered":"2024-09-16T04:53:15.979935+0000","last_clean":"2024-09-16T04:53:15.979935+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:15.979935+0000","last_undegraded":"2024-09-16T04:53:15.979935+0000","last_fullsized":"2024-09-16T04:53:15.979935+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:20.946135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12875,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:14.975433+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:14.975433+0000","last_peered":"2024-09-16T04:53:14.975433+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:14.975433+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12634,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978546+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:15.978546+0000","last_peered":"2024-09-16T04:53:15.978546+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:15.978546+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12436,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978725+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:15.978725+0000","last_peered":"2024-09-16T04:53:15.978725+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:15.978725+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12632,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:14.974838+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:14.974838+0000","last_peered":"2024-09-16T04:53:14.974838+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:14.974838+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12926,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978662+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:15.978662+0000","last_peered":"2024-09-16T04:53:15.978662+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:15.978662+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14929,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:04.014146+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:04.014146+0000","last_peered":"2024-09-16T04:53:04.014146+0000","last_clean":"2024-09-16T04:53:04.014146+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:04.014146+0000","last_undegraded":"2024-09-16T04:53:04.014146+0000","last_fullsized":"2024-09-16T04:53:04.014146+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:09.394708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12413,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975272+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:15.975272+0000","last_peered":"2024-09-16T04:53:15.975272+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:15.975272+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10856,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722016,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649824,"statfs":{"total":96636764160,"available":95897419776,"internally_reserved":0,"allocated":339968,"data_stored":127361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731228,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640612,"statfs":{"total":96636764160,"available":95887986688,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569848,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93801992,"statfs":{"total":96636764160,"available":96053239808,"internally_reserved":0,"allocated":892928,"data_stored":685038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582610639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721952,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649888,"statfs":{"total":96636764160,"available":95897485312,"internally_reserved":0,"allocated":339968,"data_stored":127361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:19.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:53:19.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 12784, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.981003+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:53:15.981003+0000', 'last_peered': '2024-09-16T04:53:15.981003+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:53:15.981003+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:19.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:53:19.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12672", 'reported_seq': 12774, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.975688+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:53:15.975688+0000', 'last_peered': '2024-09-16T04:53:15.975688+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:53:15.975688+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12670, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:19.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:53:19.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12780, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:14.974152+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:53:14.974152+0000', 'last_peered': '2024-09-16T04:53:14.974152+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:53:14.974152+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:19.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:53:19.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12886, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.978428+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:53:15.978428+0000', 'last_peered': '2024-09-16T04:53:15.978428+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:53:15.978428+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:19.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:53:19.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12312", 'reported_seq': 13169, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:16.907717+0000', 'last_change': '2024-09-16T04:53:14.841453+0000', 'last_active': '2024-09-16T04:53:16.907717+0000', 'last_peered': '2024-09-16T04:53:16.907717+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:53:16.907717+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1812, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12312, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 396, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1812}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:19.899 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:53:19.899 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12875, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:14.975433+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:53:14.975433+0000', 'last_peered': '2024-09-16T04:53:14.975433+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:53:14.975433+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:19.899 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:53:19.899 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12634, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.978546+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:53:15.978546+0000', 'last_peered': '2024-09-16T04:53:15.978546+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:53:15.978546+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:19.899 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:53:19.899 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12436, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.978725+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:53:15.978725+0000', 'last_peered': '2024-09-16T04:53:15.978725+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:53:15.978725+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:19.900 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:53:19.900 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12532", 'reported_seq': 12632, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:14.974838+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:53:14.974838+0000', 'last_peered': '2024-09-16T04:53:14.974838+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:53:14.974838+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12531, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:19.900 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:53:19.900 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12926, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.978662+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:53:15.978662+0000', 'last_peered': '2024-09-16T04:53:15.978662+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:53:15.978662+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:19.900 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:53:19.900 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12305", 'reported_seq': 12413, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.975272+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:53:15.975272+0000', 'last_peered': '2024-09-16T04:53:15.975272+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:53:15.975272+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12303, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:19.901 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:53:20.269 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:20.270 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23780/112 objects degraded (21232.143%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":812,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2810925056,"bytes_avail":383736131584,"bytes_total":386547056640,"degraded_objects":23780,"degraded_total":112,"degraded_ratio":212.32142857142858,"recovering_objects_per_sec":255,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2042,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:53:20.283 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:53:20.283 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:20.579 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:20.580 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:20.592 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-09-16T04:53:18.811205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10856,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745044,"kb_used_data":1860,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742316,"statfs":{"total":386547056640,"available":383736131584,"internally_reserved":0,"allocated":1904640,"data_stored":1063646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2808923697},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2042,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001562"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12784,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.981003+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:15.981003+0000","last_peered":"2024-09-16T04:53:15.981003+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:15.981003+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12774,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975688+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:53:15.975688+0000","last_peered":"2024-09-16T04:53:15.975688+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:15.975688+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12780,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:14.974152+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:14.974152+0000","last_peered":"2024-09-16T04:53:14.974152+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:14.974152+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17223,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.945996+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:03.945996+0000","last_peered":"2024-09-16T04:53:03.945996+0000","last_clean":"2024-09-16T04:53:03.945996+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:03.945996+0000","last_undegraded":"2024-09-16T04:53:03.945996+0000","last_fullsized":"2024-09-16T04:53:03.945996+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:12.421461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12886,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978428+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:15.978428+0000","last_peered":"2024-09-16T04:53:15.978428+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:15.978428+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":13169,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:16.907717+0000","last_change":"2024-09-16T04:53:14.841453+0000","last_active":"2024-09-16T04:53:16.907717+0000","last_peered":"2024-09-16T04:53:16.907717+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:16.907717+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1812}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17149,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:16.100110+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:16.100110+0000","last_peered":"2024-09-16T04:53:16.100110+0000","last_clean":"2024-09-16T04:53:16.100110+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:16.100110+0000","last_undegraded":"2024-09-16T04:53:16.100110+0000","last_fullsized":"2024-09-16T04:53:16.100110+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:03.946220+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:03.946220+0000","last_peered":"2024-09-16T04:53:03.946220+0000","last_clean":"2024-09-16T04:53:03.946220+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:03.946220+0000","last_undegraded":"2024-09-16T04:53:03.946220+0000","last_fullsized":"2024-09-16T04:53:03.946220+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:07.357133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17523,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.979935+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:15.979935+0000","last_peered":"2024-09-16T04:53:15.979935+0000","last_clean":"2024-09-16T04:53:15.979935+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:15.979935+0000","last_undegraded":"2024-09-16T04:53:15.979935+0000","last_fullsized":"2024-09-16T04:53:15.979935+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:20.946135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12875,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:14.975433+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:14.975433+0000","last_peered":"2024-09-16T04:53:14.975433+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:14.975433+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12634,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978546+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:15.978546+0000","last_peered":"2024-09-16T04:53:15.978546+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:15.978546+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12436,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978725+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:15.978725+0000","last_peered":"2024-09-16T04:53:15.978725+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:15.978725+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12632,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:14.974838+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:14.974838+0000","last_peered":"2024-09-16T04:53:14.974838+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:14.974838+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12926,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978662+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:15.978662+0000","last_peered":"2024-09-16T04:53:15.978662+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:15.978662+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14929,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T04:53:04.014146+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:04.014146+0000","last_peered":"2024-09-16T04:53:04.014146+0000","last_clean":"2024-09-16T04:53:04.014146+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:04.014146+0000","last_undegraded":"2024-09-16T04:53:04.014146+0000","last_fullsized":"2024-09-16T04:53:04.014146+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:09.394708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12413,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975272+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:15.975272+0000","last_peered":"2024-09-16T04:53:15.975272+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:15.975272+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10856,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722016,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649824,"statfs":{"total":96636764160,"available":95897419776,"internally_reserved":0,"allocated":339968,"data_stored":127361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731228,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640612,"statfs":{"total":96636764160,"available":95887986688,"internally_reserved":0,"allocated":331776,"data_stored":123886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569848,"kb_used_data":872,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93801992,"statfs":{"total":96636764160,"available":96053239808,"internally_reserved":0,"allocated":892928,"data_stored":685038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582610639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721952,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649888,"statfs":{"total":96636764160,"available":95897485312,"internally_reserved":0,"allocated":339968,"data_stored":127361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:23.594 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:23.893 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:23.893 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:23.906 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-16T04:53:22.811909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11734,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2753416,"kb_used_data":1908,"kb_used_omap":14,"kb_used_meta":2751409,"kb_avail":374733944,"statfs":{"total":386547056640,"available":383727558656,"internally_reserved":0,"allocated":1953792,"data_stored":1084412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2817443377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2067,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001389"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12784,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.981003+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:15.981003+0000","last_peered":"2024-09-16T04:53:15.981003+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:15.981003+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12774,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975688+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:53:15.975688+0000","last_peered":"2024-09-16T04:53:15.975688+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:15.975688+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12782,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.979903+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:15.979903+0000","last_peered":"2024-09-16T04:53:15.979903+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:15.979903+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17227,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977622+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:15.977622+0000","last_peered":"2024-09-16T04:53:15.977622+0000","last_clean":"2024-09-16T04:53:15.977622+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:15.977622+0000","last_undegraded":"2024-09-16T04:53:15.977622+0000","last_fullsized":"2024-09-16T04:53:15.977622+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:17.471503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12886,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978428+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:15.978428+0000","last_peered":"2024-09-16T04:53:15.978428+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:15.978428+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":14925,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:21.913268+0000","last_change":"2024-09-16T04:53:14.841453+0000","last_active":"2024-09-16T04:53:21.913268+0000","last_peered":"2024-09-16T04:53:21.913268+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:21.913268+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-934}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17149,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:16.100110+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:16.100110+0000","last_peered":"2024-09-16T04:53:16.100110+0000","last_clean":"2024-09-16T04:53:16.100110+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:16.100110+0000","last_undegraded":"2024-09-16T04:53:16.100110+0000","last_fullsized":"2024-09-16T04:53:16.100110+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17359,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977987+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:15.977987+0000","last_peered":"2024-09-16T04:53:15.977987+0000","last_clean":"2024-09-16T04:53:15.977987+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:15.977987+0000","last_undegraded":"2024-09-16T04:53:15.977987+0000","last_fullsized":"2024-09-16T04:53:15.977987+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:17.471488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17524,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.979987+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:15.979987+0000","last_peered":"2024-09-16T04:53:15.979987+0000","last_clean":"2024-09-16T04:53:15.979987+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:15.979987+0000","last_undegraded":"2024-09-16T04:53:15.979987+0000","last_fullsized":"2024-09-16T04:53:15.979987+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:25.982711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12877,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.981122+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:15.981122+0000","last_peered":"2024-09-16T04:53:15.981122+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:15.981122+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12634,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978546+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:15.978546+0000","last_peered":"2024-09-16T04:53:15.978546+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:15.978546+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12436,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978725+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:15.978725+0000","last_peered":"2024-09-16T04:53:15.978725+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:15.978725+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12634,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.977926+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:15.977926+0000","last_peered":"2024-09-16T04:53:15.977926+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:15.977926+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12926,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978662+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:15.978662+0000","last_peered":"2024-09-16T04:53:15.978662+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:15.978662+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14933,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977993+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:15.977993+0000","last_peered":"2024-09-16T04:53:15.977993+0000","last_clean":"2024-09-16T04:53:15.977993+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:15.977993+0000","last_undegraded":"2024-09-16T04:53:15.977993+0000","last_fullsized":"2024-09-16T04:53:15.977993+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:20.507737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12413,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975272+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:15.975272+0000","last_peered":"2024-09-16T04:53:15.975272+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:15.975272+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11734,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722024,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649816,"statfs":{"total":96636764160,"available":95897411584,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731244,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640596,"statfs":{"total":96636764160,"available":95887970304,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578184,"kb_used_data":888,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93793656,"statfs":{"total":96636764160,"available":96044703744,"internally_reserved":0,"allocated":909312,"data_stored":691967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591130319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721964,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649876,"statfs":{"total":96636764160,"available":95897473024,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:23.907 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:24.202 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:24.202 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:24.215 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-16T04:53:22.811909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11734,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2753416,"kb_used_data":1908,"kb_used_omap":14,"kb_used_meta":2751409,"kb_avail":374733944,"statfs":{"total":386547056640,"available":383727558656,"internally_reserved":0,"allocated":1953792,"data_stored":1084412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2817443377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2067,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001389"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12784,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.981003+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:15.981003+0000","last_peered":"2024-09-16T04:53:15.981003+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:15.981003+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12774,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975688+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:53:15.975688+0000","last_peered":"2024-09-16T04:53:15.975688+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:15.975688+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12782,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.979903+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:15.979903+0000","last_peered":"2024-09-16T04:53:15.979903+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:15.979903+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17227,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977622+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:15.977622+0000","last_peered":"2024-09-16T04:53:15.977622+0000","last_clean":"2024-09-16T04:53:15.977622+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:15.977622+0000","last_undegraded":"2024-09-16T04:53:15.977622+0000","last_fullsized":"2024-09-16T04:53:15.977622+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:17.471503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12886,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978428+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:15.978428+0000","last_peered":"2024-09-16T04:53:15.978428+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:15.978428+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":14925,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:21.913268+0000","last_change":"2024-09-16T04:53:14.841453+0000","last_active":"2024-09-16T04:53:21.913268+0000","last_peered":"2024-09-16T04:53:21.913268+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:21.913268+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-934}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17149,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:16.100110+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:16.100110+0000","last_peered":"2024-09-16T04:53:16.100110+0000","last_clean":"2024-09-16T04:53:16.100110+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:16.100110+0000","last_undegraded":"2024-09-16T04:53:16.100110+0000","last_fullsized":"2024-09-16T04:53:16.100110+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17359,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977987+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:15.977987+0000","last_peered":"2024-09-16T04:53:15.977987+0000","last_clean":"2024-09-16T04:53:15.977987+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:15.977987+0000","last_undegraded":"2024-09-16T04:53:15.977987+0000","last_fullsized":"2024-09-16T04:53:15.977987+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:17.471488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17524,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.979987+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:15.979987+0000","last_peered":"2024-09-16T04:53:15.979987+0000","last_clean":"2024-09-16T04:53:15.979987+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:15.979987+0000","last_undegraded":"2024-09-16T04:53:15.979987+0000","last_fullsized":"2024-09-16T04:53:15.979987+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:25.982711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12877,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.981122+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:15.981122+0000","last_peered":"2024-09-16T04:53:15.981122+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:15.981122+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12634,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978546+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:15.978546+0000","last_peered":"2024-09-16T04:53:15.978546+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:15.978546+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12436,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978725+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:15.978725+0000","last_peered":"2024-09-16T04:53:15.978725+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:15.978725+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12634,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.977926+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:15.977926+0000","last_peered":"2024-09-16T04:53:15.977926+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:15.977926+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12926,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978662+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:15.978662+0000","last_peered":"2024-09-16T04:53:15.978662+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:15.978662+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14933,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977993+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:15.977993+0000","last_peered":"2024-09-16T04:53:15.977993+0000","last_clean":"2024-09-16T04:53:15.977993+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:15.977993+0000","last_undegraded":"2024-09-16T04:53:15.977993+0000","last_fullsized":"2024-09-16T04:53:15.977993+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:20.507737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12413,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975272+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:15.975272+0000","last_peered":"2024-09-16T04:53:15.975272+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:15.975272+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11734,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722024,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649816,"statfs":{"total":96636764160,"available":95897411584,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731244,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640596,"statfs":{"total":96636764160,"available":95887970304,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578184,"kb_used_data":888,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93793656,"statfs":{"total":96636764160,"available":96044703744,"internally_reserved":0,"allocated":909312,"data_stored":691967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591130319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721964,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649876,"statfs":{"total":96636764160,"available":95897473024,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:24.216 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:53:24.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 12784, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.981003+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:53:15.981003+0000', 'last_peered': '2024-09-16T04:53:15.981003+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:53:15.981003+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:24.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:53:24.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12672", 'reported_seq': 12774, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.975688+0000', 'last_change': '2024-09-16T04:41:27.871224+0000', 'last_active': '2024-09-16T04:53:15.975688+0000', 'last_peered': '2024-09-16T04:53:15.975688+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:53:15.975688+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12670, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:24.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:53:24.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12782, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.979903+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:53:15.979903+0000', 'last_peered': '2024-09-16T04:53:15.979903+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:53:15.979903+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:24.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:53:24.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12886, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.978428+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:53:15.978428+0000', 'last_peered': '2024-09-16T04:53:15.978428+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:53:15.978428+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:24.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:53:24.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12312", 'reported_seq': 14925, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:21.913268+0000', 'last_change': '2024-09-16T04:53:14.841453+0000', 'last_active': '2024-09-16T04:53:21.913268+0000', 'last_peered': '2024-09-16T04:53:21.913268+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:53:21.913268+0000', 'last_undegraded': '2024-09-16T04:41:27.558213+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 934, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12312, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1274, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -934}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:24.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:53:24.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12877, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.981122+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:53:15.981122+0000', 'last_peered': '2024-09-16T04:53:15.981122+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:53:15.981122+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:24.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:53:24.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12634, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.978546+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:53:15.978546+0000', 'last_peered': '2024-09-16T04:53:15.978546+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:53:15.978546+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:24.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:53:24.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12436, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.978725+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:53:15.978725+0000', 'last_peered': '2024-09-16T04:53:15.978725+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:53:15.978725+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:24.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:53:24.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12532", 'reported_seq': 12634, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.977926+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:53:15.977926+0000', 'last_peered': '2024-09-16T04:53:15.977926+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:53:15.977926+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12531, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:24.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:53:24.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12926, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.978662+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:53:15.978662+0000', 'last_peered': '2024-09-16T04:53:15.978662+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:53:15.978662+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:24.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:53:24.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12305", 'reported_seq': 12413, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.975272+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:53:15.975272+0000', 'last_peered': '2024-09-16T04:53:15.975272+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:53:15.975272+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12303, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:24.219 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:53:24.603 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:24.603 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22902/112 objects degraded (20448.214%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":816,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2819497984,"bytes_avail":383727558656,"bytes_total":386547056640,"degraded_objects":22902,"degraded_total":112,"degraded_ratio":204.48214285714286,"recovering_objects_per_sec":258,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2067,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:53:24.616 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:53:24.617 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:24.908 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:24.908 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:24.922 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":441,"stamp":"2024-09-16T04:53:24.812290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11734,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745096,"kb_used_data":1908,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742264,"statfs":{"total":386547056640,"available":383736078336,"internally_reserved":0,"allocated":1953792,"data_stored":1084412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2808923697},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2067,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.840679"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12784,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.981003+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:15.981003+0000","last_peered":"2024-09-16T04:53:15.981003+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:15.981003+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12774,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975688+0000","last_change":"2024-09-16T04:41:27.871224+0000","last_active":"2024-09-16T04:53:15.975688+0000","last_peered":"2024-09-16T04:53:15.975688+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:15.975688+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12782,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.979903+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:15.979903+0000","last_peered":"2024-09-16T04:53:15.979903+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:15.979903+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17227,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977622+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:15.977622+0000","last_peered":"2024-09-16T04:53:15.977622+0000","last_clean":"2024-09-16T04:53:15.977622+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:15.977622+0000","last_undegraded":"2024-09-16T04:53:15.977622+0000","last_fullsized":"2024-09-16T04:53:15.977622+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:17.471503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12886,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978428+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:15.978428+0000","last_peered":"2024-09-16T04:53:15.978428+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:15.978428+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":14925,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:21.913268+0000","last_change":"2024-09-16T04:53:14.841453+0000","last_active":"2024-09-16T04:53:21.913268+0000","last_peered":"2024-09-16T04:53:21.913268+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:21.913268+0000","last_undegraded":"2024-09-16T04:41:27.558213+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-934}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17149,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:16.100110+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:16.100110+0000","last_peered":"2024-09-16T04:53:16.100110+0000","last_clean":"2024-09-16T04:53:16.100110+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:16.100110+0000","last_undegraded":"2024-09-16T04:53:16.100110+0000","last_fullsized":"2024-09-16T04:53:16.100110+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17359,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977987+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:15.977987+0000","last_peered":"2024-09-16T04:53:15.977987+0000","last_clean":"2024-09-16T04:53:15.977987+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:15.977987+0000","last_undegraded":"2024-09-16T04:53:15.977987+0000","last_fullsized":"2024-09-16T04:53:15.977987+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:17.471488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17524,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.979987+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:15.979987+0000","last_peered":"2024-09-16T04:53:15.979987+0000","last_clean":"2024-09-16T04:53:15.979987+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:15.979987+0000","last_undegraded":"2024-09-16T04:53:15.979987+0000","last_fullsized":"2024-09-16T04:53:15.979987+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:25.982711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12877,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.981122+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:15.981122+0000","last_peered":"2024-09-16T04:53:15.981122+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:15.981122+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12634,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978546+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:15.978546+0000","last_peered":"2024-09-16T04:53:15.978546+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:15.978546+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12436,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978725+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:15.978725+0000","last_peered":"2024-09-16T04:53:15.978725+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:15.978725+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12634,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.977926+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:15.977926+0000","last_peered":"2024-09-16T04:53:15.977926+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:15.977926+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12926,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.978662+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:15.978662+0000","last_peered":"2024-09-16T04:53:15.978662+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:15.978662+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14933,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977993+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:15.977993+0000","last_peered":"2024-09-16T04:53:15.977993+0000","last_clean":"2024-09-16T04:53:15.977993+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:15.977993+0000","last_undegraded":"2024-09-16T04:53:15.977993+0000","last_fullsized":"2024-09-16T04:53:15.977993+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:20.507737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12413,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975272+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:15.975272+0000","last_peered":"2024-09-16T04:53:15.975272+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:15.975272+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11734,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722024,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649816,"statfs":{"total":96636764160,"available":95897411584,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731244,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640596,"statfs":{"total":96636764160,"available":95887970304,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569864,"kb_used_data":888,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93801976,"statfs":{"total":96636764160,"available":96053223424,"internally_reserved":0,"allocated":909312,"data_stored":691967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582610639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721964,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649876,"statfs":{"total":96636764160,"available":95897473024,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:27.924 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:28.229 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:28.229 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:28.241 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-16T04:53:27.584276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12697,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745104,"kb_used_data":1916,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742256,"statfs":{"total":386547056640,"available":383736070144,"internally_reserved":0,"allocated":1961984,"data_stored":1087850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2808923697},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1841,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1841,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.773071"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12786,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.581158+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:26.581158+0000","last_peered":"2024-09-16T04:53:26.581158+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:26.581158+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12833,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.226197+0000","last_change":"2024-09-16T04:53:26.019219+0000","last_active":"2024-09-16T04:53:26.226197+0000","last_peered":"2024-09-16T04:53:26.226197+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:26.226197+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12782,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.979903+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:15.979903+0000","last_peered":"2024-09-16T04:53:15.979903+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:15.979903+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17227,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977622+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:15.977622+0000","last_peered":"2024-09-16T04:53:15.977622+0000","last_clean":"2024-09-16T04:53:15.977622+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:15.977622+0000","last_undegraded":"2024-09-16T04:53:15.977622+0000","last_fullsized":"2024-09-16T04:53:15.977622+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:17.471503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12888,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.579111+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:26.579111+0000","last_peered":"2024-09-16T04:53:26.579111+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:26.579111+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16795,"reported_epoch":41,"state":"peering","last_fresh":"2024-09-16T04:53:26.018418+0000","last_change":"2024-09-16T04:53:26.585410+0000","last_active":"2024-09-16T04:53:26.585410+0000","last_peered":"2024-09-16T04:53:26.018418+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:26.018418+0000","last_undegraded":"2024-09-16T04:53:26.018418+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17149,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:16.100110+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:16.100110+0000","last_peered":"2024-09-16T04:53:16.100110+0000","last_clean":"2024-09-16T04:53:16.100110+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:16.100110+0000","last_undegraded":"2024-09-16T04:53:16.100110+0000","last_fullsized":"2024-09-16T04:53:16.100110+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17359,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977987+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:15.977987+0000","last_peered":"2024-09-16T04:53:15.977987+0000","last_clean":"2024-09-16T04:53:15.977987+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:15.977987+0000","last_undegraded":"2024-09-16T04:53:15.977987+0000","last_fullsized":"2024-09-16T04:53:15.977987+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:17.471488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17526,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T04:53:26.579113+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:26.579113+0000","last_peered":"2024-09-16T04:53:26.579113+0000","last_clean":"2024-09-16T04:53:26.579113+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:26.579113+0000","last_undegraded":"2024-09-16T04:53:26.579113+0000","last_fullsized":"2024-09-16T04:53:26.579113+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12877,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.981122+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:15.981122+0000","last_peered":"2024-09-16T04:53:15.981122+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:15.981122+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12637,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.581918+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:26.581918+0000","last_peered":"2024-09-16T04:53:26.581918+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:26.581918+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12438,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.581763+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:26.581763+0000","last_peered":"2024-09-16T04:53:26.581763+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:26.581763+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12634,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.977926+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:15.977926+0000","last_peered":"2024-09-16T04:53:15.977926+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:15.977926+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12928,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.579298+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:26.579298+0000","last_peered":"2024-09-16T04:53:26.579298+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:26.579298+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14933,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977993+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:15.977993+0000","last_peered":"2024-09-16T04:53:15.977993+0000","last_clean":"2024-09-16T04:53:15.977993+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:15.977993+0000","last_undegraded":"2024-09-16T04:53:15.977993+0000","last_fullsized":"2024-09-16T04:53:15.977993+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:20.507737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12413,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975272+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:15.975272+0000","last_peered":"2024-09-16T04:53:15.975272+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:15.975272+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12697,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722032,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649808,"statfs":{"total":96636764160,"available":95897403392,"internally_reserved":0,"allocated":356352,"data_stored":134253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731244,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640596,"statfs":{"total":96636764160,"available":95887970304,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569864,"kb_used_data":888,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93801976,"statfs":{"total":96636764160,"available":96053223424,"internally_reserved":0,"allocated":909312,"data_stored":691967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582610639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721964,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649876,"statfs":{"total":96636764160,"available":95897473024,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:28.242 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:28.571 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:28.572 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:28.584 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-16T04:53:27.584276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12697,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745104,"kb_used_data":1916,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742256,"statfs":{"total":386547056640,"available":383736070144,"internally_reserved":0,"allocated":1961984,"data_stored":1087850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2808923697},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1841,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1841,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.773071"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12786,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.581158+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:26.581158+0000","last_peered":"2024-09-16T04:53:26.581158+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:26.581158+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12833,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.226197+0000","last_change":"2024-09-16T04:53:26.019219+0000","last_active":"2024-09-16T04:53:26.226197+0000","last_peered":"2024-09-16T04:53:26.226197+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:26.226197+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12782,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.979903+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:15.979903+0000","last_peered":"2024-09-16T04:53:15.979903+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:15.979903+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17227,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977622+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:15.977622+0000","last_peered":"2024-09-16T04:53:15.977622+0000","last_clean":"2024-09-16T04:53:15.977622+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:15.977622+0000","last_undegraded":"2024-09-16T04:53:15.977622+0000","last_fullsized":"2024-09-16T04:53:15.977622+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:17.471503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12888,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.579111+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:26.579111+0000","last_peered":"2024-09-16T04:53:26.579111+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:26.579111+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16795,"reported_epoch":41,"state":"peering","last_fresh":"2024-09-16T04:53:26.018418+0000","last_change":"2024-09-16T04:53:26.585410+0000","last_active":"2024-09-16T04:53:26.585410+0000","last_peered":"2024-09-16T04:53:26.018418+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:26.018418+0000","last_undegraded":"2024-09-16T04:53:26.018418+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17149,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:16.100110+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:16.100110+0000","last_peered":"2024-09-16T04:53:16.100110+0000","last_clean":"2024-09-16T04:53:16.100110+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:16.100110+0000","last_undegraded":"2024-09-16T04:53:16.100110+0000","last_fullsized":"2024-09-16T04:53:16.100110+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17359,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977987+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:15.977987+0000","last_peered":"2024-09-16T04:53:15.977987+0000","last_clean":"2024-09-16T04:53:15.977987+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:15.977987+0000","last_undegraded":"2024-09-16T04:53:15.977987+0000","last_fullsized":"2024-09-16T04:53:15.977987+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:17.471488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17526,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T04:53:26.579113+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:26.579113+0000","last_peered":"2024-09-16T04:53:26.579113+0000","last_clean":"2024-09-16T04:53:26.579113+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:26.579113+0000","last_undegraded":"2024-09-16T04:53:26.579113+0000","last_fullsized":"2024-09-16T04:53:26.579113+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12877,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.981122+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:15.981122+0000","last_peered":"2024-09-16T04:53:15.981122+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:15.981122+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12637,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.581918+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:26.581918+0000","last_peered":"2024-09-16T04:53:26.581918+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:26.581918+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12438,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.581763+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:26.581763+0000","last_peered":"2024-09-16T04:53:26.581763+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:26.581763+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12634,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.977926+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:15.977926+0000","last_peered":"2024-09-16T04:53:15.977926+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:15.977926+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12928,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.579298+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:26.579298+0000","last_peered":"2024-09-16T04:53:26.579298+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:26.579298+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14933,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977993+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:15.977993+0000","last_peered":"2024-09-16T04:53:15.977993+0000","last_clean":"2024-09-16T04:53:15.977993+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:15.977993+0000","last_undegraded":"2024-09-16T04:53:15.977993+0000","last_fullsized":"2024-09-16T04:53:15.977993+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:20.507737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12413,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975272+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:15.975272+0000","last_peered":"2024-09-16T04:53:15.975272+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:15.975272+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12697,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722032,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649808,"statfs":{"total":96636764160,"available":95897403392,"internally_reserved":0,"allocated":356352,"data_stored":134253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731244,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640596,"statfs":{"total":96636764160,"available":95887970304,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569864,"kb_used_data":888,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93801976,"statfs":{"total":96636764160,"available":96053223424,"internally_reserved":0,"allocated":909312,"data_stored":691967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582610639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721964,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649876,"statfs":{"total":96636764160,"available":95897473024,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:28.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:53:28.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 12786, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:26.581158+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:53:26.581158+0000', 'last_peered': '2024-09-16T04:53:26.581158+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:53:26.581158+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:28.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:53:28.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12672", 'reported_seq': 12833, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:26.226197+0000', 'last_change': '2024-09-16T04:53:26.019219+0000', 'last_active': '2024-09-16T04:53:26.226197+0000', 'last_peered': '2024-09-16T04:53:26.226197+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:53:26.226197+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2176, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12670, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -2174}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:28.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:53:28.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12782, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.979903+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:53:15.979903+0000', 'last_peered': '2024-09-16T04:53:15.979903+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:53:15.979903+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:28.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:53:28.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12888, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:26.579111+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:53:26.579111+0000', 'last_peered': '2024-09-16T04:53:26.579111+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:53:26.579111+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:28.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-16T04:53:28.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12312", 'reported_seq': 16795, 'reported_epoch': 41, 'state': 'peering', 'last_fresh': '2024-09-16T04:53:26.018418+0000', 'last_change': '2024-09-16T04:53:26.585410+0000', 'last_active': '2024-09-16T04:53:26.585410+0000', 'last_peered': '2024-09-16T04:53:26.018418+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.781816+0000', 'last_became_peered': '2024-09-16T04:41:27.781816+0000', 'last_unstale': '2024-09-16T04:53:26.018418+0000', 'last_undegraded': '2024-09-16T04:53:26.018418+0000', 'last_fullsized': '2024-09-16T04:41:27.483719+0000', 'mapping_epoch': 42, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12312, 'num_write_kb': 49248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2208, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:28.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:53:28.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12877, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.981122+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:53:15.981122+0000', 'last_peered': '2024-09-16T04:53:15.981122+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:53:15.981122+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:28.587 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:53:28.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12637, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:26.581918+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:53:26.581918+0000', 'last_peered': '2024-09-16T04:53:26.581918+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:53:26.581918+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:28.587 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:53:28.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12438, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:26.581763+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:53:26.581763+0000', 'last_peered': '2024-09-16T04:53:26.581763+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:53:26.581763+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:28.587 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:53:28.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12532", 'reported_seq': 12634, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.977926+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:53:15.977926+0000', 'last_peered': '2024-09-16T04:53:15.977926+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:53:15.977926+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12531, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:28.587 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:53:28.588 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12928, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:26.579298+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:53:26.579298+0000', 'last_peered': '2024-09-16T04:53:26.579298+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:53:26.579298+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:28.588 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:53:28.588 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12305", 'reported_seq': 12413, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:15.975272+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:53:15.975272+0000', 'last_peered': '2024-09-16T04:53:15.975272+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:53:15.975272+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12303, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:28.588 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:53:28.981 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:28.981 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22873/112 objects degraded (20422.321%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":821,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2810978304,"bytes_avail":383736078336,"bytes_total":386547056640,"degraded_objects":22873,"degraded_total":112,"degraded_ratio":204.22321428571428,"recovering_objects_per_sec":127,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1273,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:53:28.994 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:53:28.994 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:29.284 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:29.285 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:29.297 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":445,"stamp":"2024-09-16T04:53:28.813037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12697,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745104,"kb_used_data":1916,"kb_used_omap":14,"kb_used_meta":2743089,"kb_avail":374742256,"statfs":{"total":386547056640,"available":383736070144,"internally_reserved":0,"allocated":1961984,"data_stored":1087850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2808923697},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1841,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1841,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001488"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12786,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.581158+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:26.581158+0000","last_peered":"2024-09-16T04:53:26.581158+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:26.581158+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12833,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.226197+0000","last_change":"2024-09-16T04:53:26.019219+0000","last_active":"2024-09-16T04:53:26.226197+0000","last_peered":"2024-09-16T04:53:26.226197+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:26.226197+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12782,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.979903+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:15.979903+0000","last_peered":"2024-09-16T04:53:15.979903+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:15.979903+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17227,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977622+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:15.977622+0000","last_peered":"2024-09-16T04:53:15.977622+0000","last_clean":"2024-09-16T04:53:15.977622+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:15.977622+0000","last_undegraded":"2024-09-16T04:53:15.977622+0000","last_fullsized":"2024-09-16T04:53:15.977622+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:17.471503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12888,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.579111+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:26.579111+0000","last_peered":"2024-09-16T04:53:26.579111+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:26.579111+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16795,"reported_epoch":41,"state":"peering","last_fresh":"2024-09-16T04:53:26.018418+0000","last_change":"2024-09-16T04:53:26.585410+0000","last_active":"2024-09-16T04:53:26.585410+0000","last_peered":"2024-09-16T04:53:26.018418+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:26.018418+0000","last_undegraded":"2024-09-16T04:53:26.018418+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17149,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:16.100110+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:16.100110+0000","last_peered":"2024-09-16T04:53:16.100110+0000","last_clean":"2024-09-16T04:53:16.100110+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:16.100110+0000","last_undegraded":"2024-09-16T04:53:16.100110+0000","last_fullsized":"2024-09-16T04:53:16.100110+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17359,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977987+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:15.977987+0000","last_peered":"2024-09-16T04:53:15.977987+0000","last_clean":"2024-09-16T04:53:15.977987+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:15.977987+0000","last_undegraded":"2024-09-16T04:53:15.977987+0000","last_fullsized":"2024-09-16T04:53:15.977987+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:17.471488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17526,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T04:53:26.579113+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:26.579113+0000","last_peered":"2024-09-16T04:53:26.579113+0000","last_clean":"2024-09-16T04:53:26.579113+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:26.579113+0000","last_undegraded":"2024-09-16T04:53:26.579113+0000","last_fullsized":"2024-09-16T04:53:26.579113+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12877,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.981122+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:15.981122+0000","last_peered":"2024-09-16T04:53:15.981122+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:15.981122+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12637,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.581918+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:26.581918+0000","last_peered":"2024-09-16T04:53:26.581918+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:26.581918+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12438,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.581763+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:26.581763+0000","last_peered":"2024-09-16T04:53:26.581763+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:26.581763+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12634,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.977926+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:15.977926+0000","last_peered":"2024-09-16T04:53:15.977926+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:15.977926+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12928,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.579298+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:26.579298+0000","last_peered":"2024-09-16T04:53:26.579298+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:26.579298+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14933,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:15.977993+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:15.977993+0000","last_peered":"2024-09-16T04:53:15.977993+0000","last_clean":"2024-09-16T04:53:15.977993+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:15.977993+0000","last_undegraded":"2024-09-16T04:53:15.977993+0000","last_fullsized":"2024-09-16T04:53:15.977993+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:20.507737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12413,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975272+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:15.975272+0000","last_peered":"2024-09-16T04:53:15.975272+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:15.975272+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12697,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722032,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649808,"statfs":{"total":96636764160,"available":95897403392,"internally_reserved":0,"allocated":356352,"data_stored":134253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731244,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640596,"statfs":{"total":96636764160,"available":95887970304,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569864,"kb_used_data":888,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93801976,"statfs":{"total":96636764160,"available":96053223424,"internally_reserved":0,"allocated":909312,"data_stored":691967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":582610639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721964,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649876,"statfs":{"total":96636764160,"available":95897473024,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:32.299 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:32.606 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:32.607 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:32.620 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-16T04:53:30.813496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12697,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771500,"kb_used_data":1940,"kb_used_omap":14,"kb_used_meta":2769457,"kb_avail":374715860,"statfs":{"total":386547056640,"available":383709040640,"internally_reserved":0,"allocated":1986560,"data_stored":1093368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2835924529},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":963,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001587"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12786,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.581158+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:26.581158+0000","last_peered":"2024-09-16T04:53:26.581158+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:26.581158+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12833,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.226197+0000","last_change":"2024-09-16T04:53:26.019219+0000","last_active":"2024-09-16T04:53:26.226197+0000","last_peered":"2024-09-16T04:53:26.226197+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:26.226197+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-2174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12786,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587400+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:27.587400+0000","last_peered":"2024-09-16T04:53:27.587400+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:27.587400+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17231,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.585962+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:27.585962+0000","last_peered":"2024-09-16T04:53:27.585962+0000","last_clean":"2024-09-16T04:53:27.585962+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:27.585962+0000","last_undegraded":"2024-09-16T04:53:27.585962+0000","last_fullsized":"2024-09-16T04:53:27.585962+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12888,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.579111+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:26.579111+0000","last_peered":"2024-09-16T04:53:26.579111+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:26.579111+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16795,"reported_epoch":41,"state":"peering","last_fresh":"2024-09-16T04:53:26.018418+0000","last_change":"2024-09-16T04:53:26.585410+0000","last_active":"2024-09-16T04:53:26.585410+0000","last_peered":"2024-09-16T04:53:26.018418+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.781816+0000","last_became_peered":"2024-09-16T04:41:27.781816+0000","last_unstale":"2024-09-16T04:53:26.018418+0000","last_undegraded":"2024-09-16T04:53:26.018418+0000","last_fullsized":"2024-09-16T04:41:27.483719+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17149,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:53:16.100110+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:16.100110+0000","last_peered":"2024-09-16T04:53:16.100110+0000","last_clean":"2024-09-16T04:53:16.100110+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:16.100110+0000","last_undegraded":"2024-09-16T04:53:16.100110+0000","last_fullsized":"2024-09-16T04:53:16.100110+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17363,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.585655+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:27.585655+0000","last_peered":"2024-09-16T04:53:27.585655+0000","last_clean":"2024-09-16T04:53:27.585655+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:27.585655+0000","last_undegraded":"2024-09-16T04:53:27.585655+0000","last_fullsized":"2024-09-16T04:53:27.585655+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17526,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T04:53:26.579113+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:26.579113+0000","last_peered":"2024-09-16T04:53:26.579113+0000","last_clean":"2024-09-16T04:53:26.579113+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:26.579113+0000","last_undegraded":"2024-09-16T04:53:26.579113+0000","last_fullsized":"2024-09-16T04:53:26.579113+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12881,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587402+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:27.587402+0000","last_peered":"2024-09-16T04:53:27.587402+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:27.587402+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12637,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.581918+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:26.581918+0000","last_peered":"2024-09-16T04:53:26.581918+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:26.581918+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12438,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.581763+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:26.581763+0000","last_peered":"2024-09-16T04:53:26.581763+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:26.581763+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12638,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586322+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:27.586322+0000","last_peered":"2024-09-16T04:53:27.586322+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:27.586322+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12928,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:26.579298+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:26.579298+0000","last_peered":"2024-09-16T04:53:26.579298+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:26.579298+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14937,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.586662+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:27.586662+0000","last_peered":"2024-09-16T04:53:27.586662+0000","last_clean":"2024-09-16T04:53:27.586662+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:27.586662+0000","last_undegraded":"2024-09-16T04:53:27.586662+0000","last_fullsized":"2024-09-16T04:53:27.586662+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:30.618767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12413,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:15.975272+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:15.975272+0000","last_peered":"2024-09-16T04:53:15.975272+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:15.975272+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12697,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":569344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722032,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649808,"statfs":{"total":96636764160,"available":95897403392,"internally_reserved":0,"allocated":356352,"data_stored":134253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731260,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640580,"statfs":{"total":96636764160,"available":95887953920,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596244,"kb_used_data":896,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93775596,"statfs":{"total":96636764160,"available":96026210304,"internally_reserved":0,"allocated":917504,"data_stored":690630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609611471},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721964,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649876,"statfs":{"total":96636764160,"available":95897473024,"internally_reserved":0,"allocated":348160,"data_stored":130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:32.621 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:32.916 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:32.917 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:32.929 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2024-09-16T04:53:32.813955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13451,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771520,"kb_used_data":1964,"kb_used_omap":14,"kb_used_meta":2769457,"kb_avail":374715840,"statfs":{"total":386547056640,"available":383709020160,"internally_reserved":0,"allocated":2011136,"data_stored":1103640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2835924529},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1717,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001664"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12788,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588564+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:27.588564+0000","last_peered":"2024-09-16T04:53:27.588564+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:27.588564+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":14345,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:31.231296+0000","last_change":"2024-09-16T04:53:26.019219+0000","last_active":"2024-09-16T04:53:31.231296+0000","last_peered":"2024-09-16T04:53:31.231296+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:31.231296+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":793,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12786,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587400+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:27.587400+0000","last_peered":"2024-09-16T04:53:27.587400+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:27.587400+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17231,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.585962+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:27.585962+0000","last_peered":"2024-09-16T04:53:27.585962+0000","last_clean":"2024-09-16T04:53:27.585962+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:27.585962+0000","last_undegraded":"2024-09-16T04:53:27.585962+0000","last_fullsized":"2024-09-16T04:53:27.585962+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12890,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586829+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:27.586829+0000","last_peered":"2024-09-16T04:53:27.586829+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:27.586829+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16805,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.692108+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:27.692108+0000","last_peered":"2024-09-16T04:53:27.692108+0000","last_clean":"2024-09-16T04:53:27.692108+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:27.692108+0000","last_undegraded":"2024-09-16T04:53:27.692108+0000","last_fullsized":"2024-09-16T04:53:27.692108+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17153,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.590015+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:27.590015+0000","last_peered":"2024-09-16T04:53:27.590015+0000","last_clean":"2024-09-16T04:53:27.590015+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:27.590015+0000","last_undegraded":"2024-09-16T04:53:27.590015+0000","last_fullsized":"2024-09-16T04:53:27.590015+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:32.136045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17363,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.585655+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:27.585655+0000","last_peered":"2024-09-16T04:53:27.585655+0000","last_clean":"2024-09-16T04:53:27.585655+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:27.585655+0000","last_undegraded":"2024-09-16T04:53:27.585655+0000","last_fullsized":"2024-09-16T04:53:27.585655+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17528,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.586012+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:27.586012+0000","last_peered":"2024-09-16T04:53:27.586012+0000","last_clean":"2024-09-16T04:53:27.586012+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:27.586012+0000","last_undegraded":"2024-09-16T04:53:27.586012+0000","last_fullsized":"2024-09-16T04:53:27.586012+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:31.782581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12881,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587402+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:27.587402+0000","last_peered":"2024-09-16T04:53:27.587402+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:27.587402+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12639,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586115+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:27.586115+0000","last_peered":"2024-09-16T04:53:27.586115+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:27.586115+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12440,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.589550+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:27.589550+0000","last_peered":"2024-09-16T04:53:27.589550+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:27.589550+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12638,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586322+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:27.586322+0000","last_peered":"2024-09-16T04:53:27.586322+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:27.586322+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12930,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:27.587609+0000","last_peered":"2024-09-16T04:53:27.587609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:27.587609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14937,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.586662+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:27.586662+0000","last_peered":"2024-09-16T04:53:27.586662+0000","last_clean":"2024-09-16T04:53:27.586662+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:27.586662+0000","last_undegraded":"2024-09-16T04:53:27.586662+0000","last_fullsized":"2024-09-16T04:53:27.586662+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:30.618767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12417,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588874+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:27.588874+0000","last_peered":"2024-09-16T04:53:27.588874+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:27.588874+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13451,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":569344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731260,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640580,"statfs":{"total":96636764160,"available":95887953920,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596244,"kb_used_data":896,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93775596,"statfs":{"total":96636764160,"available":96026210304,"internally_reserved":0,"allocated":917504,"data_stored":690630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609611471},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721976,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649864,"statfs":{"total":96636764160,"available":95897460736,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:32.930 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:53:32.930 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 12788, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.588564+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:53:27.588564+0000', 'last_peered': '2024-09-16T04:53:27.588564+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:53:27.588564+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:32.930 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:53:32.930 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12672", 'reported_seq': 14345, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:31.231296+0000', 'last_change': '2024-09-16T04:53:26.019219+0000', 'last_active': '2024-09-16T04:53:31.231296+0000', 'last_peered': '2024-09-16T04:53:31.231296+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:53:31.231296+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1422, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12670, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 793, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -1420}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:32.931 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:53:32.931 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12786, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.587400+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:53:27.587400+0000', 'last_peered': '2024-09-16T04:53:27.587400+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:53:27.587400+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:32.931 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:53:32.931 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12890, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.586829+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:53:27.586829+0000', 'last_peered': '2024-09-16T04:53:27.586829+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:53:27.586829+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:32.931 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:53:32.931 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12881, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.587402+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:53:27.587402+0000', 'last_peered': '2024-09-16T04:53:27.587402+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:53:27.587402+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:32.931 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:53:32.931 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12639, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.586115+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:53:27.586115+0000', 'last_peered': '2024-09-16T04:53:27.586115+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:53:27.586115+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:32.931 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:53:32.931 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12440, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.589550+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:53:27.589550+0000', 'last_peered': '2024-09-16T04:53:27.589550+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:53:27.589550+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:32.931 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:53:32.932 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12532", 'reported_seq': 12638, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.586322+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:53:27.586322+0000', 'last_peered': '2024-09-16T04:53:27.586322+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:53:27.586322+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12531, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:32.932 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:53:32.932 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12930, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.587609+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:53:27.587609+0000', 'last_peered': '2024-09-16T04:53:27.587609+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:53:27.587609+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:32.932 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:53:32.932 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12305", 'reported_seq': 12417, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.588874+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:53:27.588874+0000', 'last_peered': '2024-09-16T04:53:27.588874+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:53:27.588874+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12303, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:32.932 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:53:33.306 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:33.307 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21185/112 objects degraded (18915.179%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":825,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2838036480,"bytes_avail":383709020160,"bytes_total":386547056640,"degraded_objects":21185,"degraded_total":112,"degraded_ratio":189.15178571428572,"recovering_objects_per_sec":214,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1717,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:53:33.320 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:53:33.320 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:33.627 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:33.627 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:33.640 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2024-09-16T04:53:32.813955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13451,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771520,"kb_used_data":1964,"kb_used_omap":14,"kb_used_meta":2769457,"kb_avail":374715840,"statfs":{"total":386547056640,"available":383709020160,"internally_reserved":0,"allocated":2011136,"data_stored":1103640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2835924529},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1717,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001664"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12788,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588564+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:27.588564+0000","last_peered":"2024-09-16T04:53:27.588564+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:27.588564+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":14345,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:31.231296+0000","last_change":"2024-09-16T04:53:26.019219+0000","last_active":"2024-09-16T04:53:31.231296+0000","last_peered":"2024-09-16T04:53:31.231296+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:31.231296+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":793,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-1420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12786,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587400+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:27.587400+0000","last_peered":"2024-09-16T04:53:27.587400+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:27.587400+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17231,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.585962+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:27.585962+0000","last_peered":"2024-09-16T04:53:27.585962+0000","last_clean":"2024-09-16T04:53:27.585962+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:27.585962+0000","last_undegraded":"2024-09-16T04:53:27.585962+0000","last_fullsized":"2024-09-16T04:53:27.585962+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12890,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586829+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:27.586829+0000","last_peered":"2024-09-16T04:53:27.586829+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:27.586829+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16805,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.692108+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:27.692108+0000","last_peered":"2024-09-16T04:53:27.692108+0000","last_clean":"2024-09-16T04:53:27.692108+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:27.692108+0000","last_undegraded":"2024-09-16T04:53:27.692108+0000","last_fullsized":"2024-09-16T04:53:27.692108+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17153,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.590015+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:27.590015+0000","last_peered":"2024-09-16T04:53:27.590015+0000","last_clean":"2024-09-16T04:53:27.590015+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:27.590015+0000","last_undegraded":"2024-09-16T04:53:27.590015+0000","last_fullsized":"2024-09-16T04:53:27.590015+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:32.136045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17363,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.585655+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:27.585655+0000","last_peered":"2024-09-16T04:53:27.585655+0000","last_clean":"2024-09-16T04:53:27.585655+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:27.585655+0000","last_undegraded":"2024-09-16T04:53:27.585655+0000","last_fullsized":"2024-09-16T04:53:27.585655+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17528,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.586012+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:27.586012+0000","last_peered":"2024-09-16T04:53:27.586012+0000","last_clean":"2024-09-16T04:53:27.586012+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:27.586012+0000","last_undegraded":"2024-09-16T04:53:27.586012+0000","last_fullsized":"2024-09-16T04:53:27.586012+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:31.782581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12881,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587402+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:27.587402+0000","last_peered":"2024-09-16T04:53:27.587402+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:27.587402+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12639,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586115+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:27.586115+0000","last_peered":"2024-09-16T04:53:27.586115+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:27.586115+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12440,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.589550+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:27.589550+0000","last_peered":"2024-09-16T04:53:27.589550+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:27.589550+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12638,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586322+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:27.586322+0000","last_peered":"2024-09-16T04:53:27.586322+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:27.586322+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12930,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:27.587609+0000","last_peered":"2024-09-16T04:53:27.587609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:27.587609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14937,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.586662+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:27.586662+0000","last_peered":"2024-09-16T04:53:27.586662+0000","last_clean":"2024-09-16T04:53:27.586662+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:27.586662+0000","last_undegraded":"2024-09-16T04:53:27.586662+0000","last_fullsized":"2024-09-16T04:53:27.586662+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:30.618767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12417,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588874+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:27.588874+0000","last_peered":"2024-09-16T04:53:27.588874+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:27.588874+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13451,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":569344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731260,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640580,"statfs":{"total":96636764160,"available":95887953920,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596244,"kb_used_data":896,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93775596,"statfs":{"total":96636764160,"available":96026210304,"internally_reserved":0,"allocated":917504,"data_stored":690630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609611471},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721976,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649864,"statfs":{"total":96636764160,"available":95897460736,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:36.642 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:36.953 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:36.953 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:36.967 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-09-16T04:53:36.814650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14386,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793724,"kb_used_data":1964,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693636,"statfs":{"total":386547056640,"available":383686283264,"internally_reserved":0,"allocated":2011136,"data_stored":1103640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2623,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002043"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12788,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588564+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:27.588564+0000","last_peered":"2024-09-16T04:53:27.588564+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:27.588564+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":16215,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:36.234308+0000","last_change":"2024-09-16T04:53:26.019219+0000","last_active":"2024-09-16T04:53:36.234308+0000","last_peered":"2024-09-16T04:53:36.234308+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:36.234308+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12786,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587400+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:27.587400+0000","last_peered":"2024-09-16T04:53:27.587400+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:27.587400+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17231,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.585962+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:27.585962+0000","last_peered":"2024-09-16T04:53:27.585962+0000","last_clean":"2024-09-16T04:53:27.585962+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:27.585962+0000","last_undegraded":"2024-09-16T04:53:27.585962+0000","last_fullsized":"2024-09-16T04:53:27.585962+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12890,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586829+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:27.586829+0000","last_peered":"2024-09-16T04:53:27.586829+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:27.586829+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16805,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.692108+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:27.692108+0000","last_peered":"2024-09-16T04:53:27.692108+0000","last_clean":"2024-09-16T04:53:27.692108+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:27.692108+0000","last_undegraded":"2024-09-16T04:53:27.692108+0000","last_fullsized":"2024-09-16T04:53:27.692108+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17153,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.590015+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:27.590015+0000","last_peered":"2024-09-16T04:53:27.590015+0000","last_clean":"2024-09-16T04:53:27.590015+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:27.590015+0000","last_undegraded":"2024-09-16T04:53:27.590015+0000","last_fullsized":"2024-09-16T04:53:27.590015+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:32.136045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17363,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.585655+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:27.585655+0000","last_peered":"2024-09-16T04:53:27.585655+0000","last_clean":"2024-09-16T04:53:27.585655+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:27.585655+0000","last_undegraded":"2024-09-16T04:53:27.585655+0000","last_fullsized":"2024-09-16T04:53:27.585655+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17528,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.586012+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:27.586012+0000","last_peered":"2024-09-16T04:53:27.586012+0000","last_clean":"2024-09-16T04:53:27.586012+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:27.586012+0000","last_undegraded":"2024-09-16T04:53:27.586012+0000","last_fullsized":"2024-09-16T04:53:27.586012+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:31.782581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12881,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587402+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:27.587402+0000","last_peered":"2024-09-16T04:53:27.587402+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:27.587402+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12639,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586115+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:27.586115+0000","last_peered":"2024-09-16T04:53:27.586115+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:27.586115+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12440,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.589550+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:27.589550+0000","last_peered":"2024-09-16T04:53:27.589550+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:27.589550+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12639,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587727+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:27.587727+0000","last_peered":"2024-09-16T04:53:27.587727+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:27.587727+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12930,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:27.587609+0000","last_peered":"2024-09-16T04:53:27.587609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:27.587609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14937,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.586662+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:27.586662+0000","last_peered":"2024-09-16T04:53:27.586662+0000","last_clean":"2024-09-16T04:53:27.586662+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:27.586662+0000","last_undegraded":"2024-09-16T04:53:27.586662+0000","last_fullsized":"2024-09-16T04:53:27.586662+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:30.618767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12417,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588874+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:27.588874+0000","last_peered":"2024-09-16T04:53:27.588874+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:27.588874+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14386,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":569344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731260,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640580,"statfs":{"total":96636764160,"available":95887953920,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618448,"kb_used_data":896,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753392,"statfs":{"total":96636764160,"available":96003473408,"internally_reserved":0,"allocated":917504,"data_stored":690630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721976,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649864,"statfs":{"total":96636764160,"available":95897460736,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:36.968 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:37.265 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:37.266 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:37.278 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-09-16T04:53:36.814650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14386,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793724,"kb_used_data":1964,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693636,"statfs":{"total":386547056640,"available":383686283264,"internally_reserved":0,"allocated":2011136,"data_stored":1103640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2623,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002043"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12788,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588564+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:27.588564+0000","last_peered":"2024-09-16T04:53:27.588564+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:27.588564+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":16215,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:36.234308+0000","last_change":"2024-09-16T04:53:26.019219+0000","last_active":"2024-09-16T04:53:36.234308+0000","last_peered":"2024-09-16T04:53:36.234308+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:36.234308+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12786,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587400+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:27.587400+0000","last_peered":"2024-09-16T04:53:27.587400+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:27.587400+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17231,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.585962+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:27.585962+0000","last_peered":"2024-09-16T04:53:27.585962+0000","last_clean":"2024-09-16T04:53:27.585962+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:27.585962+0000","last_undegraded":"2024-09-16T04:53:27.585962+0000","last_fullsized":"2024-09-16T04:53:27.585962+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12890,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586829+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:27.586829+0000","last_peered":"2024-09-16T04:53:27.586829+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:27.586829+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16805,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.692108+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:27.692108+0000","last_peered":"2024-09-16T04:53:27.692108+0000","last_clean":"2024-09-16T04:53:27.692108+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:27.692108+0000","last_undegraded":"2024-09-16T04:53:27.692108+0000","last_fullsized":"2024-09-16T04:53:27.692108+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17153,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.590015+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:27.590015+0000","last_peered":"2024-09-16T04:53:27.590015+0000","last_clean":"2024-09-16T04:53:27.590015+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:27.590015+0000","last_undegraded":"2024-09-16T04:53:27.590015+0000","last_fullsized":"2024-09-16T04:53:27.590015+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:32.136045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17363,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.585655+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:27.585655+0000","last_peered":"2024-09-16T04:53:27.585655+0000","last_clean":"2024-09-16T04:53:27.585655+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:27.585655+0000","last_undegraded":"2024-09-16T04:53:27.585655+0000","last_fullsized":"2024-09-16T04:53:27.585655+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17528,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.586012+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:27.586012+0000","last_peered":"2024-09-16T04:53:27.586012+0000","last_clean":"2024-09-16T04:53:27.586012+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:27.586012+0000","last_undegraded":"2024-09-16T04:53:27.586012+0000","last_fullsized":"2024-09-16T04:53:27.586012+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:31.782581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12881,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587402+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:27.587402+0000","last_peered":"2024-09-16T04:53:27.587402+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:27.587402+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12639,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586115+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:27.586115+0000","last_peered":"2024-09-16T04:53:27.586115+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:27.586115+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12440,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.589550+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:27.589550+0000","last_peered":"2024-09-16T04:53:27.589550+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:27.589550+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12639,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587727+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:27.587727+0000","last_peered":"2024-09-16T04:53:27.587727+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:27.587727+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12930,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:27.587609+0000","last_peered":"2024-09-16T04:53:27.587609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:27.587609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14937,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.586662+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:27.586662+0000","last_peered":"2024-09-16T04:53:27.586662+0000","last_clean":"2024-09-16T04:53:27.586662+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:27.586662+0000","last_undegraded":"2024-09-16T04:53:27.586662+0000","last_fullsized":"2024-09-16T04:53:27.586662+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:30.618767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12417,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588874+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:27.588874+0000","last_peered":"2024-09-16T04:53:27.588874+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:27.588874+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14386,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":569344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731260,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640580,"statfs":{"total":96636764160,"available":95887953920,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618448,"kb_used_data":896,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753392,"statfs":{"total":96636764160,"available":96003473408,"internally_reserved":0,"allocated":917504,"data_stored":690630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721976,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649864,"statfs":{"total":96636764160,"available":95897460736,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:37.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:53:37.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 12788, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.588564+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:53:27.588564+0000', 'last_peered': '2024-09-16T04:53:27.588564+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:53:27.588564+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:37.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:53:37.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12672", 'reported_seq': 16215, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:36.234308+0000', 'last_change': '2024-09-16T04:53:26.019219+0000', 'last_active': '2024-09-16T04:53:36.234308+0000', 'last_peered': '2024-09-16T04:53:36.234308+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:53:36.234308+0000', 'last_undegraded': '2024-09-16T04:41:27.544232+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 487, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12670, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1728, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -485}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:37.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:53:37.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12786, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.587400+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:53:27.587400+0000', 'last_peered': '2024-09-16T04:53:27.587400+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:53:27.587400+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:37.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:53:37.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12890, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.586829+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:53:27.586829+0000', 'last_peered': '2024-09-16T04:53:27.586829+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:53:27.586829+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:37.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:53:37.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12881, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.587402+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:53:27.587402+0000', 'last_peered': '2024-09-16T04:53:27.587402+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:53:27.587402+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:37.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:53:37.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12639, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.586115+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:53:27.586115+0000', 'last_peered': '2024-09-16T04:53:27.586115+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:53:27.586115+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:37.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:53:37.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12440, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.589550+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:53:27.589550+0000', 'last_peered': '2024-09-16T04:53:27.589550+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:53:27.589550+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:37.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:53:37.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12532", 'reported_seq': 12639, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.587727+0000', 'last_change': '2024-09-16T04:41:27.783941+0000', 'last_active': '2024-09-16T04:53:27.587727+0000', 'last_peered': '2024-09-16T04:53:27.587727+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:53:27.587727+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2222, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12531, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:37.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:53:37.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12930, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.587609+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:53:27.587609+0000', 'last_peered': '2024-09-16T04:53:27.587609+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:53:27.587609+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:37.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:53:37.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12305", 'reported_seq': 12417, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.588874+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:53:27.588874+0000', 'last_peered': '2024-09-16T04:53:27.588874+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:53:27.588874+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12303, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:37.281 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:53:37.652 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:37.652 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20250/112 objects degraded (18080.357%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":829,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2860773376,"bytes_avail":383686283264,"bytes_total":386547056640,"degraded_objects":20250,"degraded_total":112,"degraded_ratio":180.80357142857142,"recovering_objects_per_sec":262,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2623,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:53:37.665 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:53:37.666 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:37.959 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:37.960 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:37.972 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-09-16T04:53:36.814650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14386,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793724,"kb_used_data":1964,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693636,"statfs":{"total":386547056640,"available":383686283264,"internally_reserved":0,"allocated":2011136,"data_stored":1103640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2623,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002043"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12788,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588564+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:27.588564+0000","last_peered":"2024-09-16T04:53:27.588564+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:27.588564+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":16215,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:36.234308+0000","last_change":"2024-09-16T04:53:26.019219+0000","last_active":"2024-09-16T04:53:36.234308+0000","last_peered":"2024-09-16T04:53:36.234308+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:36.234308+0000","last_undegraded":"2024-09-16T04:41:27.544232+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12786,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587400+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:27.587400+0000","last_peered":"2024-09-16T04:53:27.587400+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:27.587400+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17231,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.585962+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:27.585962+0000","last_peered":"2024-09-16T04:53:27.585962+0000","last_clean":"2024-09-16T04:53:27.585962+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:27.585962+0000","last_undegraded":"2024-09-16T04:53:27.585962+0000","last_fullsized":"2024-09-16T04:53:27.585962+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12890,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586829+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:27.586829+0000","last_peered":"2024-09-16T04:53:27.586829+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:27.586829+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16805,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.692108+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:27.692108+0000","last_peered":"2024-09-16T04:53:27.692108+0000","last_clean":"2024-09-16T04:53:27.692108+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:27.692108+0000","last_undegraded":"2024-09-16T04:53:27.692108+0000","last_fullsized":"2024-09-16T04:53:27.692108+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17153,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.590015+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:27.590015+0000","last_peered":"2024-09-16T04:53:27.590015+0000","last_clean":"2024-09-16T04:53:27.590015+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:27.590015+0000","last_undegraded":"2024-09-16T04:53:27.590015+0000","last_fullsized":"2024-09-16T04:53:27.590015+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:32.136045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17363,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.585655+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:27.585655+0000","last_peered":"2024-09-16T04:53:27.585655+0000","last_clean":"2024-09-16T04:53:27.585655+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:27.585655+0000","last_undegraded":"2024-09-16T04:53:27.585655+0000","last_fullsized":"2024-09-16T04:53:27.585655+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17528,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.586012+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:27.586012+0000","last_peered":"2024-09-16T04:53:27.586012+0000","last_clean":"2024-09-16T04:53:27.586012+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:27.586012+0000","last_undegraded":"2024-09-16T04:53:27.586012+0000","last_fullsized":"2024-09-16T04:53:27.586012+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:31.782581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12881,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587402+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:27.587402+0000","last_peered":"2024-09-16T04:53:27.587402+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:27.587402+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12639,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586115+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:27.586115+0000","last_peered":"2024-09-16T04:53:27.586115+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:27.586115+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12440,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.589550+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:27.589550+0000","last_peered":"2024-09-16T04:53:27.589550+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:27.589550+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12639,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587727+0000","last_change":"2024-09-16T04:41:27.783941+0000","last_active":"2024-09-16T04:53:27.587727+0000","last_peered":"2024-09-16T04:53:27.587727+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:27.587727+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12930,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:27.587609+0000","last_peered":"2024-09-16T04:53:27.587609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:27.587609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14937,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.586662+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:27.586662+0000","last_peered":"2024-09-16T04:53:27.586662+0000","last_clean":"2024-09-16T04:53:27.586662+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:27.586662+0000","last_undegraded":"2024-09-16T04:53:27.586662+0000","last_fullsized":"2024-09-16T04:53:27.586662+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:30.618767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12417,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588874+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:27.588874+0000","last_peered":"2024-09-16T04:53:27.588874+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:27.588874+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14386,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":569344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731260,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640580,"statfs":{"total":96636764160,"available":95887953920,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618448,"kb_used_data":896,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753392,"statfs":{"total":96636764160,"available":96003473408,"internally_reserved":0,"allocated":917504,"data_stored":690630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721976,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649864,"statfs":{"total":96636764160,"available":95897460736,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:40.974 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:41.285 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:41.286 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:41.298 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-09-16T04:53:40.815380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15118,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789564,"kb_used_data":1968,"kb_used_omap":14,"kb_used_meta":2787505,"kb_avail":374697796,"statfs":{"total":386547056640,"available":383690543104,"internally_reserved":0,"allocated":2015232,"data_stored":1102945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2854405681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1667,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1667,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001424"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12788,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588564+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:27.588564+0000","last_peered":"2024-09-16T04:53:27.588564+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:27.588564+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12770,"reported_epoch":44,"state":"peering","last_fresh":"2024-09-16T04:53:38.388145+0000","last_change":"2024-09-16T04:53:39.414279+0000","last_active":"2024-09-16T04:53:39.414279+0000","last_peered":"2024-09-16T04:53:38.388145+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:38.388145+0000","last_undegraded":"2024-09-16T04:53:38.388145+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12788,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:39.045800+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:39.045800+0000","last_peered":"2024-09-16T04:53:39.045800+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:39.045800+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17233,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:53:39.041747+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:39.041747+0000","last_peered":"2024-09-16T04:53:39.041747+0000","last_clean":"2024-09-16T04:53:39.041747+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:39.041747+0000","last_undegraded":"2024-09-16T04:53:39.041747+0000","last_fullsized":"2024-09-16T04:53:39.041747+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12890,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586829+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:27.586829+0000","last_peered":"2024-09-16T04:53:27.586829+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:27.586829+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16805,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.692108+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:27.692108+0000","last_peered":"2024-09-16T04:53:27.692108+0000","last_clean":"2024-09-16T04:53:27.692108+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:27.692108+0000","last_undegraded":"2024-09-16T04:53:27.692108+0000","last_fullsized":"2024-09-16T04:53:27.692108+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17153,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.590015+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:27.590015+0000","last_peered":"2024-09-16T04:53:27.590015+0000","last_clean":"2024-09-16T04:53:27.590015+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:27.590015+0000","last_undegraded":"2024-09-16T04:53:27.590015+0000","last_fullsized":"2024-09-16T04:53:27.590015+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:32.136045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17365,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:53:39.041661+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:39.041661+0000","last_peered":"2024-09-16T04:53:39.041661+0000","last_clean":"2024-09-16T04:53:39.041661+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:39.041661+0000","last_undegraded":"2024-09-16T04:53:39.041661+0000","last_fullsized":"2024-09-16T04:53:39.041661+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17528,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.586012+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:27.586012+0000","last_peered":"2024-09-16T04:53:27.586012+0000","last_clean":"2024-09-16T04:53:27.586012+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:27.586012+0000","last_undegraded":"2024-09-16T04:53:27.586012+0000","last_fullsized":"2024-09-16T04:53:27.586012+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:31.782581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12883,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:39.046053+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:39.046053+0000","last_peered":"2024-09-16T04:53:39.046053+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:39.046053+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12639,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586115+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:27.586115+0000","last_peered":"2024-09-16T04:53:27.586115+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:27.586115+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12440,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.589550+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:27.589550+0000","last_peered":"2024-09-16T04:53:27.589550+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:27.589550+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":13132,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.008688+0000","last_change":"2024-09-16T04:53:38.388749+0000","last_active":"2024-09-16T04:53:40.008688+0000","last_peered":"2024-09-16T04:53:40.008688+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:40.008688+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1976}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12930,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:27.587609+0000","last_peered":"2024-09-16T04:53:27.587609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:27.587609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14939,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:53:39.041956+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:39.041956+0000","last_peered":"2024-09-16T04:53:39.041956+0000","last_clean":"2024-09-16T04:53:39.041956+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:39.041956+0000","last_undegraded":"2024-09-16T04:53:39.041956+0000","last_fullsized":"2024-09-16T04:53:39.041956+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.632079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12417,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588874+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:27.588874+0000","last_peered":"2024-09-16T04:53:27.588874+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:27.588874+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15118,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731264,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640576,"statfs":{"total":96636764160,"available":95887949824,"internally_reserved":0,"allocated":372736,"data_stored":141071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614284,"kb_used_data":892,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757556,"statfs":{"total":96636764160,"available":96007737344,"internally_reserved":0,"allocated":913408,"data_stored":686534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628092623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721976,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649864,"statfs":{"total":96636764160,"available":95897460736,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":552960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:41.299 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:41.626 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:41.627 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:41.640 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-09-16T04:53:40.815380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15118,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789564,"kb_used_data":1968,"kb_used_omap":14,"kb_used_meta":2787505,"kb_avail":374697796,"statfs":{"total":386547056640,"available":383690543104,"internally_reserved":0,"allocated":2015232,"data_stored":1102945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2854405681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1667,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1667,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001424"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12788,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588564+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:27.588564+0000","last_peered":"2024-09-16T04:53:27.588564+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:27.588564+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12770,"reported_epoch":44,"state":"peering","last_fresh":"2024-09-16T04:53:38.388145+0000","last_change":"2024-09-16T04:53:39.414279+0000","last_active":"2024-09-16T04:53:39.414279+0000","last_peered":"2024-09-16T04:53:38.388145+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:38.388145+0000","last_undegraded":"2024-09-16T04:53:38.388145+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12788,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:39.045800+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:39.045800+0000","last_peered":"2024-09-16T04:53:39.045800+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:39.045800+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17233,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:53:39.041747+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:39.041747+0000","last_peered":"2024-09-16T04:53:39.041747+0000","last_clean":"2024-09-16T04:53:39.041747+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:39.041747+0000","last_undegraded":"2024-09-16T04:53:39.041747+0000","last_fullsized":"2024-09-16T04:53:39.041747+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12890,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586829+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:27.586829+0000","last_peered":"2024-09-16T04:53:27.586829+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:27.586829+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16805,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.692108+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:27.692108+0000","last_peered":"2024-09-16T04:53:27.692108+0000","last_clean":"2024-09-16T04:53:27.692108+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:27.692108+0000","last_undegraded":"2024-09-16T04:53:27.692108+0000","last_fullsized":"2024-09-16T04:53:27.692108+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17153,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.590015+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:27.590015+0000","last_peered":"2024-09-16T04:53:27.590015+0000","last_clean":"2024-09-16T04:53:27.590015+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:27.590015+0000","last_undegraded":"2024-09-16T04:53:27.590015+0000","last_fullsized":"2024-09-16T04:53:27.590015+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:32.136045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17365,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:53:39.041661+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:39.041661+0000","last_peered":"2024-09-16T04:53:39.041661+0000","last_clean":"2024-09-16T04:53:39.041661+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:39.041661+0000","last_undegraded":"2024-09-16T04:53:39.041661+0000","last_fullsized":"2024-09-16T04:53:39.041661+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17528,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.586012+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:27.586012+0000","last_peered":"2024-09-16T04:53:27.586012+0000","last_clean":"2024-09-16T04:53:27.586012+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:27.586012+0000","last_undegraded":"2024-09-16T04:53:27.586012+0000","last_fullsized":"2024-09-16T04:53:27.586012+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:31.782581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12883,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:39.046053+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:39.046053+0000","last_peered":"2024-09-16T04:53:39.046053+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:39.046053+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12639,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586115+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:27.586115+0000","last_peered":"2024-09-16T04:53:27.586115+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:27.586115+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12440,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.589550+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:27.589550+0000","last_peered":"2024-09-16T04:53:27.589550+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:27.589550+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":13132,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.008688+0000","last_change":"2024-09-16T04:53:38.388749+0000","last_active":"2024-09-16T04:53:40.008688+0000","last_peered":"2024-09-16T04:53:40.008688+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:40.008688+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1976}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12930,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:27.587609+0000","last_peered":"2024-09-16T04:53:27.587609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:27.587609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14939,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:53:39.041956+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:39.041956+0000","last_peered":"2024-09-16T04:53:39.041956+0000","last_clean":"2024-09-16T04:53:39.041956+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:39.041956+0000","last_undegraded":"2024-09-16T04:53:39.041956+0000","last_fullsized":"2024-09-16T04:53:39.041956+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.632079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12417,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588874+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:27.588874+0000","last_peered":"2024-09-16T04:53:27.588874+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:27.588874+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15118,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731264,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640576,"statfs":{"total":96636764160,"available":95887949824,"internally_reserved":0,"allocated":372736,"data_stored":141071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614284,"kb_used_data":892,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757556,"statfs":{"total":96636764160,"available":96007737344,"internally_reserved":0,"allocated":913408,"data_stored":686534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628092623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721976,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649864,"statfs":{"total":96636764160,"available":95897460736,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":552960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:41.641 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:53:41.641 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 12788, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.588564+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:53:27.588564+0000', 'last_peered': '2024-09-16T04:53:27.588564+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:53:27.588564+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:41.641 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-16T04:53:41.641 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'12672", 'reported_seq': 12770, 'reported_epoch': 44, 'state': 'peering', 'last_fresh': '2024-09-16T04:53:38.388145+0000', 'last_change': '2024-09-16T04:53:39.414279+0000', 'last_active': '2024-09-16T04:53:39.414279+0000', 'last_peered': '2024-09-16T04:53:38.388145+0000', 'last_clean': '2024-09-16T04:40:22.866482+0000', 'last_became_active': '2024-09-16T04:41:27.855321+0000', 'last_became_peered': '2024-09-16T04:41:27.855321+0000', 'last_unstale': '2024-09-16T04:53:38.388145+0000', 'last_undegraded': '2024-09-16T04:53:38.388145+0000', 'last_fullsized': '2024-09-16T04:41:27.483394+0000', 'mapping_epoch': 44, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12670, 'num_write_kb': 50692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2215, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [1, 0, 3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T04:53:41.641 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:53:41.641 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12788, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:39.045800+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:53:39.045800+0000', 'last_peered': '2024-09-16T04:53:39.045800+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:53:39.045800+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:41.641 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:53:41.641 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12890, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.586829+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:53:27.586829+0000', 'last_peered': '2024-09-16T04:53:27.586829+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:53:27.586829+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:41.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:53:41.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12883, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:39.046053+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:53:39.046053+0000', 'last_peered': '2024-09-16T04:53:39.046053+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:53:39.046053+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:41.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:53:41.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12639, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.586115+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:53:27.586115+0000', 'last_peered': '2024-09-16T04:53:27.586115+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:53:27.586115+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:41.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:53:41.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12440, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.589550+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:53:27.589550+0000', 'last_peered': '2024-09-16T04:53:27.589550+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:53:27.589550+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:41.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:53:41.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12532", 'reported_seq': 13132, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.008688+0000', 'last_change': '2024-09-16T04:53:38.388749+0000', 'last_active': '2024-09-16T04:53:40.008688+0000', 'last_peered': '2024-09-16T04:53:40.008688+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:53:40.008688+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12531, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 250, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1976}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:41.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:53:41.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12930, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.587609+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:53:27.587609+0000', 'last_peered': '2024-09-16T04:53:27.587609+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:53:27.587609+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:41.643 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:53:41.643 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12305", 'reported_seq': 12417, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:27.588874+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:53:27.588874+0000', 'last_peered': '2024-09-16T04:53:27.588874+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:53:27.588874+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12303, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:41.643 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:53:42.012 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:42.012 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19518/112 objects degraded (17426.786%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":834,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"peering","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2856513536,"bytes_avail":383690543104,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":19518,"degraded_total":112,"degraded_ratio":174.26785714285714,"recovering_objects_per_sec":208,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1667,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:53:42.026 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:53:42.026 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:42.337 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:42.338 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:42.352 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-09-16T04:53:40.815380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15118,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789564,"kb_used_data":1968,"kb_used_omap":14,"kb_used_meta":2787505,"kb_avail":374697796,"statfs":{"total":386547056640,"available":383690543104,"internally_reserved":0,"allocated":2015232,"data_stored":1102945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2854405681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1667,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1667,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001424"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12788,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588564+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:27.588564+0000","last_peered":"2024-09-16T04:53:27.588564+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:27.588564+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12770,"reported_epoch":44,"state":"peering","last_fresh":"2024-09-16T04:53:38.388145+0000","last_change":"2024-09-16T04:53:39.414279+0000","last_active":"2024-09-16T04:53:39.414279+0000","last_peered":"2024-09-16T04:53:38.388145+0000","last_clean":"2024-09-16T04:40:22.866482+0000","last_became_active":"2024-09-16T04:41:27.855321+0000","last_became_peered":"2024-09-16T04:41:27.855321+0000","last_unstale":"2024-09-16T04:53:38.388145+0000","last_undegraded":"2024-09-16T04:53:38.388145+0000","last_fullsized":"2024-09-16T04:41:27.483394+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12788,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:39.045800+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:39.045800+0000","last_peered":"2024-09-16T04:53:39.045800+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:39.045800+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17233,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:53:39.041747+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:39.041747+0000","last_peered":"2024-09-16T04:53:39.041747+0000","last_clean":"2024-09-16T04:53:39.041747+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:39.041747+0000","last_undegraded":"2024-09-16T04:53:39.041747+0000","last_fullsized":"2024-09-16T04:53:39.041747+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12890,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586829+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:27.586829+0000","last_peered":"2024-09-16T04:53:27.586829+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:27.586829+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16805,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.692108+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:27.692108+0000","last_peered":"2024-09-16T04:53:27.692108+0000","last_clean":"2024-09-16T04:53:27.692108+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:27.692108+0000","last_undegraded":"2024-09-16T04:53:27.692108+0000","last_fullsized":"2024-09-16T04:53:27.692108+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17153,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.590015+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:27.590015+0000","last_peered":"2024-09-16T04:53:27.590015+0000","last_clean":"2024-09-16T04:53:27.590015+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:27.590015+0000","last_undegraded":"2024-09-16T04:53:27.590015+0000","last_fullsized":"2024-09-16T04:53:27.590015+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:32.136045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17365,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:53:39.041661+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:39.041661+0000","last_peered":"2024-09-16T04:53:39.041661+0000","last_clean":"2024-09-16T04:53:39.041661+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:39.041661+0000","last_undegraded":"2024-09-16T04:53:39.041661+0000","last_fullsized":"2024-09-16T04:53:39.041661+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17528,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:53:27.586012+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:27.586012+0000","last_peered":"2024-09-16T04:53:27.586012+0000","last_clean":"2024-09-16T04:53:27.586012+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:27.586012+0000","last_undegraded":"2024-09-16T04:53:27.586012+0000","last_fullsized":"2024-09-16T04:53:27.586012+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:31.782581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12883,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:39.046053+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:39.046053+0000","last_peered":"2024-09-16T04:53:39.046053+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:39.046053+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12639,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.586115+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:27.586115+0000","last_peered":"2024-09-16T04:53:27.586115+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:27.586115+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12440,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.589550+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:27.589550+0000","last_peered":"2024-09-16T04:53:27.589550+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:27.589550+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":13132,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.008688+0000","last_change":"2024-09-16T04:53:38.388749+0000","last_active":"2024-09-16T04:53:40.008688+0000","last_peered":"2024-09-16T04:53:40.008688+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:40.008688+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1976}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12930,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.587609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:27.587609+0000","last_peered":"2024-09-16T04:53:27.587609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:27.587609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14939,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:53:39.041956+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:39.041956+0000","last_peered":"2024-09-16T04:53:39.041956+0000","last_clean":"2024-09-16T04:53:39.041956+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:39.041956+0000","last_undegraded":"2024-09-16T04:53:39.041956+0000","last_fullsized":"2024-09-16T04:53:39.041956+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.632079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12417,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:27.588874+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:27.588874+0000","last_peered":"2024-09-16T04:53:27.588874+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:27.588874+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15118,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731264,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640576,"statfs":{"total":96636764160,"available":95887949824,"internally_reserved":0,"allocated":372736,"data_stored":141071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614284,"kb_used_data":892,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757556,"statfs":{"total":96636764160,"available":96007737344,"internally_reserved":0,"allocated":913408,"data_stored":686534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628092623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721976,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649864,"statfs":{"total":96636764160,"available":95897460736,"internally_reserved":0,"allocated":364544,"data_stored":137670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":552960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:45.354 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:45.647 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:45.648 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:45.660 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-16T04:53:44.816092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15118,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793776,"kb_used_data":2016,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693584,"statfs":{"total":386547056640,"available":383686230016,"internally_reserved":0,"allocated":2064384,"data_stored":1123288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":732,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001441"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12792,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.054626+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:40.054626+0000","last_peered":"2024-09-16T04:53:40.054626+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:40.054626+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12780,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.146094+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:53:40.146094+0000","last_peered":"2024-09-16T04:53:40.146094+0000","last_clean":"2024-09-16T04:53:40.146094+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:53:40.146094+0000","last_undegraded":"2024-09-16T04:53:40.146094+0000","last_fullsized":"2024-09-16T04:53:40.146094+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12788,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:39.045800+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:39.045800+0000","last_peered":"2024-09-16T04:53:39.045800+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:39.045800+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17235,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056241+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:40.056241+0000","last_peered":"2024-09-16T04:53:40.056241+0000","last_clean":"2024-09-16T04:53:40.056241+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:40.056241+0000","last_undegraded":"2024-09-16T04:53:40.056241+0000","last_fullsized":"2024-09-16T04:53:40.056241+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12894,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053101+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:40.053101+0000","last_peered":"2024-09-16T04:53:40.053101+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:40.053101+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16809,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.053188+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:40.053188+0000","last_peered":"2024-09-16T04:53:40.053188+0000","last_clean":"2024-09-16T04:53:40.053188+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:40.053188+0000","last_undegraded":"2024-09-16T04:53:40.053188+0000","last_fullsized":"2024-09-16T04:53:40.053188+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17157,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.054589+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:40.054589+0000","last_peered":"2024-09-16T04:53:40.054589+0000","last_clean":"2024-09-16T04:53:40.054589+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:40.054589+0000","last_undegraded":"2024-09-16T04:53:40.054589+0000","last_fullsized":"2024-09-16T04:53:40.054589+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:42.237116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17367,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056393+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:40.056393+0000","last_peered":"2024-09-16T04:53:40.056393+0000","last_clean":"2024-09-16T04:53:40.056393+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:40.056393+0000","last_undegraded":"2024-09-16T04:53:40.056393+0000","last_fullsized":"2024-09-16T04:53:40.056393+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17532,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.053094+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:40.053094+0000","last_peered":"2024-09-16T04:53:40.053094+0000","last_clean":"2024-09-16T04:53:40.053094+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:40.053094+0000","last_undegraded":"2024-09-16T04:53:40.053094+0000","last_fullsized":"2024-09-16T04:53:40.053094+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.940245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12883,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:39.046053+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:39.046053+0000","last_peered":"2024-09-16T04:53:39.046053+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:39.046053+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12643,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053224+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:40.053224+0000","last_peered":"2024-09-16T04:53:40.053224+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:40.053224+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12444,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.055310+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:40.055310+0000","last_peered":"2024-09-16T04:53:40.055310+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:40.055310+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":13132,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.008688+0000","last_change":"2024-09-16T04:53:38.388749+0000","last_active":"2024-09-16T04:53:40.008688+0000","last_peered":"2024-09-16T04:53:40.008688+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:40.008688+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1976}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12934,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053345+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:40.053345+0000","last_peered":"2024-09-16T04:53:40.053345+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:40.053345+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14941,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056380+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:40.056380+0000","last_peered":"2024-09-16T04:53:40.056380+0000","last_clean":"2024-09-16T04:53:40.056380+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:40.056380+0000","last_undegraded":"2024-09-16T04:53:40.056380+0000","last_fullsized":"2024-09-16T04:53:40.056380+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.632079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12421,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.055194+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:40.055194+0000","last_peered":"2024-09-16T04:53:40.055194+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:40.055194+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15118,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722056,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649784,"statfs":{"total":96636764160,"available":95897378816,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731264,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640576,"statfs":{"total":96636764160,"available":95887949824,"internally_reserved":0,"allocated":372736,"data_stored":141071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618460,"kb_used_data":908,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753380,"statfs":{"total":96636764160,"available":96003461120,"internally_reserved":0,"allocated":929792,"data_stored":693315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":552960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:45.661 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:45.974 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:45.974 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:45.986 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-16T04:53:44.816092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15118,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793776,"kb_used_data":2016,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693584,"statfs":{"total":386547056640,"available":383686230016,"internally_reserved":0,"allocated":2064384,"data_stored":1123288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":732,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001441"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12792,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.054626+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:40.054626+0000","last_peered":"2024-09-16T04:53:40.054626+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:40.054626+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12780,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.146094+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:53:40.146094+0000","last_peered":"2024-09-16T04:53:40.146094+0000","last_clean":"2024-09-16T04:53:40.146094+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:53:40.146094+0000","last_undegraded":"2024-09-16T04:53:40.146094+0000","last_fullsized":"2024-09-16T04:53:40.146094+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12788,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:39.045800+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:39.045800+0000","last_peered":"2024-09-16T04:53:39.045800+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:39.045800+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17235,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056241+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:40.056241+0000","last_peered":"2024-09-16T04:53:40.056241+0000","last_clean":"2024-09-16T04:53:40.056241+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:40.056241+0000","last_undegraded":"2024-09-16T04:53:40.056241+0000","last_fullsized":"2024-09-16T04:53:40.056241+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12894,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053101+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:40.053101+0000","last_peered":"2024-09-16T04:53:40.053101+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:40.053101+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16809,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.053188+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:40.053188+0000","last_peered":"2024-09-16T04:53:40.053188+0000","last_clean":"2024-09-16T04:53:40.053188+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:40.053188+0000","last_undegraded":"2024-09-16T04:53:40.053188+0000","last_fullsized":"2024-09-16T04:53:40.053188+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17157,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.054589+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:40.054589+0000","last_peered":"2024-09-16T04:53:40.054589+0000","last_clean":"2024-09-16T04:53:40.054589+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:40.054589+0000","last_undegraded":"2024-09-16T04:53:40.054589+0000","last_fullsized":"2024-09-16T04:53:40.054589+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:42.237116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17367,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056393+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:40.056393+0000","last_peered":"2024-09-16T04:53:40.056393+0000","last_clean":"2024-09-16T04:53:40.056393+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:40.056393+0000","last_undegraded":"2024-09-16T04:53:40.056393+0000","last_fullsized":"2024-09-16T04:53:40.056393+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17532,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.053094+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:40.053094+0000","last_peered":"2024-09-16T04:53:40.053094+0000","last_clean":"2024-09-16T04:53:40.053094+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:40.053094+0000","last_undegraded":"2024-09-16T04:53:40.053094+0000","last_fullsized":"2024-09-16T04:53:40.053094+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.940245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12883,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:39.046053+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:39.046053+0000","last_peered":"2024-09-16T04:53:39.046053+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:39.046053+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12643,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053224+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:40.053224+0000","last_peered":"2024-09-16T04:53:40.053224+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:40.053224+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12444,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.055310+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:40.055310+0000","last_peered":"2024-09-16T04:53:40.055310+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:40.055310+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":13132,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.008688+0000","last_change":"2024-09-16T04:53:38.388749+0000","last_active":"2024-09-16T04:53:40.008688+0000","last_peered":"2024-09-16T04:53:40.008688+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:40.008688+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1976}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12934,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053345+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:40.053345+0000","last_peered":"2024-09-16T04:53:40.053345+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:40.053345+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14941,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056380+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:40.056380+0000","last_peered":"2024-09-16T04:53:40.056380+0000","last_clean":"2024-09-16T04:53:40.056380+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:40.056380+0000","last_undegraded":"2024-09-16T04:53:40.056380+0000","last_fullsized":"2024-09-16T04:53:40.056380+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.632079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12421,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.055194+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:40.055194+0000","last_peered":"2024-09-16T04:53:40.055194+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:40.055194+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15118,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722056,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649784,"statfs":{"total":96636764160,"available":95897378816,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731264,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640576,"statfs":{"total":96636764160,"available":95887949824,"internally_reserved":0,"allocated":372736,"data_stored":141071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618460,"kb_used_data":908,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753380,"statfs":{"total":96636764160,"available":96003461120,"internally_reserved":0,"allocated":929792,"data_stored":693315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":552960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:45.987 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:53:45.987 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 12792, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.054626+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:53:40.054626+0000', 'last_peered': '2024-09-16T04:53:40.054626+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:53:40.054626+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:45.987 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:53:45.987 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12788, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:39.045800+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:53:39.045800+0000', 'last_peered': '2024-09-16T04:53:39.045800+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:53:39.045800+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:45.988 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:53:45.988 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12894, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.053101+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:53:40.053101+0000', 'last_peered': '2024-09-16T04:53:40.053101+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:53:40.053101+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:45.988 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:53:45.988 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12883, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:39.046053+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:53:39.046053+0000', 'last_peered': '2024-09-16T04:53:39.046053+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:53:39.046053+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:45.988 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:53:45.988 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12643, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.053224+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:53:40.053224+0000', 'last_peered': '2024-09-16T04:53:40.053224+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:53:40.053224+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:45.988 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:53:45.988 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12444, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.055310+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:53:40.055310+0000', 'last_peered': '2024-09-16T04:53:40.055310+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:53:40.055310+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:45.988 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:53:45.988 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12532", 'reported_seq': 13132, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.008688+0000', 'last_change': '2024-09-16T04:53:38.388749+0000', 'last_active': '2024-09-16T04:53:40.008688+0000', 'last_peered': '2024-09-16T04:53:40.008688+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:53:40.008688+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1977, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12531, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 250, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1976}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:45.989 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:53:45.989 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12934, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.053345+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:53:40.053345+0000', 'last_peered': '2024-09-16T04:53:40.053345+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:53:40.053345+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:45.989 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:53:45.989 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12305", 'reported_seq': 12421, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.055194+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:53:40.055194+0000', 'last_peered': '2024-09-16T04:53:40.055194+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:53:40.055194+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12303, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:45.989 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:53:46.401 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:46.401 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19518/112 objects degraded (17426.786%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":838,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2860826624,"bytes_avail":383686230016,"bytes_total":386547056640,"degraded_objects":19518,"degraded_total":112,"degraded_ratio":174.26785714285714,"recovering_objects_per_sec":91,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":732,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:53:46.415 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:53:46.415 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:46.720 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:46.721 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:46.736 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-16T04:53:44.816092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15118,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793776,"kb_used_data":2016,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693584,"statfs":{"total":386547056640,"available":383686230016,"internally_reserved":0,"allocated":2064384,"data_stored":1123288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":732,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001441"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12792,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.054626+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:40.054626+0000","last_peered":"2024-09-16T04:53:40.054626+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:40.054626+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12780,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.146094+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:53:40.146094+0000","last_peered":"2024-09-16T04:53:40.146094+0000","last_clean":"2024-09-16T04:53:40.146094+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:53:40.146094+0000","last_undegraded":"2024-09-16T04:53:40.146094+0000","last_fullsized":"2024-09-16T04:53:40.146094+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12788,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:39.045800+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:39.045800+0000","last_peered":"2024-09-16T04:53:39.045800+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:39.045800+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17235,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056241+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:40.056241+0000","last_peered":"2024-09-16T04:53:40.056241+0000","last_clean":"2024-09-16T04:53:40.056241+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:40.056241+0000","last_undegraded":"2024-09-16T04:53:40.056241+0000","last_fullsized":"2024-09-16T04:53:40.056241+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12894,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053101+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:40.053101+0000","last_peered":"2024-09-16T04:53:40.053101+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:40.053101+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16809,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.053188+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:40.053188+0000","last_peered":"2024-09-16T04:53:40.053188+0000","last_clean":"2024-09-16T04:53:40.053188+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:40.053188+0000","last_undegraded":"2024-09-16T04:53:40.053188+0000","last_fullsized":"2024-09-16T04:53:40.053188+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17157,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.054589+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:40.054589+0000","last_peered":"2024-09-16T04:53:40.054589+0000","last_clean":"2024-09-16T04:53:40.054589+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:40.054589+0000","last_undegraded":"2024-09-16T04:53:40.054589+0000","last_fullsized":"2024-09-16T04:53:40.054589+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:42.237116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17367,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056393+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:40.056393+0000","last_peered":"2024-09-16T04:53:40.056393+0000","last_clean":"2024-09-16T04:53:40.056393+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:40.056393+0000","last_undegraded":"2024-09-16T04:53:40.056393+0000","last_fullsized":"2024-09-16T04:53:40.056393+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17532,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.053094+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:40.053094+0000","last_peered":"2024-09-16T04:53:40.053094+0000","last_clean":"2024-09-16T04:53:40.053094+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:40.053094+0000","last_undegraded":"2024-09-16T04:53:40.053094+0000","last_fullsized":"2024-09-16T04:53:40.053094+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.940245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12883,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:39.046053+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:39.046053+0000","last_peered":"2024-09-16T04:53:39.046053+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:39.046053+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12643,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053224+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:40.053224+0000","last_peered":"2024-09-16T04:53:40.053224+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:40.053224+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12444,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.055310+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:40.055310+0000","last_peered":"2024-09-16T04:53:40.055310+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:40.055310+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":13132,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.008688+0000","last_change":"2024-09-16T04:53:38.388749+0000","last_active":"2024-09-16T04:53:40.008688+0000","last_peered":"2024-09-16T04:53:40.008688+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:40.008688+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1976}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12934,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053345+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:40.053345+0000","last_peered":"2024-09-16T04:53:40.053345+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:40.053345+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14941,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056380+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:40.056380+0000","last_peered":"2024-09-16T04:53:40.056380+0000","last_clean":"2024-09-16T04:53:40.056380+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:40.056380+0000","last_undegraded":"2024-09-16T04:53:40.056380+0000","last_fullsized":"2024-09-16T04:53:40.056380+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.632079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12421,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.055194+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:40.055194+0000","last_peered":"2024-09-16T04:53:40.055194+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:40.055194+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15118,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722056,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649784,"statfs":{"total":96636764160,"available":95897378816,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731264,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640576,"statfs":{"total":96636764160,"available":95887949824,"internally_reserved":0,"allocated":372736,"data_stored":141071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618460,"kb_used_data":908,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753380,"statfs":{"total":96636764160,"available":96003461120,"internally_reserved":0,"allocated":929792,"data_stored":693315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":552960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:49.738 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:50.053 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:50.054 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:50.068 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-16T04:53:48.816895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18698,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15938,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793788,"kb_used_data":2024,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693572,"statfs":{"total":386547056640,"available":383686217728,"internally_reserved":0,"allocated":2072576,"data_stored":1126668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1552,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.780043"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12792,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.054626+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:40.054626+0000","last_peered":"2024-09-16T04:53:40.054626+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:40.054626+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12780,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.146094+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:53:40.146094+0000","last_peered":"2024-09-16T04:53:40.146094+0000","last_clean":"2024-09-16T04:53:40.146094+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:53:40.146094+0000","last_undegraded":"2024-09-16T04:53:40.146094+0000","last_fullsized":"2024-09-16T04:53:40.146094+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12790,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.060372+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:40.060372+0000","last_peered":"2024-09-16T04:53:40.060372+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:40.060372+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17235,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056241+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:40.056241+0000","last_peered":"2024-09-16T04:53:40.056241+0000","last_clean":"2024-09-16T04:53:40.056241+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:40.056241+0000","last_undegraded":"2024-09-16T04:53:40.056241+0000","last_fullsized":"2024-09-16T04:53:40.056241+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12894,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053101+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:40.053101+0000","last_peered":"2024-09-16T04:53:40.053101+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:40.053101+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16809,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.053188+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:40.053188+0000","last_peered":"2024-09-16T04:53:40.053188+0000","last_clean":"2024-09-16T04:53:40.053188+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:40.053188+0000","last_undegraded":"2024-09-16T04:53:40.053188+0000","last_fullsized":"2024-09-16T04:53:40.053188+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17157,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.054589+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:40.054589+0000","last_peered":"2024-09-16T04:53:40.054589+0000","last_clean":"2024-09-16T04:53:40.054589+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:40.054589+0000","last_undegraded":"2024-09-16T04:53:40.054589+0000","last_fullsized":"2024-09-16T04:53:40.054589+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:42.237116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17367,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056393+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:40.056393+0000","last_peered":"2024-09-16T04:53:40.056393+0000","last_clean":"2024-09-16T04:53:40.056393+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:40.056393+0000","last_undegraded":"2024-09-16T04:53:40.056393+0000","last_fullsized":"2024-09-16T04:53:40.056393+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17532,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.053094+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:40.053094+0000","last_peered":"2024-09-16T04:53:40.053094+0000","last_clean":"2024-09-16T04:53:40.053094+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:40.053094+0000","last_undegraded":"2024-09-16T04:53:40.053094+0000","last_fullsized":"2024-09-16T04:53:40.053094+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.940245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12885,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.060444+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:40.060444+0000","last_peered":"2024-09-16T04:53:40.060444+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:40.060444+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12643,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053224+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:40.053224+0000","last_peered":"2024-09-16T04:53:40.053224+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:40.053224+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12444,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.055310+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:40.055310+0000","last_peered":"2024-09-16T04:53:40.055310+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:40.055310+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":14774,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:45.010938+0000","last_change":"2024-09-16T04:53:38.388749+0000","last_active":"2024-09-16T04:53:45.010938+0000","last_peered":"2024-09-16T04:53:45.010938+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:45.010938+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1156}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12934,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053345+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:40.053345+0000","last_peered":"2024-09-16T04:53:40.053345+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:40.053345+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14941,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056380+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:40.056380+0000","last_peered":"2024-09-16T04:53:40.056380+0000","last_clean":"2024-09-16T04:53:40.056380+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:40.056380+0000","last_undegraded":"2024-09-16T04:53:40.056380+0000","last_fullsized":"2024-09-16T04:53:40.056380+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.632079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12421,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.055194+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:40.055194+0000","last_peered":"2024-09-16T04:53:40.055194+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:40.055194+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18698,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15938,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722056,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649784,"statfs":{"total":96636764160,"available":95897378816,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731276,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640564,"statfs":{"total":96636764160,"available":95887937536,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618460,"kb_used_data":908,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753380,"statfs":{"total":96636764160,"available":96003461120,"internally_reserved":0,"allocated":929792,"data_stored":693315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":552960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:50.069 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:50.366 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:50.366 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:50.378 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-16T04:53:48.816895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18698,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15938,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793788,"kb_used_data":2024,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693572,"statfs":{"total":386547056640,"available":383686217728,"internally_reserved":0,"allocated":2072576,"data_stored":1126668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1552,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.780043"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12792,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.054626+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:40.054626+0000","last_peered":"2024-09-16T04:53:40.054626+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:40.054626+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12780,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.146094+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:53:40.146094+0000","last_peered":"2024-09-16T04:53:40.146094+0000","last_clean":"2024-09-16T04:53:40.146094+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:53:40.146094+0000","last_undegraded":"2024-09-16T04:53:40.146094+0000","last_fullsized":"2024-09-16T04:53:40.146094+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12790,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.060372+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:40.060372+0000","last_peered":"2024-09-16T04:53:40.060372+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:40.060372+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17235,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056241+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:40.056241+0000","last_peered":"2024-09-16T04:53:40.056241+0000","last_clean":"2024-09-16T04:53:40.056241+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:40.056241+0000","last_undegraded":"2024-09-16T04:53:40.056241+0000","last_fullsized":"2024-09-16T04:53:40.056241+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12894,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053101+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:40.053101+0000","last_peered":"2024-09-16T04:53:40.053101+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:40.053101+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16809,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.053188+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:40.053188+0000","last_peered":"2024-09-16T04:53:40.053188+0000","last_clean":"2024-09-16T04:53:40.053188+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:40.053188+0000","last_undegraded":"2024-09-16T04:53:40.053188+0000","last_fullsized":"2024-09-16T04:53:40.053188+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17157,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.054589+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:40.054589+0000","last_peered":"2024-09-16T04:53:40.054589+0000","last_clean":"2024-09-16T04:53:40.054589+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:40.054589+0000","last_undegraded":"2024-09-16T04:53:40.054589+0000","last_fullsized":"2024-09-16T04:53:40.054589+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:42.237116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17367,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056393+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:40.056393+0000","last_peered":"2024-09-16T04:53:40.056393+0000","last_clean":"2024-09-16T04:53:40.056393+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:40.056393+0000","last_undegraded":"2024-09-16T04:53:40.056393+0000","last_fullsized":"2024-09-16T04:53:40.056393+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17532,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.053094+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:40.053094+0000","last_peered":"2024-09-16T04:53:40.053094+0000","last_clean":"2024-09-16T04:53:40.053094+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:40.053094+0000","last_undegraded":"2024-09-16T04:53:40.053094+0000","last_fullsized":"2024-09-16T04:53:40.053094+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.940245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12885,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.060444+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:40.060444+0000","last_peered":"2024-09-16T04:53:40.060444+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:40.060444+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12643,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053224+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:40.053224+0000","last_peered":"2024-09-16T04:53:40.053224+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:40.053224+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12444,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.055310+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:40.055310+0000","last_peered":"2024-09-16T04:53:40.055310+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:40.055310+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":14774,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:45.010938+0000","last_change":"2024-09-16T04:53:38.388749+0000","last_active":"2024-09-16T04:53:45.010938+0000","last_peered":"2024-09-16T04:53:45.010938+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:45.010938+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1156}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12934,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053345+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:40.053345+0000","last_peered":"2024-09-16T04:53:40.053345+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:40.053345+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14941,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056380+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:40.056380+0000","last_peered":"2024-09-16T04:53:40.056380+0000","last_clean":"2024-09-16T04:53:40.056380+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:40.056380+0000","last_undegraded":"2024-09-16T04:53:40.056380+0000","last_fullsized":"2024-09-16T04:53:40.056380+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.632079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12421,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.055194+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:40.055194+0000","last_peered":"2024-09-16T04:53:40.055194+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:40.055194+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18698,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15938,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722056,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649784,"statfs":{"total":96636764160,"available":95897378816,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731276,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640564,"statfs":{"total":96636764160,"available":95887937536,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618460,"kb_used_data":908,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753380,"statfs":{"total":96636764160,"available":96003461120,"internally_reserved":0,"allocated":929792,"data_stored":693315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":552960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:50.379 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:53:50.379 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 12792, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.054626+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:53:40.054626+0000', 'last_peered': '2024-09-16T04:53:40.054626+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:53:40.054626+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:50.380 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:53:50.380 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12790, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.060372+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:53:40.060372+0000', 'last_peered': '2024-09-16T04:53:40.060372+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:53:40.060372+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:50.380 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:53:50.380 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12894, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.053101+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:53:40.053101+0000', 'last_peered': '2024-09-16T04:53:40.053101+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:53:40.053101+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:50.380 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:53:50.380 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12885, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.060444+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:53:40.060444+0000', 'last_peered': '2024-09-16T04:53:40.060444+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:53:40.060444+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:50.380 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:53:50.381 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12643, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.053224+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:53:40.053224+0000', 'last_peered': '2024-09-16T04:53:40.053224+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:53:40.053224+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:50.381 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:53:50.381 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12444, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.055310+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:53:40.055310+0000', 'last_peered': '2024-09-16T04:53:40.055310+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:53:40.055310+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:50.381 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:53:50.381 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12532", 'reported_seq': 14774, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:45.010938+0000', 'last_change': '2024-09-16T04:53:38.388749+0000', 'last_active': '2024-09-16T04:53:45.010938+0000', 'last_peered': '2024-09-16T04:53:45.010938+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:53:45.010938+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1157, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12531, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1070, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1156}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:50.381 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:53:50.381 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12934, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.053345+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:53:40.053345+0000', 'last_peered': '2024-09-16T04:53:40.053345+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:53:40.053345+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:50.381 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:53:50.381 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12305", 'reported_seq': 12421, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.055194+0000', 'last_change': '2024-09-16T04:41:27.900855+0000', 'last_active': '2024-09-16T04:53:40.055194+0000', 'last_peered': '2024-09-16T04:53:40.055194+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:53:40.055194+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2230, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12303, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:50.382 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:53:50.753 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:50.753 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18698/112 objects degraded (16694.643%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":843,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2860838912,"bytes_avail":383686217728,"bytes_total":386547056640,"degraded_objects":18698,"degraded_total":112,"degraded_ratio":166.94642857142858,"recovering_objects_per_sec":158,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1552,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:53:50.767 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:53:50.768 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:51.065 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:51.066 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:51.078 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-09-16T04:53:50.817230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16992,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785468,"kb_used_data":2024,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701892,"statfs":{"total":386547056640,"available":383694737408,"internally_reserved":0,"allocated":2072576,"data_stored":1126668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2850145841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1874,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.765494"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12792,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.054626+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:40.054626+0000","last_peered":"2024-09-16T04:53:40.054626+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:40.054626+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12780,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.146094+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:53:40.146094+0000","last_peered":"2024-09-16T04:53:40.146094+0000","last_clean":"2024-09-16T04:53:40.146094+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:53:40.146094+0000","last_undegraded":"2024-09-16T04:53:40.146094+0000","last_fullsized":"2024-09-16T04:53:40.146094+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12791,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.061628+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:40.061628+0000","last_peered":"2024-09-16T04:53:40.061628+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:40.061628+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17235,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056241+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:40.056241+0000","last_peered":"2024-09-16T04:53:40.056241+0000","last_clean":"2024-09-16T04:53:40.056241+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:40.056241+0000","last_undegraded":"2024-09-16T04:53:40.056241+0000","last_fullsized":"2024-09-16T04:53:40.056241+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12894,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053101+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:40.053101+0000","last_peered":"2024-09-16T04:53:40.053101+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:40.053101+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16809,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.053188+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:40.053188+0000","last_peered":"2024-09-16T04:53:40.053188+0000","last_clean":"2024-09-16T04:53:40.053188+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:40.053188+0000","last_undegraded":"2024-09-16T04:53:40.053188+0000","last_fullsized":"2024-09-16T04:53:40.053188+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17157,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.054589+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:40.054589+0000","last_peered":"2024-09-16T04:53:40.054589+0000","last_clean":"2024-09-16T04:53:40.054589+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:40.054589+0000","last_undegraded":"2024-09-16T04:53:40.054589+0000","last_fullsized":"2024-09-16T04:53:40.054589+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:42.237116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17367,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056393+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:40.056393+0000","last_peered":"2024-09-16T04:53:40.056393+0000","last_clean":"2024-09-16T04:53:40.056393+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:40.056393+0000","last_undegraded":"2024-09-16T04:53:40.056393+0000","last_fullsized":"2024-09-16T04:53:40.056393+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17532,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.053094+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:40.053094+0000","last_peered":"2024-09-16T04:53:40.053094+0000","last_clean":"2024-09-16T04:53:40.053094+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:40.053094+0000","last_undegraded":"2024-09-16T04:53:40.053094+0000","last_fullsized":"2024-09-16T04:53:40.053094+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.940245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12885,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.060444+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:40.060444+0000","last_peered":"2024-09-16T04:53:40.060444+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:40.060444+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12643,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053224+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:40.053224+0000","last_peered":"2024-09-16T04:53:40.053224+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:40.053224+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12444,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.055310+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:40.055310+0000","last_peered":"2024-09-16T04:53:40.055310+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:40.055310+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":16882,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:50.011309+0000","last_change":"2024-09-16T04:53:38.388749+0000","last_active":"2024-09-16T04:53:50.011309+0000","last_peered":"2024-09-16T04:53:50.011309+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:50.011309+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-102}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12934,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.053345+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:40.053345+0000","last_peered":"2024-09-16T04:53:40.053345+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:40.053345+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14941,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056380+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:40.056380+0000","last_peered":"2024-09-16T04:53:40.056380+0000","last_clean":"2024-09-16T04:53:40.056380+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:40.056380+0000","last_undegraded":"2024-09-16T04:53:40.056380+0000","last_fullsized":"2024-09-16T04:53:40.056380+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.632079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12421,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.055194+0000","last_change":"2024-09-16T04:41:27.900855+0000","last_active":"2024-09-16T04:53:40.055194+0000","last_peered":"2024-09-16T04:53:40.055194+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:40.055194+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16992,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722056,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649784,"statfs":{"total":96636764160,"available":95897378816,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731276,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640564,"statfs":{"total":96636764160,"available":95887937536,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610140,"kb_used_data":908,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93761700,"statfs":{"total":96636764160,"available":96011980800,"internally_reserved":0,"allocated":929792,"data_stored":693315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623832783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":552960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:54.080 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:54.385 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:54.385 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:54.398 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-16T04:53:52.817612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17096,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785476,"kb_used_data":2032,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701884,"statfs":{"total":386547056640,"available":383694729216,"internally_reserved":0,"allocated":2080768,"data_stored":1130032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2850145841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1978,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001519"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12794,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.088375+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:51.088375+0000","last_peered":"2024-09-16T04:53:51.088375+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:51.088375+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12780,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.146094+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:53:40.146094+0000","last_peered":"2024-09-16T04:53:40.146094+0000","last_clean":"2024-09-16T04:53:40.146094+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:53:40.146094+0000","last_undegraded":"2024-09-16T04:53:40.146094+0000","last_fullsized":"2024-09-16T04:53:40.146094+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12791,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.061628+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:40.061628+0000","last_peered":"2024-09-16T04:53:40.061628+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:40.061628+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17235,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056241+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:40.056241+0000","last_peered":"2024-09-16T04:53:40.056241+0000","last_clean":"2024-09-16T04:53:40.056241+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:40.056241+0000","last_undegraded":"2024-09-16T04:53:40.056241+0000","last_fullsized":"2024-09-16T04:53:40.056241+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12896,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.085938+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:51.085938+0000","last_peered":"2024-09-16T04:53:51.085938+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:51.085938+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16811,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T04:53:51.088168+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:51.088168+0000","last_peered":"2024-09-16T04:53:51.088168+0000","last_clean":"2024-09-16T04:53:51.088168+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:51.088168+0000","last_undegraded":"2024-09-16T04:53:51.088168+0000","last_fullsized":"2024-09-16T04:53:51.088168+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17159,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T04:53:51.091913+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:51.091913+0000","last_peered":"2024-09-16T04:53:51.091913+0000","last_clean":"2024-09-16T04:53:51.091913+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:51.091913+0000","last_undegraded":"2024-09-16T04:53:51.091913+0000","last_fullsized":"2024-09-16T04:53:51.091913+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:53.266905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17367,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056393+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:40.056393+0000","last_peered":"2024-09-16T04:53:40.056393+0000","last_clean":"2024-09-16T04:53:40.056393+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:40.056393+0000","last_undegraded":"2024-09-16T04:53:40.056393+0000","last_fullsized":"2024-09-16T04:53:40.056393+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17534,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T04:53:51.088074+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:51.088074+0000","last_peered":"2024-09-16T04:53:51.088074+0000","last_clean":"2024-09-16T04:53:51.088074+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:51.088074+0000","last_undegraded":"2024-09-16T04:53:51.088074+0000","last_fullsized":"2024-09-16T04:53:51.088074+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.114801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12885,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.060444+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:40.060444+0000","last_peered":"2024-09-16T04:53:40.060444+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:40.060444+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12645,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.087070+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:51.087070+0000","last_peered":"2024-09-16T04:53:51.087070+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:51.087070+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12446,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.089174+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:51.089174+0000","last_peered":"2024-09-16T04:53:51.089174+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:51.089174+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":16882,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:50.011309+0000","last_change":"2024-09-16T04:53:38.388749+0000","last_active":"2024-09-16T04:53:50.011309+0000","last_peered":"2024-09-16T04:53:50.011309+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:50.011309+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-102}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12936,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.087268+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:51.087268+0000","last_peered":"2024-09-16T04:53:51.087268+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:51.087268+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14941,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056380+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:40.056380+0000","last_peered":"2024-09-16T04:53:40.056380+0000","last_clean":"2024-09-16T04:53:40.056380+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:40.056380+0000","last_undegraded":"2024-09-16T04:53:40.056380+0000","last_fullsized":"2024-09-16T04:53:40.056380+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.632079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12632,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.236009+0000","last_change":"2024-09-16T04:53:50.515593+0000","last_active":"2024-09-16T04:53:51.236009+0000","last_peered":"2024-09-16T04:53:51.236009+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:51.236009+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17096,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722064,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649776,"statfs":{"total":96636764160,"available":95897370624,"internally_reserved":0,"allocated":389120,"data_stored":147815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731276,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640564,"statfs":{"total":96636764160,"available":95887937536,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610140,"kb_used_data":908,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93761700,"statfs":{"total":96636764160,"available":96011980800,"internally_reserved":0,"allocated":929792,"data_stored":693315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623832783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":552960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:54.399 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:54.691 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:54.691 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:54.704 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-16T04:53:52.817612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17096,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785476,"kb_used_data":2032,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701884,"statfs":{"total":386547056640,"available":383694729216,"internally_reserved":0,"allocated":2080768,"data_stored":1130032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2850145841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1978,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001519"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12794,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.088375+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:51.088375+0000","last_peered":"2024-09-16T04:53:51.088375+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:51.088375+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12780,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.146094+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:53:40.146094+0000","last_peered":"2024-09-16T04:53:40.146094+0000","last_clean":"2024-09-16T04:53:40.146094+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:53:40.146094+0000","last_undegraded":"2024-09-16T04:53:40.146094+0000","last_fullsized":"2024-09-16T04:53:40.146094+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12791,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.061628+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:40.061628+0000","last_peered":"2024-09-16T04:53:40.061628+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:40.061628+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17235,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056241+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:40.056241+0000","last_peered":"2024-09-16T04:53:40.056241+0000","last_clean":"2024-09-16T04:53:40.056241+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:40.056241+0000","last_undegraded":"2024-09-16T04:53:40.056241+0000","last_fullsized":"2024-09-16T04:53:40.056241+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12896,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.085938+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:51.085938+0000","last_peered":"2024-09-16T04:53:51.085938+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:51.085938+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16811,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T04:53:51.088168+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:51.088168+0000","last_peered":"2024-09-16T04:53:51.088168+0000","last_clean":"2024-09-16T04:53:51.088168+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:51.088168+0000","last_undegraded":"2024-09-16T04:53:51.088168+0000","last_fullsized":"2024-09-16T04:53:51.088168+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17159,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T04:53:51.091913+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:51.091913+0000","last_peered":"2024-09-16T04:53:51.091913+0000","last_clean":"2024-09-16T04:53:51.091913+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:51.091913+0000","last_undegraded":"2024-09-16T04:53:51.091913+0000","last_fullsized":"2024-09-16T04:53:51.091913+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:53.266905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17367,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056393+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:40.056393+0000","last_peered":"2024-09-16T04:53:40.056393+0000","last_clean":"2024-09-16T04:53:40.056393+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:40.056393+0000","last_undegraded":"2024-09-16T04:53:40.056393+0000","last_fullsized":"2024-09-16T04:53:40.056393+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:44.517039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17534,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T04:53:51.088074+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:51.088074+0000","last_peered":"2024-09-16T04:53:51.088074+0000","last_clean":"2024-09-16T04:53:51.088074+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:51.088074+0000","last_undegraded":"2024-09-16T04:53:51.088074+0000","last_fullsized":"2024-09-16T04:53:51.088074+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.114801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12885,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.060444+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:40.060444+0000","last_peered":"2024-09-16T04:53:40.060444+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:40.060444+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12645,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.087070+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:51.087070+0000","last_peered":"2024-09-16T04:53:51.087070+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:51.087070+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12446,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.089174+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:51.089174+0000","last_peered":"2024-09-16T04:53:51.089174+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:51.089174+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":16882,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:50.011309+0000","last_change":"2024-09-16T04:53:38.388749+0000","last_active":"2024-09-16T04:53:50.011309+0000","last_peered":"2024-09-16T04:53:50.011309+0000","last_clean":"2024-09-16T04:40:14.617739+0000","last_became_active":"2024-09-16T04:41:27.773536+0000","last_became_peered":"2024-09-16T04:41:27.773536+0000","last_unstale":"2024-09-16T04:53:50.011309+0000","last_undegraded":"2024-09-16T04:41:27.522623+0000","last_fullsized":"2024-09-16T04:41:27.486959+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-102}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12936,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.087268+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:51.087268+0000","last_peered":"2024-09-16T04:53:51.087268+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:51.087268+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14941,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T04:53:40.056380+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:40.056380+0000","last_peered":"2024-09-16T04:53:40.056380+0000","last_clean":"2024-09-16T04:53:40.056380+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:40.056380+0000","last_undegraded":"2024-09-16T04:53:40.056380+0000","last_fullsized":"2024-09-16T04:53:40.056380+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:41.632079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12632,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.236009+0000","last_change":"2024-09-16T04:53:50.515593+0000","last_active":"2024-09-16T04:53:51.236009+0000","last_peered":"2024-09-16T04:53:51.236009+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:51.236009+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17096,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722064,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649776,"statfs":{"total":96636764160,"available":95897370624,"internally_reserved":0,"allocated":389120,"data_stored":147815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731276,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640564,"statfs":{"total":96636764160,"available":95887937536,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610140,"kb_used_data":908,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93761700,"statfs":{"total":96636764160,"available":96011980800,"internally_reserved":0,"allocated":929792,"data_stored":693315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623832783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":552960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:54.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:53:54.705 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 12794, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:51.088375+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:53:51.088375+0000', 'last_peered': '2024-09-16T04:53:51.088375+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:53:51.088375+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:54.705 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:53:54.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12791, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.061628+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:53:40.061628+0000', 'last_peered': '2024-09-16T04:53:40.061628+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:53:40.061628+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:54.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:53:54.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12896, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:51.085938+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:53:51.085938+0000', 'last_peered': '2024-09-16T04:53:51.085938+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:53:51.085938+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:54.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:53:54.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12885, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:40.060444+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:53:40.060444+0000', 'last_peered': '2024-09-16T04:53:40.060444+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:53:40.060444+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:54.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:53:54.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12645, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:51.087070+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:53:51.087070+0000', 'last_peered': '2024-09-16T04:53:51.087070+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:53:51.087070+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:54.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:53:54.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12446, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:51.089174+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:53:51.089174+0000', 'last_peered': '2024-09-16T04:53:51.089174+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:53:51.089174+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:54.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-16T04:53:54.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'12532", 'reported_seq': 16882, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:50.011309+0000', 'last_change': '2024-09-16T04:53:38.388749+0000', 'last_active': '2024-09-16T04:53:50.011309+0000', 'last_peered': '2024-09-16T04:53:50.011309+0000', 'last_clean': '2024-09-16T04:40:14.617739+0000', 'last_became_active': '2024-09-16T04:41:27.773536+0000', 'last_became_peered': '2024-09-16T04:41:27.773536+0000', 'last_unstale': '2024-09-16T04:53:50.011309+0000', 'last_undegraded': '2024-09-16T04:41:27.522623+0000', 'last_fullsized': '2024-09-16T04:41:27.486959+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12531, 'num_write_kb': 50127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2124, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -102}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:54.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:53:54.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12936, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:51.087268+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:53:51.087268+0000', 'last_peered': '2024-09-16T04:53:51.087268+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:53:51.087268+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:54.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:53:54.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12305", 'reported_seq': 12632, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:51.236009+0000', 'last_change': '2024-09-16T04:53:50.515593+0000', 'last_active': '2024-09-16T04:53:51.236009+0000', 'last_peered': '2024-09-16T04:53:51.236009+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:53:51.236009+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2126, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12303, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 117, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2124}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:54.707 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:53:55.080 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:55.081 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17540/112 objects degraded (15660.714%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":847,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2852327424,"bytes_avail":383694729216,"bytes_total":386547056640,"degraded_objects":17540,"degraded_total":112,"degraded_ratio":156.60714285714286,"recovering_objects_per_sec":247,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1978,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:53:55.094 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:53:55.095 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:55.408 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:55.409 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:55.425 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-09-16T04:53:54.818004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14977,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793812,"kb_used_data":2040,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693548,"statfs":{"total":386547056640,"available":383686193152,"internally_reserved":0,"allocated":2088960,"data_stored":1128547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-961,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001483"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12794,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.088375+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:51.088375+0000","last_peered":"2024-09-16T04:53:51.088375+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:51.088375+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12784,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088878+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:53:52.088878+0000","last_peered":"2024-09-16T04:53:52.088878+0000","last_clean":"2024-09-16T04:53:52.088878+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:53:52.088878+0000","last_undegraded":"2024-09-16T04:53:52.088878+0000","last_fullsized":"2024-09-16T04:53:52.088878+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:55.640173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12791,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.061628+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:40.061628+0000","last_peered":"2024-09-16T04:53:40.061628+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:40.061628+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17239,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088591+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:52.088591+0000","last_peered":"2024-09-16T04:53:52.088591+0000","last_clean":"2024-09-16T04:53:52.088591+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:52.088591+0000","last_undegraded":"2024-09-16T04:53:52.088591+0000","last_fullsized":"2024-09-16T04:53:52.088591+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:54.603396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12896,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.085938+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:51.085938+0000","last_peered":"2024-09-16T04:53:51.085938+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:51.085938+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16811,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T04:53:51.088168+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:51.088168+0000","last_peered":"2024-09-16T04:53:51.088168+0000","last_clean":"2024-09-16T04:53:51.088168+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:51.088168+0000","last_undegraded":"2024-09-16T04:53:51.088168+0000","last_fullsized":"2024-09-16T04:53:51.088168+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17159,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T04:53:51.091913+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:51.091913+0000","last_peered":"2024-09-16T04:53:51.091913+0000","last_clean":"2024-09-16T04:53:51.091913+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:51.091913+0000","last_undegraded":"2024-09-16T04:53:51.091913+0000","last_fullsized":"2024-09-16T04:53:51.091913+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:53.266905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17371,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088678+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:52.088678+0000","last_peered":"2024-09-16T04:53:52.088678+0000","last_clean":"2024-09-16T04:53:52.088678+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:52.088678+0000","last_undegraded":"2024-09-16T04:53:52.088678+0000","last_fullsized":"2024-09-16T04:53:52.088678+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:54.603373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17534,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T04:53:51.088074+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:51.088074+0000","last_peered":"2024-09-16T04:53:51.088074+0000","last_clean":"2024-09-16T04:53:51.088074+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:51.088074+0000","last_undegraded":"2024-09-16T04:53:51.088074+0000","last_fullsized":"2024-09-16T04:53:51.088074+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.114801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12885,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:40.060444+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:40.060444+0000","last_peered":"2024-09-16T04:53:40.060444+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:40.060444+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12645,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.087070+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:51.087070+0000","last_peered":"2024-09-16T04:53:51.087070+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:51.087070+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12446,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.089174+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:51.089174+0000","last_peered":"2024-09-16T04:53:51.089174+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:51.089174+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12640,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.174104+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:53:52.174104+0000","last_peered":"2024-09-16T04:53:52.174104+0000","last_clean":"2024-09-16T04:53:52.174104+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:53:52.174104+0000","last_undegraded":"2024-09-16T04:53:52.174104+0000","last_fullsized":"2024-09-16T04:53:52.174104+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:57.651124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12936,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.087268+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:51.087268+0000","last_peered":"2024-09-16T04:53:51.087268+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:51.087268+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14945,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088746+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:52.088746+0000","last_peered":"2024-09-16T04:53:52.088746+0000","last_clean":"2024-09-16T04:53:52.088746+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:52.088746+0000","last_undegraded":"2024-09-16T04:53:52.088746+0000","last_fullsized":"2024-09-16T04:53:52.088746+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.564749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12632,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.236009+0000","last_change":"2024-09-16T04:53:50.515593+0000","last_active":"2024-09-16T04:53:51.236009+0000","last_peered":"2024-09-16T04:53:51.236009+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:51.236009+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14977,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722064,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649776,"statfs":{"total":96636764160,"available":95897370624,"internally_reserved":0,"allocated":389120,"data_stored":147815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731276,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640564,"statfs":{"total":96636764160,"available":95887937536,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618476,"kb_used_data":916,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753364,"statfs":{"total":96636764160,"available":96003444736,"internally_reserved":0,"allocated":937984,"data_stored":691830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721996,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649844,"statfs":{"total":96636764160,"available":95897440256,"internally_reserved":0,"allocated":380928,"data_stored":144451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:58.428 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:58.723 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:58.723 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:58.736 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-09-16T04:53:56.818318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15857,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793844,"kb_used_data":2072,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693516,"statfs":{"total":386547056640,"available":383686160384,"internally_reserved":0,"allocated":2121728,"data_stored":1141961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001422"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12794,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.088375+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:51.088375+0000","last_peered":"2024-09-16T04:53:51.088375+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:51.088375+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12784,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088878+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:53:52.088878+0000","last_peered":"2024-09-16T04:53:52.088878+0000","last_clean":"2024-09-16T04:53:52.088878+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:53:52.088878+0000","last_undegraded":"2024-09-16T04:53:52.088878+0000","last_fullsized":"2024-09-16T04:53:52.088878+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:55.640173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12795,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.093109+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:52.093109+0000","last_peered":"2024-09-16T04:53:52.093109+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:52.093109+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17239,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088591+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:52.088591+0000","last_peered":"2024-09-16T04:53:52.088591+0000","last_clean":"2024-09-16T04:53:52.088591+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:52.088591+0000","last_undegraded":"2024-09-16T04:53:52.088591+0000","last_fullsized":"2024-09-16T04:53:52.088591+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:54.603396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12896,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.085938+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:51.085938+0000","last_peered":"2024-09-16T04:53:51.085938+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:51.085938+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16811,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T04:53:51.088168+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:51.088168+0000","last_peered":"2024-09-16T04:53:51.088168+0000","last_clean":"2024-09-16T04:53:51.088168+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:51.088168+0000","last_undegraded":"2024-09-16T04:53:51.088168+0000","last_fullsized":"2024-09-16T04:53:51.088168+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17161,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090961+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:52.090961+0000","last_peered":"2024-09-16T04:53:52.090961+0000","last_clean":"2024-09-16T04:53:52.090961+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:52.090961+0000","last_undegraded":"2024-09-16T04:53:52.090961+0000","last_fullsized":"2024-09-16T04:53:52.090961+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:53.266905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17371,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088678+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:52.088678+0000","last_peered":"2024-09-16T04:53:52.088678+0000","last_clean":"2024-09-16T04:53:52.088678+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:52.088678+0000","last_undegraded":"2024-09-16T04:53:52.088678+0000","last_fullsized":"2024-09-16T04:53:52.088678+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:54.603373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17534,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T04:53:51.088074+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:51.088074+0000","last_peered":"2024-09-16T04:53:51.088074+0000","last_clean":"2024-09-16T04:53:51.088074+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:51.088074+0000","last_undegraded":"2024-09-16T04:53:51.088074+0000","last_fullsized":"2024-09-16T04:53:51.088074+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.114801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12889,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.093281+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:52.093281+0000","last_peered":"2024-09-16T04:53:52.093281+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:52.093281+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12645,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.087070+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:51.087070+0000","last_peered":"2024-09-16T04:53:51.087070+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:51.087070+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12446,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.089174+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:51.089174+0000","last_peered":"2024-09-16T04:53:51.089174+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:51.089174+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12640,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.174104+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:53:52.174104+0000","last_peered":"2024-09-16T04:53:52.174104+0000","last_clean":"2024-09-16T04:53:52.174104+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:53:52.174104+0000","last_undegraded":"2024-09-16T04:53:52.174104+0000","last_fullsized":"2024-09-16T04:53:52.174104+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:57.651124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12936,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:51.087268+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:51.087268+0000","last_peered":"2024-09-16T04:53:51.087268+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:51.087268+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14945,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088746+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:52.088746+0000","last_peered":"2024-09-16T04:53:52.088746+0000","last_clean":"2024-09-16T04:53:52.088746+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:52.088746+0000","last_undegraded":"2024-09-16T04:53:52.088746+0000","last_fullsized":"2024-09-16T04:53:52.088746+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.564749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":14395,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:56.237154+0000","last_change":"2024-09-16T04:53:50.515593+0000","last_active":"2024-09-16T04:53:56.237154+0000","last_peered":"2024-09-16T04:53:56.237154+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:56.237154+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":997,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1244}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15857,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722064,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649776,"statfs":{"total":96636764160,"available":95897370624,"internally_reserved":0,"allocated":389120,"data_stored":147815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731292,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640548,"statfs":{"total":96636764160,"available":95887921152,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618476,"kb_used_data":916,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753364,"statfs":{"total":96636764160,"available":96003444736,"internally_reserved":0,"allocated":937984,"data_stored":691830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722012,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649828,"statfs":{"total":96636764160,"available":95897423872,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:58.737 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:59.033 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:59.034 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:59.046 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-09-16T04:53:58.818646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15857,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793852,"kb_used_data":2080,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693508,"statfs":{"total":386547056640,"available":383686152192,"internally_reserved":0,"allocated":2129920,"data_stored":1145304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1135,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001415"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12796,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090395+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:52.090395+0000","last_peered":"2024-09-16T04:53:52.090395+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:52.090395+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12784,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088878+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:53:52.088878+0000","last_peered":"2024-09-16T04:53:52.088878+0000","last_clean":"2024-09-16T04:53:52.088878+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:53:52.088878+0000","last_undegraded":"2024-09-16T04:53:52.088878+0000","last_fullsized":"2024-09-16T04:53:52.088878+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:55.640173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12795,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.093109+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:52.093109+0000","last_peered":"2024-09-16T04:53:52.093109+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:52.093109+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17239,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088591+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:52.088591+0000","last_peered":"2024-09-16T04:53:52.088591+0000","last_clean":"2024-09-16T04:53:52.088591+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:52.088591+0000","last_undegraded":"2024-09-16T04:53:52.088591+0000","last_fullsized":"2024-09-16T04:53:52.088591+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:54.603396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12898,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.088423+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:52.088423+0000","last_peered":"2024-09-16T04:53:52.088423+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:52.088423+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16813,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090691+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:52.090691+0000","last_peered":"2024-09-16T04:53:52.090691+0000","last_clean":"2024-09-16T04:53:52.090691+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:52.090691+0000","last_undegraded":"2024-09-16T04:53:52.090691+0000","last_fullsized":"2024-09-16T04:53:52.090691+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17161,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090961+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:52.090961+0000","last_peered":"2024-09-16T04:53:52.090961+0000","last_clean":"2024-09-16T04:53:52.090961+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:52.090961+0000","last_undegraded":"2024-09-16T04:53:52.090961+0000","last_fullsized":"2024-09-16T04:53:52.090961+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:53.266905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17371,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088678+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:52.088678+0000","last_peered":"2024-09-16T04:53:52.088678+0000","last_clean":"2024-09-16T04:53:52.088678+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:52.088678+0000","last_undegraded":"2024-09-16T04:53:52.088678+0000","last_fullsized":"2024-09-16T04:53:52.088678+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:54.603373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17536,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090603+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:52.090603+0000","last_peered":"2024-09-16T04:53:52.090603+0000","last_clean":"2024-09-16T04:53:52.090603+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:52.090603+0000","last_undegraded":"2024-09-16T04:53:52.090603+0000","last_fullsized":"2024-09-16T04:53:52.090603+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.114801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12889,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.093281+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:52.093281+0000","last_peered":"2024-09-16T04:53:52.093281+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:52.093281+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12647,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090697+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:52.090697+0000","last_peered":"2024-09-16T04:53:52.090697+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:52.090697+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12449,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.092832+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:52.092832+0000","last_peered":"2024-09-16T04:53:52.092832+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:52.092832+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12640,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.174104+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:53:52.174104+0000","last_peered":"2024-09-16T04:53:52.174104+0000","last_clean":"2024-09-16T04:53:52.174104+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:53:52.174104+0000","last_undegraded":"2024-09-16T04:53:52.174104+0000","last_fullsized":"2024-09-16T04:53:52.174104+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:57.651124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12939,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.091472+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:52.091472+0000","last_peered":"2024-09-16T04:53:52.091472+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:52.091472+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14945,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088746+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:52.088746+0000","last_peered":"2024-09-16T04:53:52.088746+0000","last_clean":"2024-09-16T04:53:52.088746+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:52.088746+0000","last_undegraded":"2024-09-16T04:53:52.088746+0000","last_fullsized":"2024-09-16T04:53:52.088746+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.564749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":14395,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:56.237154+0000","last_change":"2024-09-16T04:53:50.515593+0000","last_active":"2024-09-16T04:53:56.237154+0000","last_peered":"2024-09-16T04:53:56.237154+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:56.237154+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":997,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1244}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15857,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722072,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649768,"statfs":{"total":96636764160,"available":95897362432,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731292,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640548,"statfs":{"total":96636764160,"available":95887921152,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618476,"kb_used_data":916,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753364,"statfs":{"total":96636764160,"available":96003444736,"internally_reserved":0,"allocated":937984,"data_stored":691830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722012,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649828,"statfs":{"total":96636764160,"available":95897423872,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:53:59.047 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:53:59.047 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 12796, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.090395+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:53:52.090395+0000', 'last_peered': '2024-09-16T04:53:52.090395+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:53:52.090395+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:59.047 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:53:59.047 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12795, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.093109+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:53:52.093109+0000', 'last_peered': '2024-09-16T04:53:52.093109+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:53:52.093109+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:59.047 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:53:59.047 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12898, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.088423+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:53:52.088423+0000', 'last_peered': '2024-09-16T04:53:52.088423+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:53:52.088423+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:59.047 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:53:59.047 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12889, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.093281+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:53:52.093281+0000', 'last_peered': '2024-09-16T04:53:52.093281+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:53:52.093281+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:53:59.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:53:59.048 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12647, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.090697+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:53:52.090697+0000', 'last_peered': '2024-09-16T04:53:52.090697+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:53:52.090697+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:59.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:53:59.048 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12449, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.092832+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:53:52.092832+0000', 'last_peered': '2024-09-16T04:53:52.092832+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:53:52.092832+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:59.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:53:59.048 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12939, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.091472+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:53:52.091472+0000', 'last_peered': '2024-09-16T04:53:52.091472+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:53:52.091472+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:53:59.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:53:59.048 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12305", 'reported_seq': 14395, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:56.237154+0000', 'last_change': '2024-09-16T04:53:50.515593+0000', 'last_active': '2024-09-16T04:53:56.237154+0000', 'last_peered': '2024-09-16T04:53:56.237154+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:53:56.237154+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1246, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12303, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 997, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1244}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:53:59.048 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:53:59.411 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:59.411 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16557/112 objects degraded (14783.036%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":851,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2860904448,"bytes_avail":383686152192,"bytes_total":386547056640,"degraded_objects":16557,"degraded_total":112,"degraded_ratio":147.83035714285714},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:53:59.426 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T04:53:59.426 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:53:59.721 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:53:59.722 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:53:59.734 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-09-16T04:53:58.818646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15857,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793852,"kb_used_data":2080,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693508,"statfs":{"total":386547056640,"available":383686152192,"internally_reserved":0,"allocated":2129920,"data_stored":1145304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1135,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001415"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12796,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090395+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:52.090395+0000","last_peered":"2024-09-16T04:53:52.090395+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:52.090395+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12784,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088878+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:53:52.088878+0000","last_peered":"2024-09-16T04:53:52.088878+0000","last_clean":"2024-09-16T04:53:52.088878+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:53:52.088878+0000","last_undegraded":"2024-09-16T04:53:52.088878+0000","last_fullsized":"2024-09-16T04:53:52.088878+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:55.640173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12795,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.093109+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:52.093109+0000","last_peered":"2024-09-16T04:53:52.093109+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:52.093109+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17239,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088591+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:52.088591+0000","last_peered":"2024-09-16T04:53:52.088591+0000","last_clean":"2024-09-16T04:53:52.088591+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:52.088591+0000","last_undegraded":"2024-09-16T04:53:52.088591+0000","last_fullsized":"2024-09-16T04:53:52.088591+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:54.603396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12898,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.088423+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:52.088423+0000","last_peered":"2024-09-16T04:53:52.088423+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:52.088423+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16813,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090691+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:52.090691+0000","last_peered":"2024-09-16T04:53:52.090691+0000","last_clean":"2024-09-16T04:53:52.090691+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:52.090691+0000","last_undegraded":"2024-09-16T04:53:52.090691+0000","last_fullsized":"2024-09-16T04:53:52.090691+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17161,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090961+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:52.090961+0000","last_peered":"2024-09-16T04:53:52.090961+0000","last_clean":"2024-09-16T04:53:52.090961+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:52.090961+0000","last_undegraded":"2024-09-16T04:53:52.090961+0000","last_fullsized":"2024-09-16T04:53:52.090961+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:53.266905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17371,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088678+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:52.088678+0000","last_peered":"2024-09-16T04:53:52.088678+0000","last_clean":"2024-09-16T04:53:52.088678+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:52.088678+0000","last_undegraded":"2024-09-16T04:53:52.088678+0000","last_fullsized":"2024-09-16T04:53:52.088678+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:54.603373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17536,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090603+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:52.090603+0000","last_peered":"2024-09-16T04:53:52.090603+0000","last_clean":"2024-09-16T04:53:52.090603+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:52.090603+0000","last_undegraded":"2024-09-16T04:53:52.090603+0000","last_fullsized":"2024-09-16T04:53:52.090603+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.114801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12889,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.093281+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:52.093281+0000","last_peered":"2024-09-16T04:53:52.093281+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:52.093281+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12647,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090697+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:52.090697+0000","last_peered":"2024-09-16T04:53:52.090697+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:52.090697+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12449,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.092832+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:52.092832+0000","last_peered":"2024-09-16T04:53:52.092832+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:52.092832+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12640,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.174104+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:53:52.174104+0000","last_peered":"2024-09-16T04:53:52.174104+0000","last_clean":"2024-09-16T04:53:52.174104+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:53:52.174104+0000","last_undegraded":"2024-09-16T04:53:52.174104+0000","last_fullsized":"2024-09-16T04:53:52.174104+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:57.651124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12939,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.091472+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:52.091472+0000","last_peered":"2024-09-16T04:53:52.091472+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:52.091472+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14945,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088746+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:52.088746+0000","last_peered":"2024-09-16T04:53:52.088746+0000","last_clean":"2024-09-16T04:53:52.088746+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:52.088746+0000","last_undegraded":"2024-09-16T04:53:52.088746+0000","last_fullsized":"2024-09-16T04:53:52.088746+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.564749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":14395,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:56.237154+0000","last_change":"2024-09-16T04:53:50.515593+0000","last_active":"2024-09-16T04:53:56.237154+0000","last_peered":"2024-09-16T04:53:56.237154+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:53:56.237154+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":997,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1244}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15857,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722072,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649768,"statfs":{"total":96636764160,"available":95897362432,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731292,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640548,"statfs":{"total":96636764160,"available":95887921152,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618476,"kb_used_data":916,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753364,"statfs":{"total":96636764160,"available":96003444736,"internally_reserved":0,"allocated":937984,"data_stored":691830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722012,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649828,"statfs":{"total":96636764160,"available":95897423872,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:02.737 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:03.085 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:03.086 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:03.099 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2024-09-16T04:54:02.819382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15508,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16906,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785524,"kb_used_data":2080,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701836,"statfs":{"total":386547056640,"available":383694680064,"internally_reserved":0,"allocated":2129920,"data_stored":1145304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2850145841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001770"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12796,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090395+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:52.090395+0000","last_peered":"2024-09-16T04:53:52.090395+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:52.090395+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12784,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088878+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:53:52.088878+0000","last_peered":"2024-09-16T04:53:52.088878+0000","last_clean":"2024-09-16T04:53:52.088878+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:53:52.088878+0000","last_undegraded":"2024-09-16T04:53:52.088878+0000","last_fullsized":"2024-09-16T04:53:52.088878+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:55.640173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12795,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.093109+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:52.093109+0000","last_peered":"2024-09-16T04:53:52.093109+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:52.093109+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17239,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088591+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:52.088591+0000","last_peered":"2024-09-16T04:53:52.088591+0000","last_clean":"2024-09-16T04:53:52.088591+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:52.088591+0000","last_undegraded":"2024-09-16T04:53:52.088591+0000","last_fullsized":"2024-09-16T04:53:52.088591+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:54.603396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12898,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.088423+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:52.088423+0000","last_peered":"2024-09-16T04:53:52.088423+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:52.088423+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16813,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090691+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:52.090691+0000","last_peered":"2024-09-16T04:53:52.090691+0000","last_clean":"2024-09-16T04:53:52.090691+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:52.090691+0000","last_undegraded":"2024-09-16T04:53:52.090691+0000","last_fullsized":"2024-09-16T04:53:52.090691+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17161,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090961+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:52.090961+0000","last_peered":"2024-09-16T04:53:52.090961+0000","last_clean":"2024-09-16T04:53:52.090961+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:52.090961+0000","last_undegraded":"2024-09-16T04:53:52.090961+0000","last_fullsized":"2024-09-16T04:53:52.090961+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:53.266905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17371,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088678+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:52.088678+0000","last_peered":"2024-09-16T04:53:52.088678+0000","last_clean":"2024-09-16T04:53:52.088678+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:52.088678+0000","last_undegraded":"2024-09-16T04:53:52.088678+0000","last_fullsized":"2024-09-16T04:53:52.088678+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:54.603373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17536,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090603+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:52.090603+0000","last_peered":"2024-09-16T04:53:52.090603+0000","last_clean":"2024-09-16T04:53:52.090603+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:52.090603+0000","last_undegraded":"2024-09-16T04:53:52.090603+0000","last_fullsized":"2024-09-16T04:53:52.090603+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.114801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12889,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.093281+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:52.093281+0000","last_peered":"2024-09-16T04:53:52.093281+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:52.093281+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12647,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090697+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:52.090697+0000","last_peered":"2024-09-16T04:53:52.090697+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:52.090697+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12449,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.092832+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:52.092832+0000","last_peered":"2024-09-16T04:53:52.092832+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:52.092832+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12640,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.174104+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:53:52.174104+0000","last_peered":"2024-09-16T04:53:52.174104+0000","last_clean":"2024-09-16T04:53:52.174104+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:53:52.174104+0000","last_undegraded":"2024-09-16T04:53:52.174104+0000","last_fullsized":"2024-09-16T04:53:52.174104+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:57.651124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12939,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.091472+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:52.091472+0000","last_peered":"2024-09-16T04:53:52.091472+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:52.091472+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14945,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088746+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:52.088746+0000","last_peered":"2024-09-16T04:53:52.088746+0000","last_clean":"2024-09-16T04:53:52.088746+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:52.088746+0000","last_undegraded":"2024-09-16T04:53:52.088746+0000","last_fullsized":"2024-09-16T04:53:52.088746+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.564749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":16493,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:01.236469+0000","last_change":"2024-09-16T04:53:50.515593+0000","last_active":"2024-09-16T04:54:01.236469+0000","last_peered":"2024-09-16T04:54:01.236469+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:54:01.236469+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2046,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-195}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15508,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16906,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722072,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649768,"statfs":{"total":96636764160,"available":95897362432,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731292,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640548,"statfs":{"total":96636764160,"available":95887921152,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610148,"kb_used_data":916,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93761692,"statfs":{"total":96636764160,"available":96011972608,"internally_reserved":0,"allocated":937984,"data_stored":691830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623832783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722012,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649828,"statfs":{"total":96636764160,"available":95897423872,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:03.100 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:03.422 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:03.422 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:03.436 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-16T04:54:03.131423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15508,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16906,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785524,"kb_used_data":2080,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701836,"statfs":{"total":386547056640,"available":383694680064,"internally_reserved":0,"allocated":2129920,"data_stored":1145304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2850145841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1929,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.313418"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12796,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090395+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:52.090395+0000","last_peered":"2024-09-16T04:53:52.090395+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:52.090395+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12784,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088878+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:53:52.088878+0000","last_peered":"2024-09-16T04:53:52.088878+0000","last_clean":"2024-09-16T04:53:52.088878+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:53:52.088878+0000","last_undegraded":"2024-09-16T04:53:52.088878+0000","last_fullsized":"2024-09-16T04:53:52.088878+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:55.640173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12795,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.093109+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:52.093109+0000","last_peered":"2024-09-16T04:53:52.093109+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:52.093109+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17239,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088591+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:52.088591+0000","last_peered":"2024-09-16T04:53:52.088591+0000","last_clean":"2024-09-16T04:53:52.088591+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:52.088591+0000","last_undegraded":"2024-09-16T04:53:52.088591+0000","last_fullsized":"2024-09-16T04:53:52.088591+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:54.603396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12898,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.088423+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:52.088423+0000","last_peered":"2024-09-16T04:53:52.088423+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:52.088423+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16813,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090691+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:52.090691+0000","last_peered":"2024-09-16T04:53:52.090691+0000","last_clean":"2024-09-16T04:53:52.090691+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:52.090691+0000","last_undegraded":"2024-09-16T04:53:52.090691+0000","last_fullsized":"2024-09-16T04:53:52.090691+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17161,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090961+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:52.090961+0000","last_peered":"2024-09-16T04:53:52.090961+0000","last_clean":"2024-09-16T04:53:52.090961+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:52.090961+0000","last_undegraded":"2024-09-16T04:53:52.090961+0000","last_fullsized":"2024-09-16T04:53:52.090961+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:53.266905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17371,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088678+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:52.088678+0000","last_peered":"2024-09-16T04:53:52.088678+0000","last_clean":"2024-09-16T04:53:52.088678+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:52.088678+0000","last_undegraded":"2024-09-16T04:53:52.088678+0000","last_fullsized":"2024-09-16T04:53:52.088678+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:54.603373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17536,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090603+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:52.090603+0000","last_peered":"2024-09-16T04:53:52.090603+0000","last_clean":"2024-09-16T04:53:52.090603+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:52.090603+0000","last_undegraded":"2024-09-16T04:53:52.090603+0000","last_fullsized":"2024-09-16T04:53:52.090603+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.114801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12889,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.093281+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:52.093281+0000","last_peered":"2024-09-16T04:53:52.093281+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:52.093281+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12647,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090697+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:52.090697+0000","last_peered":"2024-09-16T04:53:52.090697+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:52.090697+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12449,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.092832+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:52.092832+0000","last_peered":"2024-09-16T04:53:52.092832+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:52.092832+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12640,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.174104+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:53:52.174104+0000","last_peered":"2024-09-16T04:53:52.174104+0000","last_clean":"2024-09-16T04:53:52.174104+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:53:52.174104+0000","last_undegraded":"2024-09-16T04:53:52.174104+0000","last_fullsized":"2024-09-16T04:53:52.174104+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:57.651124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12939,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.091472+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:52.091472+0000","last_peered":"2024-09-16T04:53:52.091472+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:52.091472+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14945,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088746+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:52.088746+0000","last_peered":"2024-09-16T04:53:52.088746+0000","last_clean":"2024-09-16T04:53:52.088746+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:52.088746+0000","last_undegraded":"2024-09-16T04:53:52.088746+0000","last_fullsized":"2024-09-16T04:53:52.088746+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.564749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":16493,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:01.236469+0000","last_change":"2024-09-16T04:53:50.515593+0000","last_active":"2024-09-16T04:54:01.236469+0000","last_peered":"2024-09-16T04:54:01.236469+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:54:01.236469+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2046,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-195}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15508,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16906,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722072,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649768,"statfs":{"total":96636764160,"available":95897362432,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731292,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640548,"statfs":{"total":96636764160,"available":95887921152,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610148,"kb_used_data":916,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93761692,"statfs":{"total":96636764160,"available":96011972608,"internally_reserved":0,"allocated":937984,"data_stored":691830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623832783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722012,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649828,"statfs":{"total":96636764160,"available":95897423872,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:03.437 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:54:03.437 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 12796, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.090395+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:53:52.090395+0000', 'last_peered': '2024-09-16T04:53:52.090395+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:53:52.090395+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:03.437 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:54:03.438 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12795, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.093109+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:53:52.093109+0000', 'last_peered': '2024-09-16T04:53:52.093109+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:53:52.093109+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:03.438 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:54:03.438 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12898, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.088423+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:53:52.088423+0000', 'last_peered': '2024-09-16T04:53:52.088423+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:53:52.088423+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:03.438 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:54:03.438 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12889, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.093281+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:53:52.093281+0000', 'last_peered': '2024-09-16T04:53:52.093281+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:53:52.093281+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:03.438 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:54:03.438 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12647, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.090697+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:53:52.090697+0000', 'last_peered': '2024-09-16T04:53:52.090697+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:53:52.090697+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:03.438 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:54:03.438 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12449, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.092832+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:53:52.092832+0000', 'last_peered': '2024-09-16T04:53:52.092832+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:53:52.092832+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:03.438 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:54:03.439 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12939, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.091472+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:53:52.091472+0000', 'last_peered': '2024-09-16T04:53:52.091472+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:53:52.091472+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:03.439 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-16T04:54:03.439 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12305", 'reported_seq': 16493, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:01.236469+0000', 'last_change': '2024-09-16T04:53:50.515593+0000', 'last_active': '2024-09-16T04:54:01.236469+0000', 'last_peered': '2024-09-16T04:54:01.236469+0000', 'last_clean': '2024-09-16T04:40:22.506308+0000', 'last_became_active': '2024-09-16T04:41:27.888194+0000', 'last_became_peered': '2024-09-16T04:41:27.888194+0000', 'last_unstale': '2024-09-16T04:54:01.236469+0000', 'last_undegraded': '2024-09-16T04:41:27.536976+0000', 'last_fullsized': '2024-09-16T04:41:27.484563+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 197, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12303, 'num_write_kb': 49220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2046, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -195}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T04:54:03.439 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:54:03.808 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:03.809 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15508/112 objects degraded (13846.429%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":856,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":49,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2852376576,"bytes_avail":383694680064,"bytes_total":386547056640,"degraded_objects":15508,"degraded_total":112,"degraded_ratio":138.46428571428572},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:54:03.821 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T04:54:03.822 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:04.126 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:04.127 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:04.140 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-16T04:54:03.131423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15508,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16906,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785524,"kb_used_data":2080,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701836,"statfs":{"total":386547056640,"available":383694680064,"internally_reserved":0,"allocated":2129920,"data_stored":1145304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2850145841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1929,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.313418"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12796,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090395+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:52.090395+0000","last_peered":"2024-09-16T04:53:52.090395+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:52.090395+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12784,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088878+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:53:52.088878+0000","last_peered":"2024-09-16T04:53:52.088878+0000","last_clean":"2024-09-16T04:53:52.088878+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:53:52.088878+0000","last_undegraded":"2024-09-16T04:53:52.088878+0000","last_fullsized":"2024-09-16T04:53:52.088878+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:55.640173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12795,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.093109+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:53:52.093109+0000","last_peered":"2024-09-16T04:53:52.093109+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:53:52.093109+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17239,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088591+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:53:52.088591+0000","last_peered":"2024-09-16T04:53:52.088591+0000","last_clean":"2024-09-16T04:53:52.088591+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:53:52.088591+0000","last_undegraded":"2024-09-16T04:53:52.088591+0000","last_fullsized":"2024-09-16T04:53:52.088591+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:54.603396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12898,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.088423+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:52.088423+0000","last_peered":"2024-09-16T04:53:52.088423+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:52.088423+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16813,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090691+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:52.090691+0000","last_peered":"2024-09-16T04:53:52.090691+0000","last_clean":"2024-09-16T04:53:52.090691+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:52.090691+0000","last_undegraded":"2024-09-16T04:53:52.090691+0000","last_fullsized":"2024-09-16T04:53:52.090691+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17161,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090961+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:53:52.090961+0000","last_peered":"2024-09-16T04:53:52.090961+0000","last_clean":"2024-09-16T04:53:52.090961+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:53:52.090961+0000","last_undegraded":"2024-09-16T04:53:52.090961+0000","last_fullsized":"2024-09-16T04:53:52.090961+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:53.266905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17371,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088678+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:53:52.088678+0000","last_peered":"2024-09-16T04:53:52.088678+0000","last_clean":"2024-09-16T04:53:52.088678+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:53:52.088678+0000","last_undegraded":"2024-09-16T04:53:52.088678+0000","last_fullsized":"2024-09-16T04:53:52.088678+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:54.603373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17536,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090603+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:52.090603+0000","last_peered":"2024-09-16T04:53:52.090603+0000","last_clean":"2024-09-16T04:53:52.090603+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:52.090603+0000","last_undegraded":"2024-09-16T04:53:52.090603+0000","last_fullsized":"2024-09-16T04:53:52.090603+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.114801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12889,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.093281+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:53:52.093281+0000","last_peered":"2024-09-16T04:53:52.093281+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:53:52.093281+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12647,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090697+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:52.090697+0000","last_peered":"2024-09-16T04:53:52.090697+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:52.090697+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12449,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.092832+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:52.092832+0000","last_peered":"2024-09-16T04:53:52.092832+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:52.092832+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12640,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.174104+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:53:52.174104+0000","last_peered":"2024-09-16T04:53:52.174104+0000","last_clean":"2024-09-16T04:53:52.174104+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:53:52.174104+0000","last_undegraded":"2024-09-16T04:53:52.174104+0000","last_fullsized":"2024-09-16T04:53:52.174104+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:57.651124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12939,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.091472+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:52.091472+0000","last_peered":"2024-09-16T04:53:52.091472+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:52.091472+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14945,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.088746+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:53:52.088746+0000","last_peered":"2024-09-16T04:53:52.088746+0000","last_clean":"2024-09-16T04:53:52.088746+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:53:52.088746+0000","last_undegraded":"2024-09-16T04:53:52.088746+0000","last_fullsized":"2024-09-16T04:53:52.088746+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.564749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":16493,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:01.236469+0000","last_change":"2024-09-16T04:53:50.515593+0000","last_active":"2024-09-16T04:54:01.236469+0000","last_peered":"2024-09-16T04:54:01.236469+0000","last_clean":"2024-09-16T04:40:22.506308+0000","last_became_active":"2024-09-16T04:41:27.888194+0000","last_became_peered":"2024-09-16T04:41:27.888194+0000","last_unstale":"2024-09-16T04:54:01.236469+0000","last_undegraded":"2024-09-16T04:41:27.536976+0000","last_fullsized":"2024-09-16T04:41:27.484563+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2046,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-195}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15508,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16906,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722072,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649768,"statfs":{"total":96636764160,"available":95897362432,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731292,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640548,"statfs":{"total":96636764160,"available":95887921152,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610148,"kb_used_data":916,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93761692,"statfs":{"total":96636764160,"available":96011972608,"internally_reserved":0,"allocated":937984,"data_stored":691830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623832783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722012,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649828,"statfs":{"total":96636764160,"available":95897423872,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:07.142 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:07.591 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:07.592 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:07.604 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-16T04:54:06.820043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17103,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793888,"kb_used_data":2128,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693472,"statfs":{"total":386547056640,"available":383686115328,"internally_reserved":0,"allocated":2179072,"data_stored":1165203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1246,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001397"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12796,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090395+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:52.090395+0000","last_peered":"2024-09-16T04:53:52.090395+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:52.090395+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12788,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133534+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:03.133534+0000","last_peered":"2024-09-16T04:54:03.133534+0000","last_clean":"2024-09-16T04:54:03.133534+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:03.133534+0000","last_undegraded":"2024-09-16T04:54:03.133534+0000","last_fullsized":"2024-09-16T04:54:03.133534+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:07.507043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12799,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.133558+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:03.133558+0000","last_peered":"2024-09-16T04:54:03.133558+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:03.133558+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17243,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.132906+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:03.132906+0000","last_peered":"2024-09-16T04:54:03.132906+0000","last_clean":"2024-09-16T04:54:03.132906+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:03.132906+0000","last_undegraded":"2024-09-16T04:54:03.132906+0000","last_fullsized":"2024-09-16T04:54:03.132906+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:05.544890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12898,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.088423+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:52.088423+0000","last_peered":"2024-09-16T04:53:52.088423+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:52.088423+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16813,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090691+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:52.090691+0000","last_peered":"2024-09-16T04:53:52.090691+0000","last_clean":"2024-09-16T04:53:52.090691+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:52.090691+0000","last_undegraded":"2024-09-16T04:53:52.090691+0000","last_fullsized":"2024-09-16T04:53:52.090691+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17165,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.138447+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:03.138447+0000","last_peered":"2024-09-16T04:54:03.138447+0000","last_clean":"2024-09-16T04:54:03.138447+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:03.138447+0000","last_undegraded":"2024-09-16T04:54:03.138447+0000","last_fullsized":"2024-09-16T04:54:03.138447+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:04.117558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17375,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.132960+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:03.132960+0000","last_peered":"2024-09-16T04:54:03.132960+0000","last_clean":"2024-09-16T04:54:03.132960+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:03.132960+0000","last_undegraded":"2024-09-16T04:54:03.132960+0000","last_fullsized":"2024-09-16T04:54:03.132960+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:05.544876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17536,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090603+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:52.090603+0000","last_peered":"2024-09-16T04:53:52.090603+0000","last_clean":"2024-09-16T04:53:52.090603+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:52.090603+0000","last_undegraded":"2024-09-16T04:53:52.090603+0000","last_fullsized":"2024-09-16T04:53:52.090603+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.114801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12893,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.133664+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:54:03.133664+0000","last_peered":"2024-09-16T04:54:03.133664+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:03.133664+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12647,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090697+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:52.090697+0000","last_peered":"2024-09-16T04:53:52.090697+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:52.090697+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12449,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.092832+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:52.092832+0000","last_peered":"2024-09-16T04:53:52.092832+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:52.092832+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12644,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133023+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:03.133023+0000","last_peered":"2024-09-16T04:54:03.133023+0000","last_clean":"2024-09-16T04:54:03.133023+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:03.133023+0000","last_undegraded":"2024-09-16T04:54:03.133023+0000","last_fullsized":"2024-09-16T04:54:03.133023+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:03.608762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12939,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.091472+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:52.091472+0000","last_peered":"2024-09-16T04:53:52.091472+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:52.091472+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14949,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133189+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:03.133189+0000","last_peered":"2024-09-16T04:54:03.133189+0000","last_clean":"2024-09-16T04:54:03.133189+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:03.133189+0000","last_undegraded":"2024-09-16T04:54:03.133189+0000","last_fullsized":"2024-09-16T04:54:03.133189+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:03.608747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12419,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.226235+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:03.226235+0000","last_peered":"2024-09-16T04:54:03.226235+0000","last_clean":"2024-09-16T04:54:03.226235+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:03.226235+0000","last_undegraded":"2024-09-16T04:54:03.226235+0000","last_fullsized":"2024-09-16T04:54:03.226235+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:08.484177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17103,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722072,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649768,"statfs":{"total":96636764160,"available":95897362432,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731304,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640536,"statfs":{"total":96636764160,"available":95887908864,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618488,"kb_used_data":932,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753352,"statfs":{"total":96636764160,"available":96003432448,"internally_reserved":0,"allocated":954368,"data_stored":698463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722024,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649816,"statfs":{"total":96636764160,"available":95897411584,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:07.605 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:07.937 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:07.937 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:07.950 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-16T04:54:06.820043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17103,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793888,"kb_used_data":2128,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693472,"statfs":{"total":386547056640,"available":383686115328,"internally_reserved":0,"allocated":2179072,"data_stored":1165203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1246,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001397"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12796,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090395+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:52.090395+0000","last_peered":"2024-09-16T04:53:52.090395+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:52.090395+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12788,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133534+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:03.133534+0000","last_peered":"2024-09-16T04:54:03.133534+0000","last_clean":"2024-09-16T04:54:03.133534+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:03.133534+0000","last_undegraded":"2024-09-16T04:54:03.133534+0000","last_fullsized":"2024-09-16T04:54:03.133534+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:07.507043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12799,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.133558+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:03.133558+0000","last_peered":"2024-09-16T04:54:03.133558+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:03.133558+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17243,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.132906+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:03.132906+0000","last_peered":"2024-09-16T04:54:03.132906+0000","last_clean":"2024-09-16T04:54:03.132906+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:03.132906+0000","last_undegraded":"2024-09-16T04:54:03.132906+0000","last_fullsized":"2024-09-16T04:54:03.132906+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:05.544890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12898,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.088423+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:52.088423+0000","last_peered":"2024-09-16T04:53:52.088423+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:52.088423+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16813,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090691+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:52.090691+0000","last_peered":"2024-09-16T04:53:52.090691+0000","last_clean":"2024-09-16T04:53:52.090691+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:52.090691+0000","last_undegraded":"2024-09-16T04:53:52.090691+0000","last_fullsized":"2024-09-16T04:53:52.090691+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17165,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.138447+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:03.138447+0000","last_peered":"2024-09-16T04:54:03.138447+0000","last_clean":"2024-09-16T04:54:03.138447+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:03.138447+0000","last_undegraded":"2024-09-16T04:54:03.138447+0000","last_fullsized":"2024-09-16T04:54:03.138447+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:04.117558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17375,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.132960+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:03.132960+0000","last_peered":"2024-09-16T04:54:03.132960+0000","last_clean":"2024-09-16T04:54:03.132960+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:03.132960+0000","last_undegraded":"2024-09-16T04:54:03.132960+0000","last_fullsized":"2024-09-16T04:54:03.132960+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:05.544876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17536,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090603+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:52.090603+0000","last_peered":"2024-09-16T04:53:52.090603+0000","last_clean":"2024-09-16T04:53:52.090603+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:52.090603+0000","last_undegraded":"2024-09-16T04:53:52.090603+0000","last_fullsized":"2024-09-16T04:53:52.090603+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.114801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12893,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.133664+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:54:03.133664+0000","last_peered":"2024-09-16T04:54:03.133664+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:03.133664+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12647,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090697+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:52.090697+0000","last_peered":"2024-09-16T04:53:52.090697+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:52.090697+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12449,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.092832+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:52.092832+0000","last_peered":"2024-09-16T04:53:52.092832+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:52.092832+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12644,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133023+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:03.133023+0000","last_peered":"2024-09-16T04:54:03.133023+0000","last_clean":"2024-09-16T04:54:03.133023+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:03.133023+0000","last_undegraded":"2024-09-16T04:54:03.133023+0000","last_fullsized":"2024-09-16T04:54:03.133023+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:03.608762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12939,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.091472+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:52.091472+0000","last_peered":"2024-09-16T04:53:52.091472+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:52.091472+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14949,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133189+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:03.133189+0000","last_peered":"2024-09-16T04:54:03.133189+0000","last_clean":"2024-09-16T04:54:03.133189+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:03.133189+0000","last_undegraded":"2024-09-16T04:54:03.133189+0000","last_fullsized":"2024-09-16T04:54:03.133189+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:03.608747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12419,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.226235+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:03.226235+0000","last_peered":"2024-09-16T04:54:03.226235+0000","last_clean":"2024-09-16T04:54:03.226235+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:03.226235+0000","last_undegraded":"2024-09-16T04:54:03.226235+0000","last_fullsized":"2024-09-16T04:54:03.226235+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:08.484177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17103,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722072,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649768,"statfs":{"total":96636764160,"available":95897362432,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731304,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640536,"statfs":{"total":96636764160,"available":95887908864,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618488,"kb_used_data":932,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753352,"statfs":{"total":96636764160,"available":96003432448,"internally_reserved":0,"allocated":954368,"data_stored":698463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722024,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649816,"statfs":{"total":96636764160,"available":95897411584,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:07.951 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:54:07.951 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 12796, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.090395+0000', 'last_change': '2024-09-16T04:41:27.824955+0000', 'last_active': '2024-09-16T04:53:52.090395+0000', 'last_peered': '2024-09-16T04:53:52.090395+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:53:52.090395+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:07.951 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:54:07.951 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12799, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:03.133558+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:54:03.133558+0000', 'last_peered': '2024-09-16T04:54:03.133558+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:54:03.133558+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:07.951 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:54:07.951 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12898, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.088423+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:53:52.088423+0000', 'last_peered': '2024-09-16T04:53:52.088423+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:53:52.088423+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:07.952 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:54:07.952 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12893, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:03.133664+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:54:03.133664+0000', 'last_peered': '2024-09-16T04:54:03.133664+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:54:03.133664+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:07.952 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:54:07.952 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12647, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.090697+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:53:52.090697+0000', 'last_peered': '2024-09-16T04:53:52.090697+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:53:52.090697+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:07.952 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:54:07.952 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12449, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.092832+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:53:52.092832+0000', 'last_peered': '2024-09-16T04:53:52.092832+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:53:52.092832+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:07.952 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:54:07.952 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12939, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:53:52.091472+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:53:52.091472+0000', 'last_peered': '2024-09-16T04:53:52.091472+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:53:52.091472+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:07.953 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:54:08.323 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:08.323 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15311/112 objects degraded (13670.536%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":860,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":49,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2860941312,"bytes_avail":383686115328,"bytes_total":386547056640,"degraded_objects":15311,"degraded_total":112,"degraded_ratio":136.70535714285714,"recovering_objects_per_sec":155,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1246,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:54:08.337 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:54:08.340 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:08.672 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:08.672 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:08.683 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-16T04:54:06.820043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17103,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793888,"kb_used_data":2128,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693472,"statfs":{"total":386547056640,"available":383686115328,"internally_reserved":0,"allocated":2179072,"data_stored":1165203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1246,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001397"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":12796,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090395+0000","last_change":"2024-09-16T04:41:27.824955+0000","last_active":"2024-09-16T04:53:52.090395+0000","last_peered":"2024-09-16T04:53:52.090395+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:53:52.090395+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12788,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133534+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:03.133534+0000","last_peered":"2024-09-16T04:54:03.133534+0000","last_clean":"2024-09-16T04:54:03.133534+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:03.133534+0000","last_undegraded":"2024-09-16T04:54:03.133534+0000","last_fullsized":"2024-09-16T04:54:03.133534+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:07.507043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12799,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.133558+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:03.133558+0000","last_peered":"2024-09-16T04:54:03.133558+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:03.133558+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17243,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.132906+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:03.132906+0000","last_peered":"2024-09-16T04:54:03.132906+0000","last_clean":"2024-09-16T04:54:03.132906+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:03.132906+0000","last_undegraded":"2024-09-16T04:54:03.132906+0000","last_fullsized":"2024-09-16T04:54:03.132906+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:05.544890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12898,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.088423+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:53:52.088423+0000","last_peered":"2024-09-16T04:53:52.088423+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:53:52.088423+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16813,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090691+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:53:52.090691+0000","last_peered":"2024-09-16T04:53:52.090691+0000","last_clean":"2024-09-16T04:53:52.090691+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:53:52.090691+0000","last_undegraded":"2024-09-16T04:53:52.090691+0000","last_fullsized":"2024-09-16T04:53:52.090691+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17165,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.138447+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:03.138447+0000","last_peered":"2024-09-16T04:54:03.138447+0000","last_clean":"2024-09-16T04:54:03.138447+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:03.138447+0000","last_undegraded":"2024-09-16T04:54:03.138447+0000","last_fullsized":"2024-09-16T04:54:03.138447+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:04.117558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17375,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.132960+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:03.132960+0000","last_peered":"2024-09-16T04:54:03.132960+0000","last_clean":"2024-09-16T04:54:03.132960+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:03.132960+0000","last_undegraded":"2024-09-16T04:54:03.132960+0000","last_fullsized":"2024-09-16T04:54:03.132960+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:05.544876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17536,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:53:52.090603+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:53:52.090603+0000","last_peered":"2024-09-16T04:53:52.090603+0000","last_clean":"2024-09-16T04:53:52.090603+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:53:52.090603+0000","last_undegraded":"2024-09-16T04:53:52.090603+0000","last_fullsized":"2024-09-16T04:53:52.090603+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:53:52.114801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12893,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.133664+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:54:03.133664+0000","last_peered":"2024-09-16T04:54:03.133664+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:03.133664+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12647,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.090697+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:53:52.090697+0000","last_peered":"2024-09-16T04:53:52.090697+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:53:52.090697+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12449,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.092832+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:53:52.092832+0000","last_peered":"2024-09-16T04:53:52.092832+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:53:52.092832+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12644,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133023+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:03.133023+0000","last_peered":"2024-09-16T04:54:03.133023+0000","last_clean":"2024-09-16T04:54:03.133023+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:03.133023+0000","last_undegraded":"2024-09-16T04:54:03.133023+0000","last_fullsized":"2024-09-16T04:54:03.133023+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:03.608762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12939,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:53:52.091472+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:53:52.091472+0000","last_peered":"2024-09-16T04:53:52.091472+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:53:52.091472+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14949,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133189+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:03.133189+0000","last_peered":"2024-09-16T04:54:03.133189+0000","last_clean":"2024-09-16T04:54:03.133189+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:03.133189+0000","last_undegraded":"2024-09-16T04:54:03.133189+0000","last_fullsized":"2024-09-16T04:54:03.133189+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:03.608747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12419,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.226235+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:03.226235+0000","last_peered":"2024-09-16T04:54:03.226235+0000","last_clean":"2024-09-16T04:54:03.226235+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:03.226235+0000","last_undegraded":"2024-09-16T04:54:03.226235+0000","last_fullsized":"2024-09-16T04:54:03.226235+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:08.484177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17103,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722072,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649768,"statfs":{"total":96636764160,"available":95897362432,"internally_reserved":0,"allocated":397312,"data_stored":151158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731304,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640536,"statfs":{"total":96636764160,"available":95887908864,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618488,"kb_used_data":932,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753352,"statfs":{"total":96636764160,"available":96003432448,"internally_reserved":0,"allocated":954368,"data_stored":698463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722024,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649816,"statfs":{"total":96636764160,"available":95897411584,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:11.685 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:12.052 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:12.053 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:12.066 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-16T04:54:10.820705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17918,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785584,"kb_used_data":2144,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701776,"statfs":{"total":386547056640,"available":383694618624,"internally_reserved":0,"allocated":2195456,"data_stored":1171836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2850145841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.693546"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":14431,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:06.914353+0000","last_change":"2024-09-16T04:54:01.997093+0000","last_active":"2024-09-16T04:54:06.914353+0000","last_peered":"2024-09-16T04:54:06.914353+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:54:06.914353+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":848,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1371}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12788,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133534+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:03.133534+0000","last_peered":"2024-09-16T04:54:03.133534+0000","last_clean":"2024-09-16T04:54:03.133534+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:03.133534+0000","last_undegraded":"2024-09-16T04:54:03.133534+0000","last_fullsized":"2024-09-16T04:54:03.133534+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:07.507043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12799,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.133558+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:03.133558+0000","last_peered":"2024-09-16T04:54:03.133558+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:03.133558+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17243,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.132906+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:03.132906+0000","last_peered":"2024-09-16T04:54:03.132906+0000","last_clean":"2024-09-16T04:54:03.132906+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:03.132906+0000","last_undegraded":"2024-09-16T04:54:03.132906+0000","last_fullsized":"2024-09-16T04:54:03.132906+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:05.544890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12902,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.134907+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:03.134907+0000","last_peered":"2024-09-16T04:54:03.134907+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:03.134907+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16817,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.135943+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:03.135943+0000","last_peered":"2024-09-16T04:54:03.135943+0000","last_clean":"2024-09-16T04:54:03.135943+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:03.135943+0000","last_undegraded":"2024-09-16T04:54:03.135943+0000","last_fullsized":"2024-09-16T04:54:03.135943+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17165,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.138447+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:03.138447+0000","last_peered":"2024-09-16T04:54:03.138447+0000","last_clean":"2024-09-16T04:54:03.138447+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:03.138447+0000","last_undegraded":"2024-09-16T04:54:03.138447+0000","last_fullsized":"2024-09-16T04:54:03.138447+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:04.117558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17375,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.132960+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:03.132960+0000","last_peered":"2024-09-16T04:54:03.132960+0000","last_clean":"2024-09-16T04:54:03.132960+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:03.132960+0000","last_undegraded":"2024-09-16T04:54:03.132960+0000","last_fullsized":"2024-09-16T04:54:03.132960+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:05.544876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17540,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.135852+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:03.135852+0000","last_peered":"2024-09-16T04:54:03.135852+0000","last_clean":"2024-09-16T04:54:03.135852+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:03.135852+0000","last_undegraded":"2024-09-16T04:54:03.135852+0000","last_fullsized":"2024-09-16T04:54:03.135852+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12893,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.133664+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:54:03.133664+0000","last_peered":"2024-09-16T04:54:03.133664+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:03.133664+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12651,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.136221+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:03.136221+0000","last_peered":"2024-09-16T04:54:03.136221+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:03.136221+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12453,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.136618+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:54:03.136618+0000","last_peered":"2024-09-16T04:54:03.136618+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:03.136618+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12644,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133023+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:03.133023+0000","last_peered":"2024-09-16T04:54:03.133023+0000","last_clean":"2024-09-16T04:54:03.133023+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:03.133023+0000","last_undegraded":"2024-09-16T04:54:03.133023+0000","last_fullsized":"2024-09-16T04:54:03.133023+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:03.608762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12943,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.134988+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:03.134988+0000","last_peered":"2024-09-16T04:54:03.134988+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:03.134988+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14949,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133189+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:03.133189+0000","last_peered":"2024-09-16T04:54:03.133189+0000","last_clean":"2024-09-16T04:54:03.133189+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:03.133189+0000","last_undegraded":"2024-09-16T04:54:03.133189+0000","last_fullsized":"2024-09-16T04:54:03.133189+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:03.608747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12419,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.226235+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:03.226235+0000","last_peered":"2024-09-16T04:54:03.226235+0000","last_clean":"2024-09-16T04:54:03.226235+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:03.226235+0000","last_undegraded":"2024-09-16T04:54:03.226235+0000","last_fullsized":"2024-09-16T04:54:03.226235+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:08.484177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17918,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731304,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640536,"statfs":{"total":96636764160,"available":95887908864,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610164,"kb_used_data":932,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93761676,"statfs":{"total":96636764160,"available":96011956224,"internally_reserved":0,"allocated":954368,"data_stored":698463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623832783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722024,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649816,"statfs":{"total":96636764160,"available":95897411584,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:12.067 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:12.364 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:12.364 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:12.376 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-16T04:54:10.820705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17918,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785584,"kb_used_data":2144,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701776,"statfs":{"total":386547056640,"available":383694618624,"internally_reserved":0,"allocated":2195456,"data_stored":1171836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2850145841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.693546"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":14431,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:06.914353+0000","last_change":"2024-09-16T04:54:01.997093+0000","last_active":"2024-09-16T04:54:06.914353+0000","last_peered":"2024-09-16T04:54:06.914353+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:54:06.914353+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":848,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1371}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12788,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133534+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:03.133534+0000","last_peered":"2024-09-16T04:54:03.133534+0000","last_clean":"2024-09-16T04:54:03.133534+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:03.133534+0000","last_undegraded":"2024-09-16T04:54:03.133534+0000","last_fullsized":"2024-09-16T04:54:03.133534+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:07.507043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12799,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.133558+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:03.133558+0000","last_peered":"2024-09-16T04:54:03.133558+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:03.133558+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17243,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.132906+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:03.132906+0000","last_peered":"2024-09-16T04:54:03.132906+0000","last_clean":"2024-09-16T04:54:03.132906+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:03.132906+0000","last_undegraded":"2024-09-16T04:54:03.132906+0000","last_fullsized":"2024-09-16T04:54:03.132906+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:05.544890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12902,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.134907+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:03.134907+0000","last_peered":"2024-09-16T04:54:03.134907+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:03.134907+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16817,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.135943+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:03.135943+0000","last_peered":"2024-09-16T04:54:03.135943+0000","last_clean":"2024-09-16T04:54:03.135943+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:03.135943+0000","last_undegraded":"2024-09-16T04:54:03.135943+0000","last_fullsized":"2024-09-16T04:54:03.135943+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17165,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.138447+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:03.138447+0000","last_peered":"2024-09-16T04:54:03.138447+0000","last_clean":"2024-09-16T04:54:03.138447+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:03.138447+0000","last_undegraded":"2024-09-16T04:54:03.138447+0000","last_fullsized":"2024-09-16T04:54:03.138447+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:04.117558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17375,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.132960+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:03.132960+0000","last_peered":"2024-09-16T04:54:03.132960+0000","last_clean":"2024-09-16T04:54:03.132960+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:03.132960+0000","last_undegraded":"2024-09-16T04:54:03.132960+0000","last_fullsized":"2024-09-16T04:54:03.132960+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:05.544876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17540,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.135852+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:03.135852+0000","last_peered":"2024-09-16T04:54:03.135852+0000","last_clean":"2024-09-16T04:54:03.135852+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:03.135852+0000","last_undegraded":"2024-09-16T04:54:03.135852+0000","last_fullsized":"2024-09-16T04:54:03.135852+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12893,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.133664+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:54:03.133664+0000","last_peered":"2024-09-16T04:54:03.133664+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:03.133664+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12651,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.136221+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:03.136221+0000","last_peered":"2024-09-16T04:54:03.136221+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:03.136221+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12453,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.136618+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:54:03.136618+0000","last_peered":"2024-09-16T04:54:03.136618+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:03.136618+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12644,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133023+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:03.133023+0000","last_peered":"2024-09-16T04:54:03.133023+0000","last_clean":"2024-09-16T04:54:03.133023+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:03.133023+0000","last_undegraded":"2024-09-16T04:54:03.133023+0000","last_fullsized":"2024-09-16T04:54:03.133023+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:03.608762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12943,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.134988+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:03.134988+0000","last_peered":"2024-09-16T04:54:03.134988+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:03.134988+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14949,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133189+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:03.133189+0000","last_peered":"2024-09-16T04:54:03.133189+0000","last_clean":"2024-09-16T04:54:03.133189+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:03.133189+0000","last_undegraded":"2024-09-16T04:54:03.133189+0000","last_fullsized":"2024-09-16T04:54:03.133189+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:03.608747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12419,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.226235+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:03.226235+0000","last_peered":"2024-09-16T04:54:03.226235+0000","last_clean":"2024-09-16T04:54:03.226235+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:03.226235+0000","last_undegraded":"2024-09-16T04:54:03.226235+0000","last_fullsized":"2024-09-16T04:54:03.226235+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:08.484177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17918,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731304,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640536,"statfs":{"total":96636764160,"available":95887908864,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610164,"kb_used_data":932,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93761676,"statfs":{"total":96636764160,"available":96011956224,"internally_reserved":0,"allocated":954368,"data_stored":698463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623832783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722024,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649816,"statfs":{"total":96636764160,"available":95897411584,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:12.377 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:54:12.378 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 14431, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:06.914353+0000', 'last_change': '2024-09-16T04:54:01.997093+0000', 'last_active': '2024-09-16T04:54:06.914353+0000', 'last_peered': '2024-09-16T04:54:06.914353+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:54:06.914353+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1371, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 848, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1371}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:12.378 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:54:12.378 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12799, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:03.133558+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:54:03.133558+0000', 'last_peered': '2024-09-16T04:54:03.133558+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:54:03.133558+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:12.378 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:54:12.378 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12902, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:03.134907+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:54:03.134907+0000', 'last_peered': '2024-09-16T04:54:03.134907+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:54:03.134907+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:12.378 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:54:12.378 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 12893, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:03.133664+0000', 'last_change': '2024-09-16T04:41:27.846533+0000', 'last_active': '2024-09-16T04:54:03.133664+0000', 'last_peered': '2024-09-16T04:54:03.133664+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:54:03.133664+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:12.378 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:54:12.378 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12651, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:03.136221+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:54:03.136221+0000', 'last_peered': '2024-09-16T04:54:03.136221+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:54:03.136221+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:12.379 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:54:12.379 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12453, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:03.136618+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:54:03.136618+0000', 'last_peered': '2024-09-16T04:54:03.136618+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:54:03.136618+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:12.379 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:54:12.379 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12943, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:03.134988+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:54:03.134988+0000', 'last_peered': '2024-09-16T04:54:03.134988+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:54:03.134988+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:12.379 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:54:12.746 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:12.747 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14496/112 objects degraded (12942.857%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":865,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":49,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2852438016,"bytes_avail":383694618624,"bytes_total":386547056640,"degraded_objects":14496,"degraded_total":112,"degraded_ratio":129.42857142857142,"recovering_objects_per_sec":116,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1012,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:54:12.760 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:54:12.761 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:13.056 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:13.057 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:13.069 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-09-16T04:54:12.821091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18868,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785584,"kb_used_data":2144,"kb_used_omap":14,"kb_used_meta":2783345,"kb_avail":374701776,"statfs":{"total":386547056640,"available":383694618624,"internally_reserved":0,"allocated":2195456,"data_stored":1171836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2850145841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1962,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001709"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":16331,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:11.916711+0000","last_change":"2024-09-16T04:54:01.997093+0000","last_active":"2024-09-16T04:54:11.916711+0000","last_peered":"2024-09-16T04:54:11.916711+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:54:11.916711+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1798,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12788,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133534+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:03.133534+0000","last_peered":"2024-09-16T04:54:03.133534+0000","last_clean":"2024-09-16T04:54:03.133534+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:03.133534+0000","last_undegraded":"2024-09-16T04:54:03.133534+0000","last_fullsized":"2024-09-16T04:54:03.133534+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:07.507043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12799,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.133558+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:03.133558+0000","last_peered":"2024-09-16T04:54:03.133558+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:03.133558+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17243,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.132906+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:03.132906+0000","last_peered":"2024-09-16T04:54:03.132906+0000","last_clean":"2024-09-16T04:54:03.132906+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:03.132906+0000","last_undegraded":"2024-09-16T04:54:03.132906+0000","last_fullsized":"2024-09-16T04:54:03.132906+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:05.544890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12902,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.134907+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:03.134907+0000","last_peered":"2024-09-16T04:54:03.134907+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:03.134907+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16818,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.135987+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:03.135987+0000","last_peered":"2024-09-16T04:54:03.135987+0000","last_clean":"2024-09-16T04:54:03.135987+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:03.135987+0000","last_undegraded":"2024-09-16T04:54:03.135987+0000","last_fullsized":"2024-09-16T04:54:03.135987+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17165,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.138447+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:03.138447+0000","last_peered":"2024-09-16T04:54:03.138447+0000","last_clean":"2024-09-16T04:54:03.138447+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:03.138447+0000","last_undegraded":"2024-09-16T04:54:03.138447+0000","last_fullsized":"2024-09-16T04:54:03.138447+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:04.117558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17375,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.132960+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:03.132960+0000","last_peered":"2024-09-16T04:54:03.132960+0000","last_clean":"2024-09-16T04:54:03.132960+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:03.132960+0000","last_undegraded":"2024-09-16T04:54:03.132960+0000","last_fullsized":"2024-09-16T04:54:03.132960+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:05.544876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17540,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.135852+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:03.135852+0000","last_peered":"2024-09-16T04:54:03.135852+0000","last_clean":"2024-09-16T04:54:03.135852+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:03.135852+0000","last_undegraded":"2024-09-16T04:54:03.135852+0000","last_fullsized":"2024-09-16T04:54:03.135852+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":12893,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.133664+0000","last_change":"2024-09-16T04:41:27.846533+0000","last_active":"2024-09-16T04:54:03.133664+0000","last_peered":"2024-09-16T04:54:03.133664+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:03.133664+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12651,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.136221+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:03.136221+0000","last_peered":"2024-09-16T04:54:03.136221+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:03.136221+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12453,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.136618+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:54:03.136618+0000","last_peered":"2024-09-16T04:54:03.136618+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:03.136618+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12644,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133023+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:03.133023+0000","last_peered":"2024-09-16T04:54:03.133023+0000","last_clean":"2024-09-16T04:54:03.133023+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:03.133023+0000","last_undegraded":"2024-09-16T04:54:03.133023+0000","last_fullsized":"2024-09-16T04:54:03.133023+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:03.608762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12943,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.134988+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:03.134988+0000","last_peered":"2024-09-16T04:54:03.134988+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:03.134988+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14949,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.133189+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:03.133189+0000","last_peered":"2024-09-16T04:54:03.133189+0000","last_clean":"2024-09-16T04:54:03.133189+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:03.133189+0000","last_undegraded":"2024-09-16T04:54:03.133189+0000","last_fullsized":"2024-09-16T04:54:03.133189+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:03.608747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12419,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.226235+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:03.226235+0000","last_peered":"2024-09-16T04:54:03.226235+0000","last_clean":"2024-09-16T04:54:03.226235+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:03.226235+0000","last_undegraded":"2024-09-16T04:54:03.226235+0000","last_fullsized":"2024-09-16T04:54:03.226235+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:08.484177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18868,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731304,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640536,"statfs":{"total":96636764160,"available":95887908864,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610164,"kb_used_data":932,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93761676,"statfs":{"total":96636764160,"available":96011956224,"internally_reserved":0,"allocated":954368,"data_stored":698463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623832783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722024,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649816,"statfs":{"total":96636764160,"available":95897411584,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:16.071 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:16.381 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:16.381 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:16.394 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-09-16T04:54:15.173383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19045,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793916,"kb_used_data":2152,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693444,"statfs":{"total":386547056640,"available":383686086656,"internally_reserved":0,"allocated":2203648,"data_stored":1170224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.353339"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":16331,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:11.916711+0000","last_change":"2024-09-16T04:54:01.997093+0000","last_active":"2024-09-16T04:54:11.916711+0000","last_peered":"2024-09-16T04:54:11.916711+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:54:11.916711+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1798,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12790,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172610+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:14.172610+0000","last_peered":"2024-09-16T04:54:14.172610+0000","last_clean":"2024-09-16T04:54:14.172610+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:14.172610+0000","last_undegraded":"2024-09-16T04:54:14.172610+0000","last_fullsized":"2024-09-16T04:54:14.172610+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:18.376544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12801,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:14.174341+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:14.174341+0000","last_peered":"2024-09-16T04:54:14.174341+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:14.174341+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17245,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.170926+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:14.170926+0000","last_peered":"2024-09-16T04:54:14.170926+0000","last_clean":"2024-09-16T04:54:14.170926+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:14.170926+0000","last_undegraded":"2024-09-16T04:54:14.170926+0000","last_fullsized":"2024-09-16T04:54:14.170926+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12902,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.134907+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:03.134907+0000","last_peered":"2024-09-16T04:54:03.134907+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:03.134907+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16818,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.135987+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:03.135987+0000","last_peered":"2024-09-16T04:54:03.135987+0000","last_clean":"2024-09-16T04:54:03.135987+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:03.135987+0000","last_undegraded":"2024-09-16T04:54:03.135987+0000","last_fullsized":"2024-09-16T04:54:03.135987+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17165,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.138447+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:03.138447+0000","last_peered":"2024-09-16T04:54:03.138447+0000","last_clean":"2024-09-16T04:54:03.138447+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:03.138447+0000","last_undegraded":"2024-09-16T04:54:03.138447+0000","last_fullsized":"2024-09-16T04:54:03.138447+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:04.117558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17377,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172200+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:14.172200+0000","last_peered":"2024-09-16T04:54:14.172200+0000","last_clean":"2024-09-16T04:54:14.172200+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:14.172200+0000","last_undegraded":"2024-09-16T04:54:14.172200+0000","last_fullsized":"2024-09-16T04:54:14.172200+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17540,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.135852+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:03.135852+0000","last_peered":"2024-09-16T04:54:03.135852+0000","last_clean":"2024-09-16T04:54:03.135852+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:03.135852+0000","last_undegraded":"2024-09-16T04:54:03.135852+0000","last_fullsized":"2024-09-16T04:54:03.135852+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":13250,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.011747+0000","last_change":"2024-09-16T04:54:13.785237+0000","last_active":"2024-09-16T04:54:15.011747+0000","last_peered":"2024-09-16T04:54:15.011747+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:15.011747+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2012}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12651,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.136221+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:03.136221+0000","last_peered":"2024-09-16T04:54:03.136221+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:03.136221+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12453,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.136618+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:54:03.136618+0000","last_peered":"2024-09-16T04:54:03.136618+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:03.136618+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.171112+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:14.171112+0000","last_peered":"2024-09-16T04:54:14.171112+0000","last_clean":"2024-09-16T04:54:14.171112+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:14.171112+0000","last_undegraded":"2024-09-16T04:54:14.171112+0000","last_fullsized":"2024-09-16T04:54:14.171112+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12943,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.134988+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:03.134988+0000","last_peered":"2024-09-16T04:54:03.134988+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:03.134988+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14951,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172083+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:14.172083+0000","last_peered":"2024-09-16T04:54:14.172083+0000","last_clean":"2024-09-16T04:54:14.172083+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:14.172083+0000","last_undegraded":"2024-09-16T04:54:14.172083+0000","last_fullsized":"2024-09-16T04:54:14.172083+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12421,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172292+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:14.172292+0000","last_peered":"2024-09-16T04:54:14.172292+0000","last_clean":"2024-09-16T04:54:14.172292+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:14.172292+0000","last_undegraded":"2024-09-16T04:54:14.172292+0000","last_fullsized":"2024-09-16T04:54:14.172292+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:14.382090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19045,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731312,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640528,"statfs":{"total":96636764160,"available":95887900672,"internally_reserved":0,"allocated":421888,"data_stored":161081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618488,"kb_used_data":932,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753352,"statfs":{"total":96636764160,"available":96003432448,"internally_reserved":0,"allocated":954368,"data_stored":693561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722024,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649816,"statfs":{"total":96636764160,"available":95897411584,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:16.395 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:16.716 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:16.717 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:16.729 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-09-16T04:54:15.173383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19045,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793916,"kb_used_data":2152,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693444,"statfs":{"total":386547056640,"available":383686086656,"internally_reserved":0,"allocated":2203648,"data_stored":1170224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.353339"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":16331,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:11.916711+0000","last_change":"2024-09-16T04:54:01.997093+0000","last_active":"2024-09-16T04:54:11.916711+0000","last_peered":"2024-09-16T04:54:11.916711+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:54:11.916711+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1798,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12790,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172610+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:14.172610+0000","last_peered":"2024-09-16T04:54:14.172610+0000","last_clean":"2024-09-16T04:54:14.172610+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:14.172610+0000","last_undegraded":"2024-09-16T04:54:14.172610+0000","last_fullsized":"2024-09-16T04:54:14.172610+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:18.376544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12801,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:14.174341+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:14.174341+0000","last_peered":"2024-09-16T04:54:14.174341+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:14.174341+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17245,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.170926+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:14.170926+0000","last_peered":"2024-09-16T04:54:14.170926+0000","last_clean":"2024-09-16T04:54:14.170926+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:14.170926+0000","last_undegraded":"2024-09-16T04:54:14.170926+0000","last_fullsized":"2024-09-16T04:54:14.170926+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12902,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.134907+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:03.134907+0000","last_peered":"2024-09-16T04:54:03.134907+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:03.134907+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16818,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.135987+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:03.135987+0000","last_peered":"2024-09-16T04:54:03.135987+0000","last_clean":"2024-09-16T04:54:03.135987+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:03.135987+0000","last_undegraded":"2024-09-16T04:54:03.135987+0000","last_fullsized":"2024-09-16T04:54:03.135987+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17165,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.138447+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:03.138447+0000","last_peered":"2024-09-16T04:54:03.138447+0000","last_clean":"2024-09-16T04:54:03.138447+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:03.138447+0000","last_undegraded":"2024-09-16T04:54:03.138447+0000","last_fullsized":"2024-09-16T04:54:03.138447+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:04.117558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17377,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172200+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:14.172200+0000","last_peered":"2024-09-16T04:54:14.172200+0000","last_clean":"2024-09-16T04:54:14.172200+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:14.172200+0000","last_undegraded":"2024-09-16T04:54:14.172200+0000","last_fullsized":"2024-09-16T04:54:14.172200+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17540,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.135852+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:03.135852+0000","last_peered":"2024-09-16T04:54:03.135852+0000","last_clean":"2024-09-16T04:54:03.135852+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:03.135852+0000","last_undegraded":"2024-09-16T04:54:03.135852+0000","last_fullsized":"2024-09-16T04:54:03.135852+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":13250,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.011747+0000","last_change":"2024-09-16T04:54:13.785237+0000","last_active":"2024-09-16T04:54:15.011747+0000","last_peered":"2024-09-16T04:54:15.011747+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:15.011747+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2012}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12651,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.136221+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:03.136221+0000","last_peered":"2024-09-16T04:54:03.136221+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:03.136221+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12453,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.136618+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:54:03.136618+0000","last_peered":"2024-09-16T04:54:03.136618+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:03.136618+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.171112+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:14.171112+0000","last_peered":"2024-09-16T04:54:14.171112+0000","last_clean":"2024-09-16T04:54:14.171112+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:14.171112+0000","last_undegraded":"2024-09-16T04:54:14.171112+0000","last_fullsized":"2024-09-16T04:54:14.171112+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12943,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.134988+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:03.134988+0000","last_peered":"2024-09-16T04:54:03.134988+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:03.134988+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14951,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172083+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:14.172083+0000","last_peered":"2024-09-16T04:54:14.172083+0000","last_clean":"2024-09-16T04:54:14.172083+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:14.172083+0000","last_undegraded":"2024-09-16T04:54:14.172083+0000","last_fullsized":"2024-09-16T04:54:14.172083+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12421,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172292+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:14.172292+0000","last_peered":"2024-09-16T04:54:14.172292+0000","last_clean":"2024-09-16T04:54:14.172292+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:14.172292+0000","last_undegraded":"2024-09-16T04:54:14.172292+0000","last_fullsized":"2024-09-16T04:54:14.172292+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:14.382090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19045,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731312,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640528,"statfs":{"total":96636764160,"available":95887900672,"internally_reserved":0,"allocated":421888,"data_stored":161081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618488,"kb_used_data":932,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753352,"statfs":{"total":96636764160,"available":96003432448,"internally_reserved":0,"allocated":954368,"data_stored":693561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722024,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649816,"statfs":{"total":96636764160,"available":95897411584,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:16.730 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-16T04:54:16.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12654", 'reported_seq': 16331, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:11.916711+0000', 'last_change': '2024-09-16T04:54:01.997093+0000', 'last_active': '2024-09-16T04:54:11.916711+0000', 'last_peered': '2024-09-16T04:54:11.916711+0000', 'last_clean': '2024-09-16T04:40:11.606968+0000', 'last_became_active': '2024-09-16T04:41:27.812511+0000', 'last_became_peered': '2024-09-16T04:41:27.812511+0000', 'last_unstale': '2024-09-16T04:54:11.916711+0000', 'last_undegraded': '2024-09-16T04:41:27.546627+0000', 'last_fullsized': '2024-09-16T04:41:27.483164+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 421, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12654, 'num_write_kb': 50616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1798, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -421}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:16.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:54:16.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12801, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:14.174341+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:54:14.174341+0000', 'last_peered': '2024-09-16T04:54:14.174341+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:54:14.174341+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:16.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:54:16.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12902, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:03.134907+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:54:03.134907+0000', 'last_peered': '2024-09-16T04:54:03.134907+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:54:03.134907+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:16.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:54:16.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 13250, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:15.011747+0000', 'last_change': '2024-09-16T04:54:13.785237+0000', 'last_active': '2024-09-16T04:54:15.011747+0000', 'last_peered': '2024-09-16T04:54:15.011747+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:54:15.011747+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2014, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 190, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2012}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:16.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:54:16.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12651, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:03.136221+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:54:03.136221+0000', 'last_peered': '2024-09-16T04:54:03.136221+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:54:03.136221+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:16.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:54:16.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12453, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:03.136618+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:54:03.136618+0000', 'last_peered': '2024-09-16T04:54:03.136618+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:54:03.136618+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:16.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:54:16.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12943, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:03.134988+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:54:03.134988+0000', 'last_peered': '2024-09-16T04:54:03.134988+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:54:03.134988+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:16.732 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:54:17.111 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:17.111 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13546/112 objects degraded (12094.643%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":869,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2860961792,"bytes_avail":383686094848,"bytes_total":386547056640,"degraded_objects":13546,"degraded_total":112,"degraded_ratio":120.94642857142857,"recovering_objects_per_sec":176,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1765,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:54:17.124 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:54:17.125 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:17.452 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:17.452 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:17.465 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-09-16T04:54:16.822011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19045,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793932,"kb_used_data":2168,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693428,"statfs":{"total":386547056640,"available":383686070272,"internally_reserved":0,"allocated":2220032,"data_stored":1176783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001644"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":16331,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:11.916711+0000","last_change":"2024-09-16T04:54:01.997093+0000","last_active":"2024-09-16T04:54:11.916711+0000","last_peered":"2024-09-16T04:54:11.916711+0000","last_clean":"2024-09-16T04:40:11.606968+0000","last_became_active":"2024-09-16T04:41:27.812511+0000","last_became_peered":"2024-09-16T04:41:27.812511+0000","last_unstale":"2024-09-16T04:54:11.916711+0000","last_undegraded":"2024-09-16T04:41:27.546627+0000","last_fullsized":"2024-09-16T04:41:27.483164+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1798,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12790,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172610+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:14.172610+0000","last_peered":"2024-09-16T04:54:14.172610+0000","last_clean":"2024-09-16T04:54:14.172610+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:14.172610+0000","last_undegraded":"2024-09-16T04:54:14.172610+0000","last_fullsized":"2024-09-16T04:54:14.172610+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:18.376544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12801,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:14.174341+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:14.174341+0000","last_peered":"2024-09-16T04:54:14.174341+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:14.174341+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17245,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.170926+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:14.170926+0000","last_peered":"2024-09-16T04:54:14.170926+0000","last_clean":"2024-09-16T04:54:14.170926+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:14.170926+0000","last_undegraded":"2024-09-16T04:54:14.170926+0000","last_fullsized":"2024-09-16T04:54:14.170926+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12902,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.134907+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:03.134907+0000","last_peered":"2024-09-16T04:54:03.134907+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:03.134907+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16818,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.135987+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:03.135987+0000","last_peered":"2024-09-16T04:54:03.135987+0000","last_clean":"2024-09-16T04:54:03.135987+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:03.135987+0000","last_undegraded":"2024-09-16T04:54:03.135987+0000","last_fullsized":"2024-09-16T04:54:03.135987+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17169,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.178620+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:15.178620+0000","last_peered":"2024-09-16T04:54:15.178620+0000","last_clean":"2024-09-16T04:54:15.178620+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:15.178620+0000","last_undegraded":"2024-09-16T04:54:15.178620+0000","last_fullsized":"2024-09-16T04:54:15.178620+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:16.036269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17377,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172200+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:14.172200+0000","last_peered":"2024-09-16T04:54:14.172200+0000","last_clean":"2024-09-16T04:54:14.172200+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:14.172200+0000","last_undegraded":"2024-09-16T04:54:14.172200+0000","last_fullsized":"2024-09-16T04:54:14.172200+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17540,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:54:03.135852+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:03.135852+0000","last_peered":"2024-09-16T04:54:03.135852+0000","last_clean":"2024-09-16T04:54:03.135852+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:03.135852+0000","last_undegraded":"2024-09-16T04:54:03.135852+0000","last_fullsized":"2024-09-16T04:54:03.135852+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":13250,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.011747+0000","last_change":"2024-09-16T04:54:13.785237+0000","last_active":"2024-09-16T04:54:15.011747+0000","last_peered":"2024-09-16T04:54:15.011747+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:15.011747+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2012}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12651,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.136221+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:03.136221+0000","last_peered":"2024-09-16T04:54:03.136221+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:03.136221+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12453,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.136618+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:54:03.136618+0000","last_peered":"2024-09-16T04:54:03.136618+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:03.136618+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.171112+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:14.171112+0000","last_peered":"2024-09-16T04:54:14.171112+0000","last_clean":"2024-09-16T04:54:14.171112+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:14.171112+0000","last_undegraded":"2024-09-16T04:54:14.171112+0000","last_fullsized":"2024-09-16T04:54:14.171112+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12943,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:03.134988+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:03.134988+0000","last_peered":"2024-09-16T04:54:03.134988+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:03.134988+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14951,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172083+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:14.172083+0000","last_peered":"2024-09-16T04:54:14.172083+0000","last_clean":"2024-09-16T04:54:14.172083+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:14.172083+0000","last_undegraded":"2024-09-16T04:54:14.172083+0000","last_fullsized":"2024-09-16T04:54:14.172083+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12421,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172292+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:14.172292+0000","last_peered":"2024-09-16T04:54:14.172292+0000","last_clean":"2024-09-16T04:54:14.172292+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:14.172292+0000","last_undegraded":"2024-09-16T04:54:14.172292+0000","last_fullsized":"2024-09-16T04:54:14.172292+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:14.382090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19045,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":413696,"data_stored":157791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731312,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640528,"statfs":{"total":96636764160,"available":95887900672,"internally_reserved":0,"allocated":421888,"data_stored":161081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618488,"kb_used_data":932,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753352,"statfs":{"total":96636764160,"available":96003432448,"internally_reserved":0,"allocated":954368,"data_stored":693561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:20.467 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:20.788 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:20.788 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:20.800 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-09-16T04:54:18.822379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19466,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2793948,"kb_used_data":2184,"kb_used_omap":14,"kb_used_meta":2791665,"kb_avail":374693412,"statfs":{"total":386547056640,"available":383686053888,"internally_reserved":0,"allocated":2236416,"data_stored":1183342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2858665521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1548,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001674"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17185,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.253890+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:15.253890+0000","last_peered":"2024-09-16T04:54:15.253890+0000","last_clean":"2024-09-16T04:54:15.253890+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:15.253890+0000","last_undegraded":"2024-09-16T04:54:15.253890+0000","last_fullsized":"2024-09-16T04:54:15.253890+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12790,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172610+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:14.172610+0000","last_peered":"2024-09-16T04:54:14.172610+0000","last_clean":"2024-09-16T04:54:14.172610+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:14.172610+0000","last_undegraded":"2024-09-16T04:54:14.172610+0000","last_fullsized":"2024-09-16T04:54:14.172610+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:18.376544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12801,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:14.174341+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:14.174341+0000","last_peered":"2024-09-16T04:54:14.174341+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:14.174341+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17245,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.170926+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:14.170926+0000","last_peered":"2024-09-16T04:54:14.170926+0000","last_clean":"2024-09-16T04:54:14.170926+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:14.170926+0000","last_undegraded":"2024-09-16T04:54:14.170926+0000","last_fullsized":"2024-09-16T04:54:14.170926+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12906,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174310+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:15.174310+0000","last_peered":"2024-09-16T04:54:15.174310+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:15.174310+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16822,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.174543+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:15.174543+0000","last_peered":"2024-09-16T04:54:15.174543+0000","last_clean":"2024-09-16T04:54:15.174543+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:15.174543+0000","last_undegraded":"2024-09-16T04:54:15.174543+0000","last_fullsized":"2024-09-16T04:54:15.174543+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17169,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.178620+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:15.178620+0000","last_peered":"2024-09-16T04:54:15.178620+0000","last_clean":"2024-09-16T04:54:15.178620+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:15.178620+0000","last_undegraded":"2024-09-16T04:54:15.178620+0000","last_fullsized":"2024-09-16T04:54:15.178620+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:16.036269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17377,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172200+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:14.172200+0000","last_peered":"2024-09-16T04:54:14.172200+0000","last_clean":"2024-09-16T04:54:14.172200+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:14.172200+0000","last_undegraded":"2024-09-16T04:54:14.172200+0000","last_fullsized":"2024-09-16T04:54:14.172200+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17544,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.174431+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:15.174431+0000","last_peered":"2024-09-16T04:54:15.174431+0000","last_clean":"2024-09-16T04:54:15.174431+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:15.174431+0000","last_undegraded":"2024-09-16T04:54:15.174431+0000","last_fullsized":"2024-09-16T04:54:15.174431+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:20.061539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":13250,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.011747+0000","last_change":"2024-09-16T04:54:13.785237+0000","last_active":"2024-09-16T04:54:15.011747+0000","last_peered":"2024-09-16T04:54:15.011747+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:15.011747+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2012}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12655,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174801+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:15.174801+0000","last_peered":"2024-09-16T04:54:15.174801+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:15.174801+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12457,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.175008+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:54:15.175008+0000","last_peered":"2024-09-16T04:54:15.175008+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:15.175008+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.171112+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:14.171112+0000","last_peered":"2024-09-16T04:54:14.171112+0000","last_clean":"2024-09-16T04:54:14.171112+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:14.171112+0000","last_undegraded":"2024-09-16T04:54:14.171112+0000","last_fullsized":"2024-09-16T04:54:14.171112+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12947,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174482+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:15.174482+0000","last_peered":"2024-09-16T04:54:15.174482+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:15.174482+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14951,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172083+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:14.172083+0000","last_peered":"2024-09-16T04:54:14.172083+0000","last_clean":"2024-09-16T04:54:14.172083+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:14.172083+0000","last_undegraded":"2024-09-16T04:54:14.172083+0000","last_fullsized":"2024-09-16T04:54:14.172083+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12421,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:54:14.172292+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:14.172292+0000","last_peered":"2024-09-16T04:54:14.172292+0000","last_clean":"2024-09-16T04:54:14.172292+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:14.172292+0000","last_undegraded":"2024-09-16T04:54:14.172292+0000","last_fullsized":"2024-09-16T04:54:14.172292+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:14.382090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19466,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731312,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640528,"statfs":{"total":96636764160,"available":95887900672,"internally_reserved":0,"allocated":421888,"data_stored":161081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618488,"kb_used_data":932,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93753352,"statfs":{"total":96636764160,"available":96003432448,"internally_reserved":0,"allocated":954368,"data_stored":693561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632352463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:20.801 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:21.100 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:21.100 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:21.112 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-16T04:54:20.822709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20264,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781480,"kb_used_data":2200,"kb_used_omap":14,"kb_used_meta":2779185,"kb_avail":374705880,"statfs":{"total":386547056640,"available":383698821120,"internally_reserved":0,"allocated":2252800,"data_stored":1189880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2845886001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001618"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17185,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.253890+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:15.253890+0000","last_peered":"2024-09-16T04:54:15.253890+0000","last_clean":"2024-09-16T04:54:15.253890+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:15.253890+0000","last_undegraded":"2024-09-16T04:54:15.253890+0000","last_fullsized":"2024-09-16T04:54:15.253890+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12792,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176835+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:15.176835+0000","last_peered":"2024-09-16T04:54:15.176835+0000","last_clean":"2024-09-16T04:54:15.176835+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:15.176835+0000","last_undegraded":"2024-09-16T04:54:15.176835+0000","last_fullsized":"2024-09-16T04:54:15.176835+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:18.376544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12803,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.178185+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:15.178185+0000","last_peered":"2024-09-16T04:54:15.178185+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:15.178185+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17247,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176413+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:15.176413+0000","last_peered":"2024-09-16T04:54:15.176413+0000","last_clean":"2024-09-16T04:54:15.176413+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:15.176413+0000","last_undegraded":"2024-09-16T04:54:15.176413+0000","last_fullsized":"2024-09-16T04:54:15.176413+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12906,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174310+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:15.174310+0000","last_peered":"2024-09-16T04:54:15.174310+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:15.174310+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16822,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.174543+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:15.174543+0000","last_peered":"2024-09-16T04:54:15.174543+0000","last_clean":"2024-09-16T04:54:15.174543+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:15.174543+0000","last_undegraded":"2024-09-16T04:54:15.174543+0000","last_fullsized":"2024-09-16T04:54:15.174543+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17169,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.178620+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:15.178620+0000","last_peered":"2024-09-16T04:54:15.178620+0000","last_clean":"2024-09-16T04:54:15.178620+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:15.178620+0000","last_undegraded":"2024-09-16T04:54:15.178620+0000","last_fullsized":"2024-09-16T04:54:15.178620+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:16.036269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17379,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176641+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:15.176641+0000","last_peered":"2024-09-16T04:54:15.176641+0000","last_clean":"2024-09-16T04:54:15.176641+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:15.176641+0000","last_undegraded":"2024-09-16T04:54:15.176641+0000","last_fullsized":"2024-09-16T04:54:15.176641+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17544,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.174431+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:15.174431+0000","last_peered":"2024-09-16T04:54:15.174431+0000","last_clean":"2024-09-16T04:54:15.174431+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:15.174431+0000","last_undegraded":"2024-09-16T04:54:15.174431+0000","last_fullsized":"2024-09-16T04:54:15.174431+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:20.061539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":14848,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:20.012052+0000","last_change":"2024-09-16T04:54:13.785237+0000","last_active":"2024-09-16T04:54:20.012052+0000","last_peered":"2024-09-16T04:54:20.012052+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:20.012052+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":988,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12655,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174801+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:15.174801+0000","last_peered":"2024-09-16T04:54:15.174801+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:15.174801+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12457,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.175008+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:54:15.175008+0000","last_peered":"2024-09-16T04:54:15.175008+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:15.175008+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12648,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176523+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:15.176523+0000","last_peered":"2024-09-16T04:54:15.176523+0000","last_clean":"2024-09-16T04:54:15.176523+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:15.176523+0000","last_undegraded":"2024-09-16T04:54:15.176523+0000","last_fullsized":"2024-09-16T04:54:15.176523+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12947,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174482+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:15.174482+0000","last_peered":"2024-09-16T04:54:15.174482+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:15.174482+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14953,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176601+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:15.176601+0000","last_peered":"2024-09-16T04:54:15.176601+0000","last_clean":"2024-09-16T04:54:15.176601+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:15.176601+0000","last_undegraded":"2024-09-16T04:54:15.176601+0000","last_fullsized":"2024-09-16T04:54:15.176601+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12423,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176778+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:15.176778+0000","last_peered":"2024-09-16T04:54:15.176778+0000","last_clean":"2024-09-16T04:54:15.176778+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:15.176778+0000","last_undegraded":"2024-09-16T04:54:15.176778+0000","last_fullsized":"2024-09-16T04:54:15.176778+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:19.386166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20264,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731320,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640520,"statfs":{"total":96636764160,"available":95887892480,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606012,"kb_used_data":940,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93765828,"statfs":{"total":96636764160,"available":96016207872,"internally_reserved":0,"allocated":962560,"data_stored":696830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619572943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:21.113 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:54:21.113 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12803, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:15.178185+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:54:15.178185+0000', 'last_peered': '2024-09-16T04:54:15.178185+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:54:15.178185+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:21.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:54:21.114 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12906, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:15.174310+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:54:15.174310+0000', 'last_peered': '2024-09-16T04:54:15.174310+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:54:15.174310+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:21.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:54:21.114 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 14848, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:20.012052+0000', 'last_change': '2024-09-16T04:54:13.785237+0000', 'last_active': '2024-09-16T04:54:20.012052+0000', 'last_peered': '2024-09-16T04:54:20.012052+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:54:20.012052+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1216, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 988, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1214}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:21.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:54:21.114 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12655, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:15.174801+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:54:15.174801+0000', 'last_peered': '2024-09-16T04:54:15.174801+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:54:15.174801+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:21.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:54:21.114 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12457, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:15.175008+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:54:15.175008+0000', 'last_peered': '2024-09-16T04:54:15.175008+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:54:15.175008+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:21.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:54:21.114 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12947, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:15.174482+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:54:15.174482+0000', 'last_peered': '2024-09-16T04:54:15.174482+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:54:15.174482+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:21.115 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:54:21.488 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:21.488 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12150/112 objects degraded (10848.214%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":873,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2848235520,"bytes_avail":383698821120,"bytes_total":386547056640,"degraded_objects":12150,"degraded_total":112,"degraded_ratio":108.48214285714286,"recovering_objects_per_sec":174,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:54:21.501 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:54:21.502 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:21.799 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:21.799 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:21.812 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-16T04:54:20.822709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20264,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781480,"kb_used_data":2200,"kb_used_omap":14,"kb_used_meta":2779185,"kb_avail":374705880,"statfs":{"total":386547056640,"available":383698821120,"internally_reserved":0,"allocated":2252800,"data_stored":1189880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2845886001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001618"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17185,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.253890+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:15.253890+0000","last_peered":"2024-09-16T04:54:15.253890+0000","last_clean":"2024-09-16T04:54:15.253890+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:15.253890+0000","last_undegraded":"2024-09-16T04:54:15.253890+0000","last_fullsized":"2024-09-16T04:54:15.253890+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12792,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176835+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:15.176835+0000","last_peered":"2024-09-16T04:54:15.176835+0000","last_clean":"2024-09-16T04:54:15.176835+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:15.176835+0000","last_undegraded":"2024-09-16T04:54:15.176835+0000","last_fullsized":"2024-09-16T04:54:15.176835+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:18.376544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12803,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.178185+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:15.178185+0000","last_peered":"2024-09-16T04:54:15.178185+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:15.178185+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17247,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176413+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:15.176413+0000","last_peered":"2024-09-16T04:54:15.176413+0000","last_clean":"2024-09-16T04:54:15.176413+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:15.176413+0000","last_undegraded":"2024-09-16T04:54:15.176413+0000","last_fullsized":"2024-09-16T04:54:15.176413+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12906,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174310+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:15.174310+0000","last_peered":"2024-09-16T04:54:15.174310+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:15.174310+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16822,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.174543+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:15.174543+0000","last_peered":"2024-09-16T04:54:15.174543+0000","last_clean":"2024-09-16T04:54:15.174543+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:15.174543+0000","last_undegraded":"2024-09-16T04:54:15.174543+0000","last_fullsized":"2024-09-16T04:54:15.174543+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17169,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.178620+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:15.178620+0000","last_peered":"2024-09-16T04:54:15.178620+0000","last_clean":"2024-09-16T04:54:15.178620+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:15.178620+0000","last_undegraded":"2024-09-16T04:54:15.178620+0000","last_fullsized":"2024-09-16T04:54:15.178620+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:16.036269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17379,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176641+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:15.176641+0000","last_peered":"2024-09-16T04:54:15.176641+0000","last_clean":"2024-09-16T04:54:15.176641+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:15.176641+0000","last_undegraded":"2024-09-16T04:54:15.176641+0000","last_fullsized":"2024-09-16T04:54:15.176641+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17544,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.174431+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:15.174431+0000","last_peered":"2024-09-16T04:54:15.174431+0000","last_clean":"2024-09-16T04:54:15.174431+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:15.174431+0000","last_undegraded":"2024-09-16T04:54:15.174431+0000","last_fullsized":"2024-09-16T04:54:15.174431+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:20.061539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":14848,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:20.012052+0000","last_change":"2024-09-16T04:54:13.785237+0000","last_active":"2024-09-16T04:54:20.012052+0000","last_peered":"2024-09-16T04:54:20.012052+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:20.012052+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":988,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12655,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174801+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:15.174801+0000","last_peered":"2024-09-16T04:54:15.174801+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:15.174801+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12457,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.175008+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:54:15.175008+0000","last_peered":"2024-09-16T04:54:15.175008+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:15.175008+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12648,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176523+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:15.176523+0000","last_peered":"2024-09-16T04:54:15.176523+0000","last_clean":"2024-09-16T04:54:15.176523+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:15.176523+0000","last_undegraded":"2024-09-16T04:54:15.176523+0000","last_fullsized":"2024-09-16T04:54:15.176523+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12947,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174482+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:15.174482+0000","last_peered":"2024-09-16T04:54:15.174482+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:15.174482+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14953,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176601+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:15.176601+0000","last_peered":"2024-09-16T04:54:15.176601+0000","last_clean":"2024-09-16T04:54:15.176601+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:15.176601+0000","last_undegraded":"2024-09-16T04:54:15.176601+0000","last_fullsized":"2024-09-16T04:54:15.176601+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12423,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176778+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:15.176778+0000","last_peered":"2024-09-16T04:54:15.176778+0000","last_clean":"2024-09-16T04:54:15.176778+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:15.176778+0000","last_undegraded":"2024-09-16T04:54:15.176778+0000","last_fullsized":"2024-09-16T04:54:15.176778+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:19.386166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20264,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731320,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640520,"statfs":{"total":96636764160,"available":95887892480,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606012,"kb_used_data":940,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93765828,"statfs":{"total":96636764160,"available":96016207872,"internally_reserved":0,"allocated":962560,"data_stored":696830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619572943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:24.814 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:25.108 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:25.109 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:25.121 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-16T04:54:24.823482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20264,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789800,"kb_used_data":2200,"kb_used_omap":14,"kb_used_meta":2787505,"kb_avail":374697560,"statfs":{"total":386547056640,"available":383690301440,"internally_reserved":0,"allocated":2252800,"data_stored":1189880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2854405681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001913"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17186,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.254857+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:15.254857+0000","last_peered":"2024-09-16T04:54:15.254857+0000","last_clean":"2024-09-16T04:54:15.254857+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:15.254857+0000","last_undegraded":"2024-09-16T04:54:15.254857+0000","last_fullsized":"2024-09-16T04:54:15.254857+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12792,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176835+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:15.176835+0000","last_peered":"2024-09-16T04:54:15.176835+0000","last_clean":"2024-09-16T04:54:15.176835+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:15.176835+0000","last_undegraded":"2024-09-16T04:54:15.176835+0000","last_fullsized":"2024-09-16T04:54:15.176835+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:18.376544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12803,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.178185+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:15.178185+0000","last_peered":"2024-09-16T04:54:15.178185+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:15.178185+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17247,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176413+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:15.176413+0000","last_peered":"2024-09-16T04:54:15.176413+0000","last_clean":"2024-09-16T04:54:15.176413+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:15.176413+0000","last_undegraded":"2024-09-16T04:54:15.176413+0000","last_fullsized":"2024-09-16T04:54:15.176413+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12906,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174310+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:15.174310+0000","last_peered":"2024-09-16T04:54:15.174310+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:15.174310+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16822,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.174543+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:15.174543+0000","last_peered":"2024-09-16T04:54:15.174543+0000","last_clean":"2024-09-16T04:54:15.174543+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:15.174543+0000","last_undegraded":"2024-09-16T04:54:15.174543+0000","last_fullsized":"2024-09-16T04:54:15.174543+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17169,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.178620+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:15.178620+0000","last_peered":"2024-09-16T04:54:15.178620+0000","last_clean":"2024-09-16T04:54:15.178620+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:15.178620+0000","last_undegraded":"2024-09-16T04:54:15.178620+0000","last_fullsized":"2024-09-16T04:54:15.178620+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:16.036269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17379,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176641+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:15.176641+0000","last_peered":"2024-09-16T04:54:15.176641+0000","last_clean":"2024-09-16T04:54:15.176641+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:15.176641+0000","last_undegraded":"2024-09-16T04:54:15.176641+0000","last_fullsized":"2024-09-16T04:54:15.176641+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17544,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.174431+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:15.174431+0000","last_peered":"2024-09-16T04:54:15.174431+0000","last_clean":"2024-09-16T04:54:15.174431+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:15.174431+0000","last_undegraded":"2024-09-16T04:54:15.174431+0000","last_fullsized":"2024-09-16T04:54:15.174431+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:20.061539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":14848,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:20.012052+0000","last_change":"2024-09-16T04:54:13.785237+0000","last_active":"2024-09-16T04:54:20.012052+0000","last_peered":"2024-09-16T04:54:20.012052+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:20.012052+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":988,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12655,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174801+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:15.174801+0000","last_peered":"2024-09-16T04:54:15.174801+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:15.174801+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12457,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.175008+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:54:15.175008+0000","last_peered":"2024-09-16T04:54:15.175008+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:15.175008+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12648,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176523+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:15.176523+0000","last_peered":"2024-09-16T04:54:15.176523+0000","last_clean":"2024-09-16T04:54:15.176523+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:15.176523+0000","last_undegraded":"2024-09-16T04:54:15.176523+0000","last_fullsized":"2024-09-16T04:54:15.176523+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12947,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174482+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:15.174482+0000","last_peered":"2024-09-16T04:54:15.174482+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:15.174482+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14953,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176601+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:15.176601+0000","last_peered":"2024-09-16T04:54:15.176601+0000","last_clean":"2024-09-16T04:54:15.176601+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:15.176601+0000","last_undegraded":"2024-09-16T04:54:15.176601+0000","last_fullsized":"2024-09-16T04:54:15.176601+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12423,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176778+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:15.176778+0000","last_peered":"2024-09-16T04:54:15.176778+0000","last_clean":"2024-09-16T04:54:15.176778+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:15.176778+0000","last_undegraded":"2024-09-16T04:54:15.176778+0000","last_fullsized":"2024-09-16T04:54:15.176778+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:19.386166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20264,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731320,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640520,"statfs":{"total":96636764160,"available":95887892480,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614332,"kb_used_data":940,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757508,"statfs":{"total":96636764160,"available":96007688192,"internally_reserved":0,"allocated":962560,"data_stored":696830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628092623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:25.122 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:25.426 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:25.427 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:25.440 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-16T04:54:24.823482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20264,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789800,"kb_used_data":2200,"kb_used_omap":14,"kb_used_meta":2787505,"kb_avail":374697560,"statfs":{"total":386547056640,"available":383690301440,"internally_reserved":0,"allocated":2252800,"data_stored":1189880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2854405681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001913"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17186,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.254857+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:15.254857+0000","last_peered":"2024-09-16T04:54:15.254857+0000","last_clean":"2024-09-16T04:54:15.254857+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:15.254857+0000","last_undegraded":"2024-09-16T04:54:15.254857+0000","last_fullsized":"2024-09-16T04:54:15.254857+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12792,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176835+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:15.176835+0000","last_peered":"2024-09-16T04:54:15.176835+0000","last_clean":"2024-09-16T04:54:15.176835+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:15.176835+0000","last_undegraded":"2024-09-16T04:54:15.176835+0000","last_fullsized":"2024-09-16T04:54:15.176835+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:18.376544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12803,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.178185+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:15.178185+0000","last_peered":"2024-09-16T04:54:15.178185+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:15.178185+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17247,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176413+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:15.176413+0000","last_peered":"2024-09-16T04:54:15.176413+0000","last_clean":"2024-09-16T04:54:15.176413+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:15.176413+0000","last_undegraded":"2024-09-16T04:54:15.176413+0000","last_fullsized":"2024-09-16T04:54:15.176413+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12906,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174310+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:15.174310+0000","last_peered":"2024-09-16T04:54:15.174310+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:15.174310+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16822,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.174543+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:15.174543+0000","last_peered":"2024-09-16T04:54:15.174543+0000","last_clean":"2024-09-16T04:54:15.174543+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:15.174543+0000","last_undegraded":"2024-09-16T04:54:15.174543+0000","last_fullsized":"2024-09-16T04:54:15.174543+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17169,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.178620+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:15.178620+0000","last_peered":"2024-09-16T04:54:15.178620+0000","last_clean":"2024-09-16T04:54:15.178620+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:15.178620+0000","last_undegraded":"2024-09-16T04:54:15.178620+0000","last_fullsized":"2024-09-16T04:54:15.178620+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:16.036269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17379,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176641+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:15.176641+0000","last_peered":"2024-09-16T04:54:15.176641+0000","last_clean":"2024-09-16T04:54:15.176641+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:15.176641+0000","last_undegraded":"2024-09-16T04:54:15.176641+0000","last_fullsized":"2024-09-16T04:54:15.176641+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17544,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.174431+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:15.174431+0000","last_peered":"2024-09-16T04:54:15.174431+0000","last_clean":"2024-09-16T04:54:15.174431+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:15.174431+0000","last_undegraded":"2024-09-16T04:54:15.174431+0000","last_fullsized":"2024-09-16T04:54:15.174431+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:20.061539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":14848,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:20.012052+0000","last_change":"2024-09-16T04:54:13.785237+0000","last_active":"2024-09-16T04:54:20.012052+0000","last_peered":"2024-09-16T04:54:20.012052+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:20.012052+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":988,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12655,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174801+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:15.174801+0000","last_peered":"2024-09-16T04:54:15.174801+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:15.174801+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12457,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.175008+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:54:15.175008+0000","last_peered":"2024-09-16T04:54:15.175008+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:15.175008+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12648,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176523+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:15.176523+0000","last_peered":"2024-09-16T04:54:15.176523+0000","last_clean":"2024-09-16T04:54:15.176523+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:15.176523+0000","last_undegraded":"2024-09-16T04:54:15.176523+0000","last_fullsized":"2024-09-16T04:54:15.176523+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12947,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174482+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:15.174482+0000","last_peered":"2024-09-16T04:54:15.174482+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:15.174482+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14953,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176601+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:15.176601+0000","last_peered":"2024-09-16T04:54:15.176601+0000","last_clean":"2024-09-16T04:54:15.176601+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:15.176601+0000","last_undegraded":"2024-09-16T04:54:15.176601+0000","last_fullsized":"2024-09-16T04:54:15.176601+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12423,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176778+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:15.176778+0000","last_peered":"2024-09-16T04:54:15.176778+0000","last_clean":"2024-09-16T04:54:15.176778+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:15.176778+0000","last_undegraded":"2024-09-16T04:54:15.176778+0000","last_fullsized":"2024-09-16T04:54:15.176778+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:19.386166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20264,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731320,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640520,"statfs":{"total":96636764160,"available":95887892480,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614332,"kb_used_data":940,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757508,"statfs":{"total":96636764160,"available":96007688192,"internally_reserved":0,"allocated":962560,"data_stored":696830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628092623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:25.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:54:25.442 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12803, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:15.178185+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:54:15.178185+0000', 'last_peered': '2024-09-16T04:54:15.178185+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:54:15.178185+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:25.442 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:54:25.442 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12906, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:15.174310+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:54:15.174310+0000', 'last_peered': '2024-09-16T04:54:15.174310+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:54:15.174310+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:25.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:54:25.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 14848, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:20.012052+0000', 'last_change': '2024-09-16T04:54:13.785237+0000', 'last_active': '2024-09-16T04:54:20.012052+0000', 'last_peered': '2024-09-16T04:54:20.012052+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:54:20.012052+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1216, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 988, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1214}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:25.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:54:25.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12655, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:15.174801+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:54:15.174801+0000', 'last_peered': '2024-09-16T04:54:15.174801+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:54:15.174801+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:25.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:54:25.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12457, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:15.175008+0000', 'last_change': '2024-09-16T04:41:27.842408+0000', 'last_active': '2024-09-16T04:54:15.175008+0000', 'last_peered': '2024-09-16T04:54:15.175008+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:54:15.175008+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:25.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:54:25.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12947, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:15.174482+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:54:15.174482+0000', 'last_peered': '2024-09-16T04:54:15.174482+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:54:15.174482+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:25.445 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:54:25.819 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:25.819 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12150/112 objects degraded (10848.214%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":878,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2856755200,"bytes_avail":383690301440,"bytes_total":386547056640,"degraded_objects":12150,"degraded_total":112,"degraded_ratio":108.48214285714286,"recovering_objects_per_sec":139,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:54:25.832 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:54:25.833 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:26.143 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:26.143 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:26.157 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-16T04:54:24.823482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20264,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789800,"kb_used_data":2200,"kb_used_omap":14,"kb_used_meta":2787505,"kb_avail":374697560,"statfs":{"total":386547056640,"available":383690301440,"internally_reserved":0,"allocated":2252800,"data_stored":1189880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2854405681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001913"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17186,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.254857+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:15.254857+0000","last_peered":"2024-09-16T04:54:15.254857+0000","last_clean":"2024-09-16T04:54:15.254857+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:15.254857+0000","last_undegraded":"2024-09-16T04:54:15.254857+0000","last_fullsized":"2024-09-16T04:54:15.254857+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12792,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176835+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:15.176835+0000","last_peered":"2024-09-16T04:54:15.176835+0000","last_clean":"2024-09-16T04:54:15.176835+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:15.176835+0000","last_undegraded":"2024-09-16T04:54:15.176835+0000","last_fullsized":"2024-09-16T04:54:15.176835+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:18.376544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12803,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.178185+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:15.178185+0000","last_peered":"2024-09-16T04:54:15.178185+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:15.178185+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17247,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176413+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:15.176413+0000","last_peered":"2024-09-16T04:54:15.176413+0000","last_clean":"2024-09-16T04:54:15.176413+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:15.176413+0000","last_undegraded":"2024-09-16T04:54:15.176413+0000","last_fullsized":"2024-09-16T04:54:15.176413+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12906,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174310+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:15.174310+0000","last_peered":"2024-09-16T04:54:15.174310+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:15.174310+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16822,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.174543+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:15.174543+0000","last_peered":"2024-09-16T04:54:15.174543+0000","last_clean":"2024-09-16T04:54:15.174543+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:15.174543+0000","last_undegraded":"2024-09-16T04:54:15.174543+0000","last_fullsized":"2024-09-16T04:54:15.174543+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17169,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.178620+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:15.178620+0000","last_peered":"2024-09-16T04:54:15.178620+0000","last_clean":"2024-09-16T04:54:15.178620+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:15.178620+0000","last_undegraded":"2024-09-16T04:54:15.178620+0000","last_fullsized":"2024-09-16T04:54:15.178620+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:16.036269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17379,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176641+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:15.176641+0000","last_peered":"2024-09-16T04:54:15.176641+0000","last_clean":"2024-09-16T04:54:15.176641+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:15.176641+0000","last_undegraded":"2024-09-16T04:54:15.176641+0000","last_fullsized":"2024-09-16T04:54:15.176641+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17544,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.174431+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:15.174431+0000","last_peered":"2024-09-16T04:54:15.174431+0000","last_clean":"2024-09-16T04:54:15.174431+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:15.174431+0000","last_undegraded":"2024-09-16T04:54:15.174431+0000","last_fullsized":"2024-09-16T04:54:15.174431+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:20.061539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":14848,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:20.012052+0000","last_change":"2024-09-16T04:54:13.785237+0000","last_active":"2024-09-16T04:54:20.012052+0000","last_peered":"2024-09-16T04:54:20.012052+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:20.012052+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":988,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12655,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174801+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:15.174801+0000","last_peered":"2024-09-16T04:54:15.174801+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:15.174801+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12457,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.175008+0000","last_change":"2024-09-16T04:41:27.842408+0000","last_active":"2024-09-16T04:54:15.175008+0000","last_peered":"2024-09-16T04:54:15.175008+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:15.175008+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12648,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176523+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:15.176523+0000","last_peered":"2024-09-16T04:54:15.176523+0000","last_clean":"2024-09-16T04:54:15.176523+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:15.176523+0000","last_undegraded":"2024-09-16T04:54:15.176523+0000","last_fullsized":"2024-09-16T04:54:15.176523+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12947,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.174482+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:15.174482+0000","last_peered":"2024-09-16T04:54:15.174482+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:15.174482+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14953,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176601+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:15.176601+0000","last_peered":"2024-09-16T04:54:15.176601+0000","last_clean":"2024-09-16T04:54:15.176601+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:15.176601+0000","last_undegraded":"2024-09-16T04:54:15.176601+0000","last_fullsized":"2024-09-16T04:54:15.176601+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12423,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176778+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:15.176778+0000","last_peered":"2024-09-16T04:54:15.176778+0000","last_clean":"2024-09-16T04:54:15.176778+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:15.176778+0000","last_undegraded":"2024-09-16T04:54:15.176778+0000","last_fullsized":"2024-09-16T04:54:15.176778+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:19.386166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20264,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731320,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640520,"statfs":{"total":96636764160,"available":95887892480,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614332,"kb_used_data":940,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757508,"statfs":{"total":96636764160,"available":96007688192,"internally_reserved":0,"allocated":962560,"data_stored":696830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628092623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:29.158 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:29.462 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:29.462 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:29.476 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-16T04:54:28.824202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21512,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789804,"kb_used_data":2208,"kb_used_omap":14,"kb_used_meta":2787505,"kb_avail":374697556,"statfs":{"total":386547056640,"available":383690297344,"internally_reserved":0,"allocated":2260992,"data_stored":1193133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2854405681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001492"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17188,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:54:26.203223+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:26.203223+0000","last_peered":"2024-09-16T04:54:26.203223+0000","last_clean":"2024-09-16T04:54:26.203223+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:26.203223+0000","last_undegraded":"2024-09-16T04:54:26.203223+0000","last_fullsized":"2024-09-16T04:54:26.203223+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12792,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176835+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:15.176835+0000","last_peered":"2024-09-16T04:54:15.176835+0000","last_clean":"2024-09-16T04:54:15.176835+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:15.176835+0000","last_undegraded":"2024-09-16T04:54:15.176835+0000","last_fullsized":"2024-09-16T04:54:15.176835+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:18.376544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12803,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.178185+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:15.178185+0000","last_peered":"2024-09-16T04:54:15.178185+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:15.178185+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17247,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176413+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:15.176413+0000","last_peered":"2024-09-16T04:54:15.176413+0000","last_clean":"2024-09-16T04:54:15.176413+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:15.176413+0000","last_undegraded":"2024-09-16T04:54:15.176413+0000","last_fullsized":"2024-09-16T04:54:15.176413+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12908,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:26.204101+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:26.204101+0000","last_peered":"2024-09-16T04:54:26.204101+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:26.204101+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16824,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:54:26.203585+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:26.203585+0000","last_peered":"2024-09-16T04:54:26.203585+0000","last_clean":"2024-09-16T04:54:26.203585+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:26.203585+0000","last_undegraded":"2024-09-16T04:54:26.203585+0000","last_fullsized":"2024-09-16T04:54:26.203585+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17171,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:54:26.211398+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:26.211398+0000","last_peered":"2024-09-16T04:54:26.211398+0000","last_clean":"2024-09-16T04:54:26.211398+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:26.211398+0000","last_undegraded":"2024-09-16T04:54:26.211398+0000","last_fullsized":"2024-09-16T04:54:26.211398+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17379,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176641+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:15.176641+0000","last_peered":"2024-09-16T04:54:15.176641+0000","last_clean":"2024-09-16T04:54:15.176641+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:15.176641+0000","last_undegraded":"2024-09-16T04:54:15.176641+0000","last_fullsized":"2024-09-16T04:54:15.176641+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17546,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:54:26.203482+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:26.203482+0000","last_peered":"2024-09-16T04:54:26.203482+0000","last_clean":"2024-09-16T04:54:26.203482+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:26.203482+0000","last_undegraded":"2024-09-16T04:54:26.203482+0000","last_fullsized":"2024-09-16T04:54:26.203482+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":16936,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:25.016415+0000","last_change":"2024-09-16T04:54:13.785237+0000","last_active":"2024-09-16T04:54:25.016415+0000","last_peered":"2024-09-16T04:54:25.016415+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:25.016415+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2032,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-170}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12657,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:26.203787+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:26.203787+0000","last_peered":"2024-09-16T04:54:26.203787+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:26.203787+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12868,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:26.921002+0000","last_change":"2024-09-16T04:54:25.748512+0000","last_active":"2024-09-16T04:54:26.921002+0000","last_peered":"2024-09-16T04:54:26.921002+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:26.921002+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1901}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12648,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176523+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:15.176523+0000","last_peered":"2024-09-16T04:54:15.176523+0000","last_clean":"2024-09-16T04:54:15.176523+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:15.176523+0000","last_undegraded":"2024-09-16T04:54:15.176523+0000","last_fullsized":"2024-09-16T04:54:15.176523+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12949,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:26.204077+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:26.204077+0000","last_peered":"2024-09-16T04:54:26.204077+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:26.204077+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14953,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176601+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:15.176601+0000","last_peered":"2024-09-16T04:54:15.176601+0000","last_clean":"2024-09-16T04:54:15.176601+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:15.176601+0000","last_undegraded":"2024-09-16T04:54:15.176601+0000","last_fullsized":"2024-09-16T04:54:15.176601+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12423,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176778+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:15.176778+0000","last_peered":"2024-09-16T04:54:15.176778+0000","last_clean":"2024-09-16T04:54:15.176778+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:15.176778+0000","last_undegraded":"2024-09-16T04:54:15.176778+0000","last_fullsized":"2024-09-16T04:54:15.176778+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:19.386166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21512,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722112,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649728,"statfs":{"total":96636764160,"available":95897321472,"internally_reserved":0,"allocated":438272,"data_stored":167603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731320,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640520,"statfs":{"total":96636764160,"available":95887892480,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614332,"kb_used_data":940,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757508,"statfs":{"total":96636764160,"available":96007688192,"internally_reserved":0,"allocated":962560,"data_stored":696830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628092623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:29.477 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:29.768 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:29.769 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:29.781 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-16T04:54:28.824202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21512,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789804,"kb_used_data":2208,"kb_used_omap":14,"kb_used_meta":2787505,"kb_avail":374697556,"statfs":{"total":386547056640,"available":383690297344,"internally_reserved":0,"allocated":2260992,"data_stored":1193133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2854405681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001492"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17188,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:54:26.203223+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:26.203223+0000","last_peered":"2024-09-16T04:54:26.203223+0000","last_clean":"2024-09-16T04:54:26.203223+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:26.203223+0000","last_undegraded":"2024-09-16T04:54:26.203223+0000","last_fullsized":"2024-09-16T04:54:26.203223+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12792,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176835+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:15.176835+0000","last_peered":"2024-09-16T04:54:15.176835+0000","last_clean":"2024-09-16T04:54:15.176835+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:15.176835+0000","last_undegraded":"2024-09-16T04:54:15.176835+0000","last_fullsized":"2024-09-16T04:54:15.176835+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:18.376544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12803,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.178185+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:15.178185+0000","last_peered":"2024-09-16T04:54:15.178185+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:15.178185+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17247,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176413+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:15.176413+0000","last_peered":"2024-09-16T04:54:15.176413+0000","last_clean":"2024-09-16T04:54:15.176413+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:15.176413+0000","last_undegraded":"2024-09-16T04:54:15.176413+0000","last_fullsized":"2024-09-16T04:54:15.176413+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12908,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:26.204101+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:26.204101+0000","last_peered":"2024-09-16T04:54:26.204101+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:26.204101+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16824,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:54:26.203585+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:26.203585+0000","last_peered":"2024-09-16T04:54:26.203585+0000","last_clean":"2024-09-16T04:54:26.203585+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:26.203585+0000","last_undegraded":"2024-09-16T04:54:26.203585+0000","last_fullsized":"2024-09-16T04:54:26.203585+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17171,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:54:26.211398+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:26.211398+0000","last_peered":"2024-09-16T04:54:26.211398+0000","last_clean":"2024-09-16T04:54:26.211398+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:26.211398+0000","last_undegraded":"2024-09-16T04:54:26.211398+0000","last_fullsized":"2024-09-16T04:54:26.211398+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17379,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176641+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:15.176641+0000","last_peered":"2024-09-16T04:54:15.176641+0000","last_clean":"2024-09-16T04:54:15.176641+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:15.176641+0000","last_undegraded":"2024-09-16T04:54:15.176641+0000","last_fullsized":"2024-09-16T04:54:15.176641+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17546,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:54:26.203482+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:26.203482+0000","last_peered":"2024-09-16T04:54:26.203482+0000","last_clean":"2024-09-16T04:54:26.203482+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:26.203482+0000","last_undegraded":"2024-09-16T04:54:26.203482+0000","last_fullsized":"2024-09-16T04:54:26.203482+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":16936,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:25.016415+0000","last_change":"2024-09-16T04:54:13.785237+0000","last_active":"2024-09-16T04:54:25.016415+0000","last_peered":"2024-09-16T04:54:25.016415+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:25.016415+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2032,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-170}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12657,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:26.203787+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:26.203787+0000","last_peered":"2024-09-16T04:54:26.203787+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:26.203787+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12868,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:26.921002+0000","last_change":"2024-09-16T04:54:25.748512+0000","last_active":"2024-09-16T04:54:26.921002+0000","last_peered":"2024-09-16T04:54:26.921002+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:26.921002+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1901}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12648,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176523+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:15.176523+0000","last_peered":"2024-09-16T04:54:15.176523+0000","last_clean":"2024-09-16T04:54:15.176523+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:15.176523+0000","last_undegraded":"2024-09-16T04:54:15.176523+0000","last_fullsized":"2024-09-16T04:54:15.176523+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12949,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:26.204077+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:26.204077+0000","last_peered":"2024-09-16T04:54:26.204077+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:26.204077+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14953,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176601+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:15.176601+0000","last_peered":"2024-09-16T04:54:15.176601+0000","last_clean":"2024-09-16T04:54:15.176601+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:15.176601+0000","last_undegraded":"2024-09-16T04:54:15.176601+0000","last_fullsized":"2024-09-16T04:54:15.176601+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12423,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176778+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:15.176778+0000","last_peered":"2024-09-16T04:54:15.176778+0000","last_clean":"2024-09-16T04:54:15.176778+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:15.176778+0000","last_undegraded":"2024-09-16T04:54:15.176778+0000","last_fullsized":"2024-09-16T04:54:15.176778+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:19.386166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21512,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722112,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649728,"statfs":{"total":96636764160,"available":95897321472,"internally_reserved":0,"allocated":438272,"data_stored":167603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731320,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640520,"statfs":{"total":96636764160,"available":95887892480,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614332,"kb_used_data":940,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757508,"statfs":{"total":96636764160,"available":96007688192,"internally_reserved":0,"allocated":962560,"data_stored":696830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628092623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:29.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:54:29.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12803, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:15.178185+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:54:15.178185+0000', 'last_peered': '2024-09-16T04:54:15.178185+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:54:15.178185+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:29.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:54:29.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12908, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:26.204101+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:54:26.204101+0000', 'last_peered': '2024-09-16T04:54:26.204101+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:54:26.204101+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:29.783 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-16T04:54:29.783 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12756", 'reported_seq': 16936, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:25.016415+0000', 'last_change': '2024-09-16T04:54:13.785237+0000', 'last_active': '2024-09-16T04:54:25.016415+0000', 'last_peered': '2024-09-16T04:54:25.016415+0000', 'last_clean': '2024-09-16T04:40:14.617336+0000', 'last_became_active': '2024-09-16T04:41:27.838073+0000', 'last_became_peered': '2024-09-16T04:41:27.838073+0000', 'last_unstale': '2024-09-16T04:54:25.016415+0000', 'last_undegraded': '2024-09-16T04:41:27.523521+0000', 'last_fullsized': '2024-09-16T04:41:27.486999+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 172, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12754, 'num_write_kb': 51022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2032, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -170}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:29.783 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:54:29.783 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12657, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:26.203787+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:54:26.203787+0000', 'last_peered': '2024-09-16T04:54:26.203787+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:54:26.203787+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:29.783 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:54:29.783 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12868, 'reported_epoch': 52, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:26.921002+0000', 'last_change': '2024-09-16T04:54:25.748512+0000', 'last_active': '2024-09-16T04:54:26.921002+0000', 'last_peered': '2024-09-16T04:54:26.921002+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:54:26.921002+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1902, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 228, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1901}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:29.783 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:54:29.783 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12949, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:26.204077+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:54:26.204077+0000', 'last_peered': '2024-09-16T04:54:26.204077+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:54:26.204077+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:29.784 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:54:30.154 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:30.154 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10902/112 objects degraded (9733.929%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":882,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2856759296,"bytes_avail":383690297344,"bytes_total":386547056640,"degraded_objects":10902,"degraded_total":112,"degraded_ratio":97.339285714285708,"recovering_objects_per_sec":155,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1248,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:54:30.170 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:54:30.170 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:30.472 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:30.472 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:30.484 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-16T04:54:28.824202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21512,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789804,"kb_used_data":2208,"kb_used_omap":14,"kb_used_meta":2787505,"kb_avail":374697556,"statfs":{"total":386547056640,"available":383690297344,"internally_reserved":0,"allocated":2260992,"data_stored":1193133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2854405681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001492"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17188,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:54:26.203223+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:26.203223+0000","last_peered":"2024-09-16T04:54:26.203223+0000","last_clean":"2024-09-16T04:54:26.203223+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:26.203223+0000","last_undegraded":"2024-09-16T04:54:26.203223+0000","last_fullsized":"2024-09-16T04:54:26.203223+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12792,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176835+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:15.176835+0000","last_peered":"2024-09-16T04:54:15.176835+0000","last_clean":"2024-09-16T04:54:15.176835+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:15.176835+0000","last_undegraded":"2024-09-16T04:54:15.176835+0000","last_fullsized":"2024-09-16T04:54:15.176835+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:18.376544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12803,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:15.178185+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:15.178185+0000","last_peered":"2024-09-16T04:54:15.178185+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:15.178185+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17247,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176413+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:15.176413+0000","last_peered":"2024-09-16T04:54:15.176413+0000","last_clean":"2024-09-16T04:54:15.176413+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:15.176413+0000","last_undegraded":"2024-09-16T04:54:15.176413+0000","last_fullsized":"2024-09-16T04:54:15.176413+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12908,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:26.204101+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:26.204101+0000","last_peered":"2024-09-16T04:54:26.204101+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:26.204101+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16824,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:54:26.203585+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:26.203585+0000","last_peered":"2024-09-16T04:54:26.203585+0000","last_clean":"2024-09-16T04:54:26.203585+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:26.203585+0000","last_undegraded":"2024-09-16T04:54:26.203585+0000","last_fullsized":"2024-09-16T04:54:26.203585+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17171,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:54:26.211398+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:26.211398+0000","last_peered":"2024-09-16T04:54:26.211398+0000","last_clean":"2024-09-16T04:54:26.211398+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:26.211398+0000","last_undegraded":"2024-09-16T04:54:26.211398+0000","last_fullsized":"2024-09-16T04:54:26.211398+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17379,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176641+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:15.176641+0000","last_peered":"2024-09-16T04:54:15.176641+0000","last_clean":"2024-09-16T04:54:15.176641+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:15.176641+0000","last_undegraded":"2024-09-16T04:54:15.176641+0000","last_fullsized":"2024-09-16T04:54:15.176641+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:17.353717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17546,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:54:26.203482+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:26.203482+0000","last_peered":"2024-09-16T04:54:26.203482+0000","last_clean":"2024-09-16T04:54:26.203482+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:26.203482+0000","last_undegraded":"2024-09-16T04:54:26.203482+0000","last_fullsized":"2024-09-16T04:54:26.203482+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":16936,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:25.016415+0000","last_change":"2024-09-16T04:54:13.785237+0000","last_active":"2024-09-16T04:54:25.016415+0000","last_peered":"2024-09-16T04:54:25.016415+0000","last_clean":"2024-09-16T04:40:14.617336+0000","last_became_active":"2024-09-16T04:41:27.838073+0000","last_became_peered":"2024-09-16T04:41:27.838073+0000","last_unstale":"2024-09-16T04:54:25.016415+0000","last_undegraded":"2024-09-16T04:41:27.523521+0000","last_fullsized":"2024-09-16T04:41:27.486999+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2032,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-170}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12657,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:26.203787+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:26.203787+0000","last_peered":"2024-09-16T04:54:26.203787+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:26.203787+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12868,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:26.921002+0000","last_change":"2024-09-16T04:54:25.748512+0000","last_active":"2024-09-16T04:54:26.921002+0000","last_peered":"2024-09-16T04:54:26.921002+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:26.921002+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1901}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12648,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176523+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:15.176523+0000","last_peered":"2024-09-16T04:54:15.176523+0000","last_clean":"2024-09-16T04:54:15.176523+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:15.176523+0000","last_undegraded":"2024-09-16T04:54:15.176523+0000","last_fullsized":"2024-09-16T04:54:15.176523+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12949,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:26.204077+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:26.204077+0000","last_peered":"2024-09-16T04:54:26.204077+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:26.204077+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14953,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176601+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:15.176601+0000","last_peered":"2024-09-16T04:54:15.176601+0000","last_clean":"2024-09-16T04:54:15.176601+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:15.176601+0000","last_undegraded":"2024-09-16T04:54:15.176601+0000","last_fullsized":"2024-09-16T04:54:15.176601+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:15.348096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12423,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T04:54:15.176778+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:15.176778+0000","last_peered":"2024-09-16T04:54:15.176778+0000","last_clean":"2024-09-16T04:54:15.176778+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:15.176778+0000","last_undegraded":"2024-09-16T04:54:15.176778+0000","last_fullsized":"2024-09-16T04:54:15.176778+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:19.386166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21512,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722112,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649728,"statfs":{"total":96636764160,"available":95897321472,"internally_reserved":0,"allocated":438272,"data_stored":167603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731320,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640520,"statfs":{"total":96636764160,"available":95887892480,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614332,"kb_used_data":940,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93757508,"statfs":{"total":96636764160,"available":96007688192,"internally_reserved":0,"allocated":962560,"data_stored":696830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628092623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722040,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649800,"statfs":{"total":96636764160,"available":95897395200,"internally_reserved":0,"allocated":430080,"data_stored":164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:33.487 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:33.787 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:33.787 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:33.800 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-09-16T04:54:32.824853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22526,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781544,"kb_used_data":2260,"kb_used_omap":14,"kb_used_meta":2779185,"kb_avail":374705816,"statfs":{"total":386547056640,"available":383698755584,"internally_reserved":0,"allocated":2314240,"data_stored":1211724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2845886001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001371"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17190,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213929+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:27.213929+0000","last_peered":"2024-09-16T04:54:27.213929+0000","last_clean":"2024-09-16T04:54:27.213929+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:27.213929+0000","last_undegraded":"2024-09-16T04:54:27.213929+0000","last_fullsized":"2024-09-16T04:54:27.213929+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12796,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212394+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:27.212394+0000","last_peered":"2024-09-16T04:54:27.212394+0000","last_clean":"2024-09-16T04:54:27.212394+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:27.212394+0000","last_undegraded":"2024-09-16T04:54:27.212394+0000","last_fullsized":"2024-09-16T04:54:27.212394+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:30.258518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12807,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.214800+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:27.214800+0000","last_peered":"2024-09-16T04:54:27.214800+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:27.214800+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17251,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.211705+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:27.211705+0000","last_peered":"2024-09-16T04:54:27.211705+0000","last_clean":"2024-09-16T04:54:27.211705+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:27.211705+0000","last_undegraded":"2024-09-16T04:54:27.211705+0000","last_fullsized":"2024-09-16T04:54:27.211705+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:28.250698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12910,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211501+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:27.211501+0000","last_peered":"2024-09-16T04:54:27.211501+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:27.211501+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16826,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213679+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:27.213679+0000","last_peered":"2024-09-16T04:54:27.213679+0000","last_clean":"2024-09-16T04:54:27.213679+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:27.213679+0000","last_undegraded":"2024-09-16T04:54:27.213679+0000","last_fullsized":"2024-09-16T04:54:27.213679+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17173,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.216145+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:27.216145+0000","last_peered":"2024-09-16T04:54:27.216145+0000","last_clean":"2024-09-16T04:54:27.216145+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:27.216145+0000","last_undegraded":"2024-09-16T04:54:27.216145+0000","last_fullsized":"2024-09-16T04:54:27.216145+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:32.082999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17383,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212497+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:27.212497+0000","last_peered":"2024-09-16T04:54:27.212497+0000","last_clean":"2024-09-16T04:54:27.212497+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:27.212497+0000","last_undegraded":"2024-09-16T04:54:27.212497+0000","last_fullsized":"2024-09-16T04:54:27.212497+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:28.250683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17548,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213671+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:27.213671+0000","last_peered":"2024-09-16T04:54:27.213671+0000","last_clean":"2024-09-16T04:54:27.213671+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:27.213671+0000","last_undegraded":"2024-09-16T04:54:27.213671+0000","last_fullsized":"2024-09-16T04:54:27.213671+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:31.896423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17292,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.294697+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:27.294697+0000","last_peered":"2024-09-16T04:54:27.294697+0000","last_clean":"2024-09-16T04:54:27.294697+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:27.294697+0000","last_undegraded":"2024-09-16T04:54:27.294697+0000","last_fullsized":"2024-09-16T04:54:27.294697+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12659,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211597+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:27.211597+0000","last_peered":"2024-09-16T04:54:27.211597+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:27.211597+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":14554,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:31.920506+0000","last_change":"2024-09-16T04:54:25.748512+0000","last_active":"2024-09-16T04:54:31.920506+0000","last_peered":"2024-09-16T04:54:31.920506+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:31.920506+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1059}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12652,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.211824+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:27.211824+0000","last_peered":"2024-09-16T04:54:27.211824+0000","last_clean":"2024-09-16T04:54:27.211824+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:27.211824+0000","last_undegraded":"2024-09-16T04:54:27.211824+0000","last_fullsized":"2024-09-16T04:54:27.211824+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12951,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.212609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:27.212609+0000","last_peered":"2024-09-16T04:54:27.212609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:27.212609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14957,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212117+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:27.212117+0000","last_peered":"2024-09-16T04:54:27.212117+0000","last_clean":"2024-09-16T04:54:27.212117+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:27.212117+0000","last_undegraded":"2024-09-16T04:54:27.212117+0000","last_fullsized":"2024-09-16T04:54:27.212117+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12427,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212585+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:27.212585+0000","last_peered":"2024-09-16T04:54:27.212585+0000","last_clean":"2024-09-16T04:54:27.212585+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:27.212585+0000","last_undegraded":"2024-09-16T04:54:27.212585+0000","last_fullsized":"2024-09-16T04:54:27.212585+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:30.258546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22526,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731336,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640504,"statfs":{"total":96636764160,"available":95887876096,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606028,"kb_used_data":952,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93765812,"statfs":{"total":96636764160,"available":96016191488,"internally_reserved":0,"allocated":974848,"data_stored":699219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619572943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722056,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649784,"statfs":{"total":96636764160,"available":95897378816,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:33.801 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:34.091 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:34.091 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:34.105 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-09-16T04:54:32.824853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22526,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781544,"kb_used_data":2260,"kb_used_omap":14,"kb_used_meta":2779185,"kb_avail":374705816,"statfs":{"total":386547056640,"available":383698755584,"internally_reserved":0,"allocated":2314240,"data_stored":1211724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2845886001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001371"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17190,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213929+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:27.213929+0000","last_peered":"2024-09-16T04:54:27.213929+0000","last_clean":"2024-09-16T04:54:27.213929+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:27.213929+0000","last_undegraded":"2024-09-16T04:54:27.213929+0000","last_fullsized":"2024-09-16T04:54:27.213929+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12796,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212394+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:27.212394+0000","last_peered":"2024-09-16T04:54:27.212394+0000","last_clean":"2024-09-16T04:54:27.212394+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:27.212394+0000","last_undegraded":"2024-09-16T04:54:27.212394+0000","last_fullsized":"2024-09-16T04:54:27.212394+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:30.258518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12807,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.214800+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:27.214800+0000","last_peered":"2024-09-16T04:54:27.214800+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:27.214800+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17251,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.211705+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:27.211705+0000","last_peered":"2024-09-16T04:54:27.211705+0000","last_clean":"2024-09-16T04:54:27.211705+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:27.211705+0000","last_undegraded":"2024-09-16T04:54:27.211705+0000","last_fullsized":"2024-09-16T04:54:27.211705+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:28.250698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12910,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211501+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:27.211501+0000","last_peered":"2024-09-16T04:54:27.211501+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:27.211501+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16826,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213679+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:27.213679+0000","last_peered":"2024-09-16T04:54:27.213679+0000","last_clean":"2024-09-16T04:54:27.213679+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:27.213679+0000","last_undegraded":"2024-09-16T04:54:27.213679+0000","last_fullsized":"2024-09-16T04:54:27.213679+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17173,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.216145+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:27.216145+0000","last_peered":"2024-09-16T04:54:27.216145+0000","last_clean":"2024-09-16T04:54:27.216145+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:27.216145+0000","last_undegraded":"2024-09-16T04:54:27.216145+0000","last_fullsized":"2024-09-16T04:54:27.216145+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:32.082999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17383,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212497+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:27.212497+0000","last_peered":"2024-09-16T04:54:27.212497+0000","last_clean":"2024-09-16T04:54:27.212497+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:27.212497+0000","last_undegraded":"2024-09-16T04:54:27.212497+0000","last_fullsized":"2024-09-16T04:54:27.212497+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:28.250683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17548,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213671+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:27.213671+0000","last_peered":"2024-09-16T04:54:27.213671+0000","last_clean":"2024-09-16T04:54:27.213671+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:27.213671+0000","last_undegraded":"2024-09-16T04:54:27.213671+0000","last_fullsized":"2024-09-16T04:54:27.213671+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:31.896423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17292,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.294697+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:27.294697+0000","last_peered":"2024-09-16T04:54:27.294697+0000","last_clean":"2024-09-16T04:54:27.294697+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:27.294697+0000","last_undegraded":"2024-09-16T04:54:27.294697+0000","last_fullsized":"2024-09-16T04:54:27.294697+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12659,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211597+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:27.211597+0000","last_peered":"2024-09-16T04:54:27.211597+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:27.211597+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":14554,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:31.920506+0000","last_change":"2024-09-16T04:54:25.748512+0000","last_active":"2024-09-16T04:54:31.920506+0000","last_peered":"2024-09-16T04:54:31.920506+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:31.920506+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1059}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12652,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.211824+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:27.211824+0000","last_peered":"2024-09-16T04:54:27.211824+0000","last_clean":"2024-09-16T04:54:27.211824+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:27.211824+0000","last_undegraded":"2024-09-16T04:54:27.211824+0000","last_fullsized":"2024-09-16T04:54:27.211824+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12951,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.212609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:27.212609+0000","last_peered":"2024-09-16T04:54:27.212609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:27.212609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14957,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212117+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:27.212117+0000","last_peered":"2024-09-16T04:54:27.212117+0000","last_clean":"2024-09-16T04:54:27.212117+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:27.212117+0000","last_undegraded":"2024-09-16T04:54:27.212117+0000","last_fullsized":"2024-09-16T04:54:27.212117+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12427,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212585+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:27.212585+0000","last_peered":"2024-09-16T04:54:27.212585+0000","last_clean":"2024-09-16T04:54:27.212585+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:27.212585+0000","last_undegraded":"2024-09-16T04:54:27.212585+0000","last_fullsized":"2024-09-16T04:54:27.212585+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:30.258546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22526,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731336,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640504,"statfs":{"total":96636764160,"available":95887876096,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":606028,"kb_used_data":952,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93765812,"statfs":{"total":96636764160,"available":96016191488,"internally_reserved":0,"allocated":974848,"data_stored":699219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":619572943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722056,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649784,"statfs":{"total":96636764160,"available":95897378816,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:34.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:54:34.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12807, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:27.214800+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:54:27.214800+0000', 'last_peered': '2024-09-16T04:54:27.214800+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:54:27.214800+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:34.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:54:34.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12910, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:27.211501+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:54:27.211501+0000', 'last_peered': '2024-09-16T04:54:27.211501+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:54:27.211501+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:34.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:54:34.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12659, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:27.211597+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:54:27.211597+0000', 'last_peered': '2024-09-16T04:54:27.211597+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:54:27.211597+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:34.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:54:34.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 14554, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:31.920506+0000', 'last_change': '2024-09-16T04:54:25.748512+0000', 'last_active': '2024-09-16T04:54:31.920506+0000', 'last_peered': '2024-09-16T04:54:31.920506+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:54:31.920506+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1070, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1059}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:34.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:54:34.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12951, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:27.212609+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:54:27.212609+0000', 'last_peered': '2024-09-16T04:54:27.212609+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:54:27.212609+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:34.107 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:54:34.485 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:34.485 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9888/112 objects degraded (8828.571%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":886,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2848301056,"bytes_avail":383698755584,"bytes_total":386547056640,"degraded_objects":9888,"degraded_total":112,"degraded_ratio":88.285714285714292,"recovering_objects_per_sec":282,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2262,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:54:34.498 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:54:34.498 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:34.842 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:34.842 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:34.854 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":488,"stamp":"2024-09-16T04:54:34.825241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22526,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2807908,"kb_used_data":2260,"kb_used_omap":14,"kb_used_meta":2805553,"kb_avail":374679452,"statfs":{"total":386547056640,"available":383671758848,"internally_reserved":0,"allocated":2314240,"data_stored":1211724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2872886833},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1218,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.623198"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17190,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213929+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:27.213929+0000","last_peered":"2024-09-16T04:54:27.213929+0000","last_clean":"2024-09-16T04:54:27.213929+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:27.213929+0000","last_undegraded":"2024-09-16T04:54:27.213929+0000","last_fullsized":"2024-09-16T04:54:27.213929+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12796,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212394+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:27.212394+0000","last_peered":"2024-09-16T04:54:27.212394+0000","last_clean":"2024-09-16T04:54:27.212394+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:27.212394+0000","last_undegraded":"2024-09-16T04:54:27.212394+0000","last_fullsized":"2024-09-16T04:54:27.212394+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:30.258518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12807,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.214800+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:27.214800+0000","last_peered":"2024-09-16T04:54:27.214800+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:27.214800+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17251,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.211705+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:27.211705+0000","last_peered":"2024-09-16T04:54:27.211705+0000","last_clean":"2024-09-16T04:54:27.211705+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:27.211705+0000","last_undegraded":"2024-09-16T04:54:27.211705+0000","last_fullsized":"2024-09-16T04:54:27.211705+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:28.250698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12910,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211501+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:27.211501+0000","last_peered":"2024-09-16T04:54:27.211501+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:27.211501+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16826,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213679+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:27.213679+0000","last_peered":"2024-09-16T04:54:27.213679+0000","last_clean":"2024-09-16T04:54:27.213679+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:27.213679+0000","last_undegraded":"2024-09-16T04:54:27.213679+0000","last_fullsized":"2024-09-16T04:54:27.213679+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17173,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.216145+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:27.216145+0000","last_peered":"2024-09-16T04:54:27.216145+0000","last_clean":"2024-09-16T04:54:27.216145+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:27.216145+0000","last_undegraded":"2024-09-16T04:54:27.216145+0000","last_fullsized":"2024-09-16T04:54:27.216145+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:32.082999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17383,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212497+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:27.212497+0000","last_peered":"2024-09-16T04:54:27.212497+0000","last_clean":"2024-09-16T04:54:27.212497+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:27.212497+0000","last_undegraded":"2024-09-16T04:54:27.212497+0000","last_fullsized":"2024-09-16T04:54:27.212497+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:28.250683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17548,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213671+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:27.213671+0000","last_peered":"2024-09-16T04:54:27.213671+0000","last_clean":"2024-09-16T04:54:27.213671+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:27.213671+0000","last_undegraded":"2024-09-16T04:54:27.213671+0000","last_fullsized":"2024-09-16T04:54:27.213671+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:31.896423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17292,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.294697+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:27.294697+0000","last_peered":"2024-09-16T04:54:27.294697+0000","last_clean":"2024-09-16T04:54:27.294697+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:27.294697+0000","last_undegraded":"2024-09-16T04:54:27.294697+0000","last_fullsized":"2024-09-16T04:54:27.294697+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12659,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211597+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:27.211597+0000","last_peered":"2024-09-16T04:54:27.211597+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:27.211597+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":14554,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:31.920506+0000","last_change":"2024-09-16T04:54:25.748512+0000","last_active":"2024-09-16T04:54:31.920506+0000","last_peered":"2024-09-16T04:54:31.920506+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:31.920506+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1059}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12652,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.211824+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:27.211824+0000","last_peered":"2024-09-16T04:54:27.211824+0000","last_clean":"2024-09-16T04:54:27.211824+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:27.211824+0000","last_undegraded":"2024-09-16T04:54:27.211824+0000","last_fullsized":"2024-09-16T04:54:27.211824+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12951,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.212609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:27.212609+0000","last_peered":"2024-09-16T04:54:27.212609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:27.212609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14957,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212117+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:27.212117+0000","last_peered":"2024-09-16T04:54:27.212117+0000","last_clean":"2024-09-16T04:54:27.212117+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:27.212117+0000","last_undegraded":"2024-09-16T04:54:27.212117+0000","last_fullsized":"2024-09-16T04:54:27.212117+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12427,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212585+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:27.212585+0000","last_peered":"2024-09-16T04:54:27.212585+0000","last_clean":"2024-09-16T04:54:27.212585+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:27.212585+0000","last_undegraded":"2024-09-16T04:54:27.212585+0000","last_fullsized":"2024-09-16T04:54:27.212585+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:30.258546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22526,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731336,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640504,"statfs":{"total":96636764160,"available":95887876096,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632392,"kb_used_data":952,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739448,"statfs":{"total":96636764160,"available":95989194752,"internally_reserved":0,"allocated":974848,"data_stored":699219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646573775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722056,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649784,"statfs":{"total":96636764160,"available":95897378816,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:37.856 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:38.159 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:38.160 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:38.172 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-09-16T04:54:37.278176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23586,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2807908,"kb_used_data":2260,"kb_used_omap":14,"kb_used_meta":2805553,"kb_avail":374679452,"statfs":{"total":386547056640,"available":383671758848,"internally_reserved":0,"allocated":2314240,"data_stored":1211724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2872886833},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2074,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2074,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.067778"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17190,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213929+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:27.213929+0000","last_peered":"2024-09-16T04:54:27.213929+0000","last_clean":"2024-09-16T04:54:27.213929+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:27.213929+0000","last_undegraded":"2024-09-16T04:54:27.213929+0000","last_fullsized":"2024-09-16T04:54:27.213929+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12796,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212394+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:27.212394+0000","last_peered":"2024-09-16T04:54:27.212394+0000","last_clean":"2024-09-16T04:54:27.212394+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:27.212394+0000","last_undegraded":"2024-09-16T04:54:27.212394+0000","last_fullsized":"2024-09-16T04:54:27.212394+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:30.258518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12807,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.214800+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:27.214800+0000","last_peered":"2024-09-16T04:54:27.214800+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:27.214800+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17251,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.211705+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:27.211705+0000","last_peered":"2024-09-16T04:54:27.211705+0000","last_clean":"2024-09-16T04:54:27.211705+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:27.211705+0000","last_undegraded":"2024-09-16T04:54:27.211705+0000","last_fullsized":"2024-09-16T04:54:27.211705+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:28.250698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12910,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211501+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:27.211501+0000","last_peered":"2024-09-16T04:54:27.211501+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:27.211501+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16826,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213679+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:27.213679+0000","last_peered":"2024-09-16T04:54:27.213679+0000","last_clean":"2024-09-16T04:54:27.213679+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:27.213679+0000","last_undegraded":"2024-09-16T04:54:27.213679+0000","last_fullsized":"2024-09-16T04:54:27.213679+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17173,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.216145+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:27.216145+0000","last_peered":"2024-09-16T04:54:27.216145+0000","last_clean":"2024-09-16T04:54:27.216145+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:27.216145+0000","last_undegraded":"2024-09-16T04:54:27.216145+0000","last_fullsized":"2024-09-16T04:54:27.216145+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:32.082999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17383,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212497+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:27.212497+0000","last_peered":"2024-09-16T04:54:27.212497+0000","last_clean":"2024-09-16T04:54:27.212497+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:27.212497+0000","last_undegraded":"2024-09-16T04:54:27.212497+0000","last_fullsized":"2024-09-16T04:54:27.212497+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:28.250683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17548,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213671+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:27.213671+0000","last_peered":"2024-09-16T04:54:27.213671+0000","last_clean":"2024-09-16T04:54:27.213671+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:27.213671+0000","last_undegraded":"2024-09-16T04:54:27.213671+0000","last_fullsized":"2024-09-16T04:54:27.213671+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:31.896423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17292,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.294697+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:27.294697+0000","last_peered":"2024-09-16T04:54:27.294697+0000","last_clean":"2024-09-16T04:54:27.294697+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:27.294697+0000","last_undegraded":"2024-09-16T04:54:27.294697+0000","last_fullsized":"2024-09-16T04:54:27.294697+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12659,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211597+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:27.211597+0000","last_peered":"2024-09-16T04:54:27.211597+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:27.211597+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":16675,"reported_epoch":53,"state":"active+undersized+remapped","last_fresh":"2024-09-16T04:54:36.735749+0000","last_change":"2024-09-16T04:54:36.736015+0000","last_active":"2024-09-16T04:54:36.735749+0000","last_peered":"2024-09-16T04:54:36.735749+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:36.735749+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","1(0)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12652,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.211824+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:27.211824+0000","last_peered":"2024-09-16T04:54:27.211824+0000","last_clean":"2024-09-16T04:54:27.211824+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:27.211824+0000","last_undegraded":"2024-09-16T04:54:27.211824+0000","last_fullsized":"2024-09-16T04:54:27.211824+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12951,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.212609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:27.212609+0000","last_peered":"2024-09-16T04:54:27.212609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:27.212609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14957,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212117+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:27.212117+0000","last_peered":"2024-09-16T04:54:27.212117+0000","last_clean":"2024-09-16T04:54:27.212117+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:27.212117+0000","last_undegraded":"2024-09-16T04:54:27.212117+0000","last_fullsized":"2024-09-16T04:54:27.212117+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12427,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212585+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:27.212585+0000","last_peered":"2024-09-16T04:54:27.212585+0000","last_clean":"2024-09-16T04:54:27.212585+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:27.212585+0000","last_undegraded":"2024-09-16T04:54:27.212585+0000","last_fullsized":"2024-09-16T04:54:27.212585+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:30.258546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23586,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731336,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640504,"statfs":{"total":96636764160,"available":95887876096,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632392,"kb_used_data":952,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739448,"statfs":{"total":96636764160,"available":95989194752,"internally_reserved":0,"allocated":974848,"data_stored":699219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646573775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722056,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649784,"statfs":{"total":96636764160,"available":95897378816,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:38.174 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:38.491 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:38.492 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:38.505 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2024-09-16T04:54:38.284177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23586,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2807908,"kb_used_data":2260,"kb_used_omap":14,"kb_used_meta":2805553,"kb_avail":374679452,"statfs":{"total":386547056640,"available":383671758848,"internally_reserved":0,"allocated":2314240,"data_stored":1211724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2872886833},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2074,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2074,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.459975"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17190,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213929+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:27.213929+0000","last_peered":"2024-09-16T04:54:27.213929+0000","last_clean":"2024-09-16T04:54:27.213929+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:27.213929+0000","last_undegraded":"2024-09-16T04:54:27.213929+0000","last_fullsized":"2024-09-16T04:54:27.213929+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12796,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212394+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:27.212394+0000","last_peered":"2024-09-16T04:54:27.212394+0000","last_clean":"2024-09-16T04:54:27.212394+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:27.212394+0000","last_undegraded":"2024-09-16T04:54:27.212394+0000","last_fullsized":"2024-09-16T04:54:27.212394+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:30.258518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12807,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.214800+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:27.214800+0000","last_peered":"2024-09-16T04:54:27.214800+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:27.214800+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17251,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.211705+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:27.211705+0000","last_peered":"2024-09-16T04:54:27.211705+0000","last_clean":"2024-09-16T04:54:27.211705+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:27.211705+0000","last_undegraded":"2024-09-16T04:54:27.211705+0000","last_fullsized":"2024-09-16T04:54:27.211705+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:28.250698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12910,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211501+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:27.211501+0000","last_peered":"2024-09-16T04:54:27.211501+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:27.211501+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16826,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213679+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:27.213679+0000","last_peered":"2024-09-16T04:54:27.213679+0000","last_clean":"2024-09-16T04:54:27.213679+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:27.213679+0000","last_undegraded":"2024-09-16T04:54:27.213679+0000","last_fullsized":"2024-09-16T04:54:27.213679+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17173,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.216145+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:27.216145+0000","last_peered":"2024-09-16T04:54:27.216145+0000","last_clean":"2024-09-16T04:54:27.216145+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:27.216145+0000","last_undegraded":"2024-09-16T04:54:27.216145+0000","last_fullsized":"2024-09-16T04:54:27.216145+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:32.082999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17383,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212497+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:27.212497+0000","last_peered":"2024-09-16T04:54:27.212497+0000","last_clean":"2024-09-16T04:54:27.212497+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:27.212497+0000","last_undegraded":"2024-09-16T04:54:27.212497+0000","last_fullsized":"2024-09-16T04:54:27.212497+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:28.250683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17548,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213671+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:27.213671+0000","last_peered":"2024-09-16T04:54:27.213671+0000","last_clean":"2024-09-16T04:54:27.213671+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:27.213671+0000","last_undegraded":"2024-09-16T04:54:27.213671+0000","last_fullsized":"2024-09-16T04:54:27.213671+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:31.896423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17292,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.294697+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:27.294697+0000","last_peered":"2024-09-16T04:54:27.294697+0000","last_clean":"2024-09-16T04:54:27.294697+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:27.294697+0000","last_undegraded":"2024-09-16T04:54:27.294697+0000","last_fullsized":"2024-09-16T04:54:27.294697+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12659,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211597+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:27.211597+0000","last_peered":"2024-09-16T04:54:27.211597+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:27.211597+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":16675,"reported_epoch":53,"state":"active+undersized+remapped","last_fresh":"2024-09-16T04:54:36.735749+0000","last_change":"2024-09-16T04:54:36.736015+0000","last_active":"2024-09-16T04:54:36.735749+0000","last_peered":"2024-09-16T04:54:36.735749+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:36.735749+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","1(0)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12652,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.211824+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:27.211824+0000","last_peered":"2024-09-16T04:54:27.211824+0000","last_clean":"2024-09-16T04:54:27.211824+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:27.211824+0000","last_undegraded":"2024-09-16T04:54:27.211824+0000","last_fullsized":"2024-09-16T04:54:27.211824+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12951,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.212609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:27.212609+0000","last_peered":"2024-09-16T04:54:27.212609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:27.212609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14957,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212117+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:27.212117+0000","last_peered":"2024-09-16T04:54:27.212117+0000","last_clean":"2024-09-16T04:54:27.212117+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:27.212117+0000","last_undegraded":"2024-09-16T04:54:27.212117+0000","last_fullsized":"2024-09-16T04:54:27.212117+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12427,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212585+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:27.212585+0000","last_peered":"2024-09-16T04:54:27.212585+0000","last_clean":"2024-09-16T04:54:27.212585+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:27.212585+0000","last_undegraded":"2024-09-16T04:54:27.212585+0000","last_fullsized":"2024-09-16T04:54:27.212585+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:30.258546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23586,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731336,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640504,"statfs":{"total":96636764160,"available":95887876096,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632392,"kb_used_data":952,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739448,"statfs":{"total":96636764160,"available":95989194752,"internally_reserved":0,"allocated":974848,"data_stored":699219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646573775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722056,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649784,"statfs":{"total":96636764160,"available":95897378816,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:38.506 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:54:38.506 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 12807, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:27.214800+0000', 'last_change': '2024-09-16T04:41:27.863527+0000', 'last_active': '2024-09-16T04:54:27.214800+0000', 'last_peered': '2024-09-16T04:54:27.214800+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:54:27.214800+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2220, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2218}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:38.506 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:54:38.506 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12910, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:27.211501+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:54:27.211501+0000', 'last_peered': '2024-09-16T04:54:27.211501+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:54:27.211501+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:38.506 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:54:38.506 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12659, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:27.211597+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:54:27.211597+0000', 'last_peered': '2024-09-16T04:54:27.211597+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:54:27.211597+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:38.506 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-16T04:54:38.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 16675, 'reported_epoch': 53, 'state': 'active+undersized+remapped', 'last_fresh': '2024-09-16T04:54:36.735749+0000', 'last_change': '2024-09-16T04:54:36.736015+0000', 'last_active': '2024-09-16T04:54:36.735749+0000', 'last_peered': '2024-09-16T04:54:36.735749+0000', 'last_clean': '2024-09-16T04:40:14.617736+0000', 'last_became_active': '2024-09-16T04:41:27.829720+0000', 'last_became_peered': '2024-09-16T04:41:27.829720+0000', 'last_unstale': '2024-09-16T04:54:36.735749+0000', 'last_undegraded': '2024-09-16T04:41:27.546906+0000', 'last_fullsized': '2024-09-16T04:41:27.483801+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2130, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '1(0)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:38.507 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:54:38.507 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12951, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:27.212609+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:54:27.212609+0000', 'last_peered': '2024-09-16T04:54:27.212609+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:54:27.212609+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:38.507 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:54:38.870 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:38.871 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9888/112 objects degraded (8828.571%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":891,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2875297792,"bytes_avail":383671758848,"bytes_total":386547056640,"degraded_objects":9888,"degraded_total":112,"degraded_ratio":88.285714285714292,"recovering_objects_per_sec":121,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1218,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:54:38.884 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:54:38.885 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:39.185 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:39.185 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:39.198 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-16T04:54:38.826035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23586,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2807908,"kb_used_data":2260,"kb_used_omap":14,"kb_used_meta":2805553,"kb_avail":374679452,"statfs":{"total":386547056640,"available":383671758848,"internally_reserved":0,"allocated":2314240,"data_stored":1211724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2872886833},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1902,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001510"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17190,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213929+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:27.213929+0000","last_peered":"2024-09-16T04:54:27.213929+0000","last_clean":"2024-09-16T04:54:27.213929+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:27.213929+0000","last_undegraded":"2024-09-16T04:54:27.213929+0000","last_fullsized":"2024-09-16T04:54:27.213929+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12796,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212394+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:27.212394+0000","last_peered":"2024-09-16T04:54:27.212394+0000","last_clean":"2024-09-16T04:54:27.212394+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:27.212394+0000","last_undegraded":"2024-09-16T04:54:27.212394+0000","last_fullsized":"2024-09-16T04:54:27.212394+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:30.258518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":12807,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.214800+0000","last_change":"2024-09-16T04:41:27.863527+0000","last_active":"2024-09-16T04:54:27.214800+0000","last_peered":"2024-09-16T04:54:27.214800+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:27.214800+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17251,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.211705+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:27.211705+0000","last_peered":"2024-09-16T04:54:27.211705+0000","last_clean":"2024-09-16T04:54:27.211705+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:27.211705+0000","last_undegraded":"2024-09-16T04:54:27.211705+0000","last_fullsized":"2024-09-16T04:54:27.211705+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:28.250698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12910,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211501+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:27.211501+0000","last_peered":"2024-09-16T04:54:27.211501+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:27.211501+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16826,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213679+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:27.213679+0000","last_peered":"2024-09-16T04:54:27.213679+0000","last_clean":"2024-09-16T04:54:27.213679+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:27.213679+0000","last_undegraded":"2024-09-16T04:54:27.213679+0000","last_fullsized":"2024-09-16T04:54:27.213679+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17173,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.216145+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:27.216145+0000","last_peered":"2024-09-16T04:54:27.216145+0000","last_clean":"2024-09-16T04:54:27.216145+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:27.216145+0000","last_undegraded":"2024-09-16T04:54:27.216145+0000","last_fullsized":"2024-09-16T04:54:27.216145+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:32.082999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17383,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212497+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:27.212497+0000","last_peered":"2024-09-16T04:54:27.212497+0000","last_clean":"2024-09-16T04:54:27.212497+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:27.212497+0000","last_undegraded":"2024-09-16T04:54:27.212497+0000","last_fullsized":"2024-09-16T04:54:27.212497+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:28.250683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17548,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213671+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:27.213671+0000","last_peered":"2024-09-16T04:54:27.213671+0000","last_clean":"2024-09-16T04:54:27.213671+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:27.213671+0000","last_undegraded":"2024-09-16T04:54:27.213671+0000","last_fullsized":"2024-09-16T04:54:27.213671+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:31.896423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17292,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.294697+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:27.294697+0000","last_peered":"2024-09-16T04:54:27.294697+0000","last_clean":"2024-09-16T04:54:27.294697+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:27.294697+0000","last_undegraded":"2024-09-16T04:54:27.294697+0000","last_fullsized":"2024-09-16T04:54:27.294697+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12659,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211597+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:27.211597+0000","last_peered":"2024-09-16T04:54:27.211597+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:27.211597+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":16675,"reported_epoch":53,"state":"active+undersized+remapped","last_fresh":"2024-09-16T04:54:36.735749+0000","last_change":"2024-09-16T04:54:36.736015+0000","last_active":"2024-09-16T04:54:36.735749+0000","last_peered":"2024-09-16T04:54:36.735749+0000","last_clean":"2024-09-16T04:40:14.617736+0000","last_became_active":"2024-09-16T04:41:27.829720+0000","last_became_peered":"2024-09-16T04:41:27.829720+0000","last_unstale":"2024-09-16T04:54:36.735749+0000","last_undegraded":"2024-09-16T04:41:27.546906+0000","last_fullsized":"2024-09-16T04:41:27.483801+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","1(0)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12652,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.211824+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:27.211824+0000","last_peered":"2024-09-16T04:54:27.211824+0000","last_clean":"2024-09-16T04:54:27.211824+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:27.211824+0000","last_undegraded":"2024-09-16T04:54:27.211824+0000","last_fullsized":"2024-09-16T04:54:27.211824+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12951,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.212609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:27.212609+0000","last_peered":"2024-09-16T04:54:27.212609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:27.212609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14957,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212117+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:27.212117+0000","last_peered":"2024-09-16T04:54:27.212117+0000","last_clean":"2024-09-16T04:54:27.212117+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:27.212117+0000","last_undegraded":"2024-09-16T04:54:27.212117+0000","last_fullsized":"2024-09-16T04:54:27.212117+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12427,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.212585+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:27.212585+0000","last_peered":"2024-09-16T04:54:27.212585+0000","last_clean":"2024-09-16T04:54:27.212585+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:27.212585+0000","last_undegraded":"2024-09-16T04:54:27.212585+0000","last_fullsized":"2024-09-16T04:54:27.212585+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:30.258546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23586,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731336,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640504,"statfs":{"total":96636764160,"available":95887876096,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632392,"kb_used_data":952,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739448,"statfs":{"total":96636764160,"available":95989194752,"internally_reserved":0,"allocated":974848,"data_stored":699219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646573775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722056,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649784,"statfs":{"total":96636764160,"available":95897378816,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:42.200 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:42.489 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:42.489 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:42.501 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2024-09-16T04:54:40.826332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21940,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2816268,"kb_used_data":2288,"kb_used_omap":14,"kb_used_meta":2813873,"kb_avail":374671092,"statfs":{"total":386547056640,"available":383663198208,"internally_reserved":0,"allocated":2342912,"data_stored":1220450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2881406513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-586,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001478"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17190,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213929+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:27.213929+0000","last_peered":"2024-09-16T04:54:27.213929+0000","last_clean":"2024-09-16T04:54:27.213929+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:27.213929+0000","last_undegraded":"2024-09-16T04:54:27.213929+0000","last_fullsized":"2024-09-16T04:54:27.213929+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12800,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.286542+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:38.286542+0000","last_peered":"2024-09-16T04:54:38.286542+0000","last_clean":"2024-09-16T04:54:38.286542+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:38.286542+0000","last_undegraded":"2024-09-16T04:54:38.286542+0000","last_fullsized":"2024-09-16T04:54:38.286542+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.023708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":13732,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:40.016084+0000","last_change":"2024-09-16T04:54:36.736779+0000","last_active":"2024-09-16T04:54:40.016084+0000","last_peered":"2024-09-16T04:54:40.016084+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:40.016084+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17255,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285241+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:38.285241+0000","last_peered":"2024-09-16T04:54:38.285241+0000","last_clean":"2024-09-16T04:54:38.285241+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:38.285241+0000","last_undegraded":"2024-09-16T04:54:38.285241+0000","last_fullsized":"2024-09-16T04:54:38.285241+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:40.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12910,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211501+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:27.211501+0000","last_peered":"2024-09-16T04:54:27.211501+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:27.211501+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16826,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213679+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:27.213679+0000","last_peered":"2024-09-16T04:54:27.213679+0000","last_clean":"2024-09-16T04:54:27.213679+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:27.213679+0000","last_undegraded":"2024-09-16T04:54:27.213679+0000","last_fullsized":"2024-09-16T04:54:27.213679+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17173,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.216145+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:27.216145+0000","last_peered":"2024-09-16T04:54:27.216145+0000","last_clean":"2024-09-16T04:54:27.216145+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:27.216145+0000","last_undegraded":"2024-09-16T04:54:27.216145+0000","last_fullsized":"2024-09-16T04:54:27.216145+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:32.082999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17387,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285634+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:38.285634+0000","last_peered":"2024-09-16T04:54:38.285634+0000","last_clean":"2024-09-16T04:54:38.285634+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:38.285634+0000","last_undegraded":"2024-09-16T04:54:38.285634+0000","last_fullsized":"2024-09-16T04:54:38.285634+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:40.014437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17548,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213671+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:27.213671+0000","last_peered":"2024-09-16T04:54:27.213671+0000","last_clean":"2024-09-16T04:54:27.213671+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:27.213671+0000","last_undegraded":"2024-09-16T04:54:27.213671+0000","last_fullsized":"2024-09-16T04:54:27.213671+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:31.896423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17296,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.287887+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:38.287887+0000","last_peered":"2024-09-16T04:54:38.287887+0000","last_clean":"2024-09-16T04:54:38.287887+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:38.287887+0000","last_undegraded":"2024-09-16T04:54:38.287887+0000","last_fullsized":"2024-09-16T04:54:38.287887+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:38.759937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12659,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211597+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:27.211597+0000","last_peered":"2024-09-16T04:54:27.211597+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:27.211597+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12442,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.367021+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:38.367021+0000","last_peered":"2024-09-16T04:54:38.367021+0000","last_clean":"2024-09-16T04:54:38.367021+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:38.367021+0000","last_undegraded":"2024-09-16T04:54:38.367021+0000","last_fullsized":"2024-09-16T04:54:38.367021+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:44.030027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12656,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285318+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:38.285318+0000","last_peered":"2024-09-16T04:54:38.285318+0000","last_clean":"2024-09-16T04:54:38.285318+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:38.285318+0000","last_undegraded":"2024-09-16T04:54:38.285318+0000","last_fullsized":"2024-09-16T04:54:38.285318+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12951,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.212609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:27.212609+0000","last_peered":"2024-09-16T04:54:27.212609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:27.212609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14961,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285505+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:38.285505+0000","last_peered":"2024-09-16T04:54:38.285505+0000","last_clean":"2024-09-16T04:54:38.285505+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:38.285505+0000","last_undegraded":"2024-09-16T04:54:38.285505+0000","last_fullsized":"2024-09-16T04:54:38.285505+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12431,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285737+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:38.285737+0000","last_peered":"2024-09-16T04:54:38.285737+0000","last_clean":"2024-09-16T04:54:38.285737+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:38.285737+0000","last_undegraded":"2024-09-16T04:54:38.285737+0000","last_fullsized":"2024-09-16T04:54:38.285737+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.023729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21940,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731356,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640484,"statfs":{"total":96636764160,"available":95887855616,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640732,"kb_used_data":964,"kb_used_omap":4,"kb_used_meta":639739,"kb_avail":93731108,"statfs":{"total":96636764160,"available":95980654592,"internally_reserved":0,"allocated":987136,"data_stored":701534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":655093455},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722056,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649784,"statfs":{"total":96636764160,"available":95897378816,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":528384,"data_stored":528384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:42.502 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:42.806 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:42.806 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:42.819 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2024-09-16T04:54:40.826332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21940,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2816268,"kb_used_data":2288,"kb_used_omap":14,"kb_used_meta":2813873,"kb_avail":374671092,"statfs":{"total":386547056640,"available":383663198208,"internally_reserved":0,"allocated":2342912,"data_stored":1220450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2881406513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-586,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001478"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17190,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213929+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:27.213929+0000","last_peered":"2024-09-16T04:54:27.213929+0000","last_clean":"2024-09-16T04:54:27.213929+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:27.213929+0000","last_undegraded":"2024-09-16T04:54:27.213929+0000","last_fullsized":"2024-09-16T04:54:27.213929+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12800,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.286542+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:38.286542+0000","last_peered":"2024-09-16T04:54:38.286542+0000","last_clean":"2024-09-16T04:54:38.286542+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:38.286542+0000","last_undegraded":"2024-09-16T04:54:38.286542+0000","last_fullsized":"2024-09-16T04:54:38.286542+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.023708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":13732,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:40.016084+0000","last_change":"2024-09-16T04:54:36.736779+0000","last_active":"2024-09-16T04:54:40.016084+0000","last_peered":"2024-09-16T04:54:40.016084+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:40.016084+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17255,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285241+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:38.285241+0000","last_peered":"2024-09-16T04:54:38.285241+0000","last_clean":"2024-09-16T04:54:38.285241+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:38.285241+0000","last_undegraded":"2024-09-16T04:54:38.285241+0000","last_fullsized":"2024-09-16T04:54:38.285241+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:40.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12910,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211501+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:27.211501+0000","last_peered":"2024-09-16T04:54:27.211501+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:27.211501+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16826,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213679+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:27.213679+0000","last_peered":"2024-09-16T04:54:27.213679+0000","last_clean":"2024-09-16T04:54:27.213679+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:27.213679+0000","last_undegraded":"2024-09-16T04:54:27.213679+0000","last_fullsized":"2024-09-16T04:54:27.213679+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17173,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.216145+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:27.216145+0000","last_peered":"2024-09-16T04:54:27.216145+0000","last_clean":"2024-09-16T04:54:27.216145+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:27.216145+0000","last_undegraded":"2024-09-16T04:54:27.216145+0000","last_fullsized":"2024-09-16T04:54:27.216145+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:32.082999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17387,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285634+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:38.285634+0000","last_peered":"2024-09-16T04:54:38.285634+0000","last_clean":"2024-09-16T04:54:38.285634+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:38.285634+0000","last_undegraded":"2024-09-16T04:54:38.285634+0000","last_fullsized":"2024-09-16T04:54:38.285634+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:40.014437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17548,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T04:54:27.213671+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:27.213671+0000","last_peered":"2024-09-16T04:54:27.213671+0000","last_clean":"2024-09-16T04:54:27.213671+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:27.213671+0000","last_undegraded":"2024-09-16T04:54:27.213671+0000","last_fullsized":"2024-09-16T04:54:27.213671+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:31.896423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17296,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.287887+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:38.287887+0000","last_peered":"2024-09-16T04:54:38.287887+0000","last_clean":"2024-09-16T04:54:38.287887+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:38.287887+0000","last_undegraded":"2024-09-16T04:54:38.287887+0000","last_fullsized":"2024-09-16T04:54:38.287887+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:38.759937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12659,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.211597+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:27.211597+0000","last_peered":"2024-09-16T04:54:27.211597+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:27.211597+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12442,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.367021+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:38.367021+0000","last_peered":"2024-09-16T04:54:38.367021+0000","last_clean":"2024-09-16T04:54:38.367021+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:38.367021+0000","last_undegraded":"2024-09-16T04:54:38.367021+0000","last_fullsized":"2024-09-16T04:54:38.367021+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:44.030027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12656,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285318+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:38.285318+0000","last_peered":"2024-09-16T04:54:38.285318+0000","last_clean":"2024-09-16T04:54:38.285318+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:38.285318+0000","last_undegraded":"2024-09-16T04:54:38.285318+0000","last_fullsized":"2024-09-16T04:54:38.285318+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12951,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:27.212609+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:27.212609+0000","last_peered":"2024-09-16T04:54:27.212609+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:27.212609+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14961,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285505+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:38.285505+0000","last_peered":"2024-09-16T04:54:38.285505+0000","last_clean":"2024-09-16T04:54:38.285505+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:38.285505+0000","last_undegraded":"2024-09-16T04:54:38.285505+0000","last_fullsized":"2024-09-16T04:54:38.285505+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12431,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285737+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:38.285737+0000","last_peered":"2024-09-16T04:54:38.285737+0000","last_clean":"2024-09-16T04:54:38.285737+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:38.285737+0000","last_undegraded":"2024-09-16T04:54:38.285737+0000","last_fullsized":"2024-09-16T04:54:38.285737+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.023729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21940,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731356,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640484,"statfs":{"total":96636764160,"available":95887855616,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640732,"kb_used_data":964,"kb_used_omap":4,"kb_used_meta":639739,"kb_avail":93731108,"statfs":{"total":96636764160,"available":95980654592,"internally_reserved":0,"allocated":987136,"data_stored":701534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":655093455},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722056,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649784,"statfs":{"total":96636764160,"available":95897378816,"internally_reserved":0,"allocated":446464,"data_stored":170835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":528384,"data_stored":528384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:42.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:54:42.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 13732, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:40.016084+0000', 'last_change': '2024-09-16T04:54:36.736779+0000', 'last_active': '2024-09-16T04:54:40.016084+0000', 'last_peered': '2024-09-16T04:54:40.016084+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:54:40.016084+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1760, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 469, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1759}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:42.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:54:42.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12910, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:27.211501+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:54:27.211501+0000', 'last_peered': '2024-09-16T04:54:27.211501+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:54:27.211501+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:42.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:54:42.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12659, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:27.211597+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:54:27.211597+0000', 'last_peered': '2024-09-16T04:54:27.211597+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:54:27.211597+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:42.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:54:42.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12951, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:27.212609+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:54:27.212609+0000', 'last_peered': '2024-09-16T04:54:27.212609+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:54:27.212609+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:42.821 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:54:43.232 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:43.233 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8368/112 objects degraded (7471.429%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":895,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2883858432,"bytes_avail":383663198208,"bytes_total":386547056640,"degraded_objects":8368,"degraded_total":112,"degraded_ratio":74.714285714285708},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:54:43.248 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T04:54:43.248 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:43.571 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:43.571 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:43.585 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-16T04:54:42.826661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21940,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2816304,"kb_used_data":2320,"kb_used_omap":14,"kb_used_meta":2813873,"kb_avail":374671056,"statfs":{"total":386547056640,"available":383663161344,"internally_reserved":0,"allocated":2375680,"data_stored":1233272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2881406513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-586,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001419"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17194,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284666+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:38.284666+0000","last_peered":"2024-09-16T04:54:38.284666+0000","last_clean":"2024-09-16T04:54:38.284666+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:38.284666+0000","last_undegraded":"2024-09-16T04:54:38.284666+0000","last_fullsized":"2024-09-16T04:54:38.284666+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12800,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.286542+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:38.286542+0000","last_peered":"2024-09-16T04:54:38.286542+0000","last_clean":"2024-09-16T04:54:38.286542+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:38.286542+0000","last_undegraded":"2024-09-16T04:54:38.286542+0000","last_fullsized":"2024-09-16T04:54:38.286542+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.023708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":13732,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:40.016084+0000","last_change":"2024-09-16T04:54:36.736779+0000","last_active":"2024-09-16T04:54:40.016084+0000","last_peered":"2024-09-16T04:54:40.016084+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:40.016084+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1759}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17255,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285241+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:38.285241+0000","last_peered":"2024-09-16T04:54:38.285241+0000","last_clean":"2024-09-16T04:54:38.285241+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:38.285241+0000","last_undegraded":"2024-09-16T04:54:38.285241+0000","last_fullsized":"2024-09-16T04:54:38.285241+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:40.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12914,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.285601+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:38.285601+0000","last_peered":"2024-09-16T04:54:38.285601+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:38.285601+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16830,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284848+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:38.284848+0000","last_peered":"2024-09-16T04:54:38.284848+0000","last_clean":"2024-09-16T04:54:38.284848+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:38.284848+0000","last_undegraded":"2024-09-16T04:54:38.284848+0000","last_fullsized":"2024-09-16T04:54:38.284848+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17177,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.289049+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:38.289049+0000","last_peered":"2024-09-16T04:54:38.289049+0000","last_clean":"2024-09-16T04:54:38.289049+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:38.289049+0000","last_undegraded":"2024-09-16T04:54:38.289049+0000","last_fullsized":"2024-09-16T04:54:38.289049+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.205935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17387,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285634+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:38.285634+0000","last_peered":"2024-09-16T04:54:38.285634+0000","last_clean":"2024-09-16T04:54:38.285634+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:38.285634+0000","last_undegraded":"2024-09-16T04:54:38.285634+0000","last_fullsized":"2024-09-16T04:54:38.285634+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:40.014437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17552,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284772+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:38.284772+0000","last_peered":"2024-09-16T04:54:38.284772+0000","last_clean":"2024-09-16T04:54:38.284772+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:38.284772+0000","last_undegraded":"2024-09-16T04:54:38.284772+0000","last_fullsized":"2024-09-16T04:54:38.284772+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.862480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17296,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.287887+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:38.287887+0000","last_peered":"2024-09-16T04:54:38.287887+0000","last_clean":"2024-09-16T04:54:38.287887+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:38.287887+0000","last_undegraded":"2024-09-16T04:54:38.287887+0000","last_fullsized":"2024-09-16T04:54:38.287887+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:38.759937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12663,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.285036+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:38.285036+0000","last_peered":"2024-09-16T04:54:38.285036+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:38.285036+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12442,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.367021+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:38.367021+0000","last_peered":"2024-09-16T04:54:38.367021+0000","last_clean":"2024-09-16T04:54:38.367021+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:38.367021+0000","last_undegraded":"2024-09-16T04:54:38.367021+0000","last_fullsized":"2024-09-16T04:54:38.367021+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:44.030027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12656,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285318+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:38.285318+0000","last_peered":"2024-09-16T04:54:38.285318+0000","last_clean":"2024-09-16T04:54:38.285318+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:38.285318+0000","last_undegraded":"2024-09-16T04:54:38.285318+0000","last_fullsized":"2024-09-16T04:54:38.285318+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12955,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.286004+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:38.286004+0000","last_peered":"2024-09-16T04:54:38.286004+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:38.286004+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14961,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285505+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:38.285505+0000","last_peered":"2024-09-16T04:54:38.285505+0000","last_clean":"2024-09-16T04:54:38.285505+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:38.285505+0000","last_undegraded":"2024-09-16T04:54:38.285505+0000","last_fullsized":"2024-09-16T04:54:38.285505+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12431,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285737+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:38.285737+0000","last_peered":"2024-09-16T04:54:38.285737+0000","last_clean":"2024-09-16T04:54:38.285737+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:38.285737+0000","last_undegraded":"2024-09-16T04:54:38.285737+0000","last_fullsized":"2024-09-16T04:54:38.285737+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.023729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21940,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731356,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640484,"statfs":{"total":96636764160,"available":95887855616,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640732,"kb_used_data":964,"kb_used_omap":4,"kb_used_meta":639739,"kb_avail":93731108,"statfs":{"total":96636764160,"available":95980654592,"internally_reserved":0,"allocated":987136,"data_stored":701534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":655093455},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":528384,"data_stored":528384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:46.587 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:46.896 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:46.897 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:46.909 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-16T04:54:46.827286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22802,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803816,"kb_used_data":2320,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683544,"statfs":{"total":386547056640,"available":383675949056,"internally_reserved":0,"allocated":2375680,"data_stored":1233272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-784,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.549109"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17194,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284666+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:38.284666+0000","last_peered":"2024-09-16T04:54:38.284666+0000","last_clean":"2024-09-16T04:54:38.284666+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:38.284666+0000","last_undegraded":"2024-09-16T04:54:38.284666+0000","last_fullsized":"2024-09-16T04:54:38.284666+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12800,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.286542+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:38.286542+0000","last_peered":"2024-09-16T04:54:38.286542+0000","last_clean":"2024-09-16T04:54:38.286542+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:38.286542+0000","last_undegraded":"2024-09-16T04:54:38.286542+0000","last_fullsized":"2024-09-16T04:54:38.286542+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.023708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":15456,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:45.016120+0000","last_change":"2024-09-16T04:54:36.736779+0000","last_active":"2024-09-16T04:54:45.016120+0000","last_peered":"2024-09-16T04:54:45.016120+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:45.016120+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1331,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-897}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17255,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285241+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:38.285241+0000","last_peered":"2024-09-16T04:54:38.285241+0000","last_clean":"2024-09-16T04:54:38.285241+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:38.285241+0000","last_undegraded":"2024-09-16T04:54:38.285241+0000","last_fullsized":"2024-09-16T04:54:38.285241+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:40.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12914,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.285601+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:38.285601+0000","last_peered":"2024-09-16T04:54:38.285601+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:38.285601+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16830,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284848+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:38.284848+0000","last_peered":"2024-09-16T04:54:38.284848+0000","last_clean":"2024-09-16T04:54:38.284848+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:38.284848+0000","last_undegraded":"2024-09-16T04:54:38.284848+0000","last_fullsized":"2024-09-16T04:54:38.284848+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17177,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.289049+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:38.289049+0000","last_peered":"2024-09-16T04:54:38.289049+0000","last_clean":"2024-09-16T04:54:38.289049+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:38.289049+0000","last_undegraded":"2024-09-16T04:54:38.289049+0000","last_fullsized":"2024-09-16T04:54:38.289049+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.205935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17388,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285673+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:38.285673+0000","last_peered":"2024-09-16T04:54:38.285673+0000","last_clean":"2024-09-16T04:54:38.285673+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:38.285673+0000","last_undegraded":"2024-09-16T04:54:38.285673+0000","last_fullsized":"2024-09-16T04:54:38.285673+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:45.038120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17552,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284772+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:38.284772+0000","last_peered":"2024-09-16T04:54:38.284772+0000","last_clean":"2024-09-16T04:54:38.284772+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:38.284772+0000","last_undegraded":"2024-09-16T04:54:38.284772+0000","last_fullsized":"2024-09-16T04:54:38.284772+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.862480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17296,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.287887+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:38.287887+0000","last_peered":"2024-09-16T04:54:38.287887+0000","last_clean":"2024-09-16T04:54:38.287887+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:38.287887+0000","last_undegraded":"2024-09-16T04:54:38.287887+0000","last_fullsized":"2024-09-16T04:54:38.287887+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:38.759937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12663,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.285036+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:38.285036+0000","last_peered":"2024-09-16T04:54:38.285036+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:38.285036+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12442,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.367021+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:38.367021+0000","last_peered":"2024-09-16T04:54:38.367021+0000","last_clean":"2024-09-16T04:54:38.367021+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:38.367021+0000","last_undegraded":"2024-09-16T04:54:38.367021+0000","last_fullsized":"2024-09-16T04:54:38.367021+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:44.030027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12656,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285318+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:38.285318+0000","last_peered":"2024-09-16T04:54:38.285318+0000","last_clean":"2024-09-16T04:54:38.285318+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:38.285318+0000","last_undegraded":"2024-09-16T04:54:38.285318+0000","last_fullsized":"2024-09-16T04:54:38.285318+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12955,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.286004+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:38.286004+0000","last_peered":"2024-09-16T04:54:38.286004+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:38.286004+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14961,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285505+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:38.285505+0000","last_peered":"2024-09-16T04:54:38.285505+0000","last_clean":"2024-09-16T04:54:38.285505+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:38.285505+0000","last_undegraded":"2024-09-16T04:54:38.285505+0000","last_fullsized":"2024-09-16T04:54:38.285505+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12431,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285737+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:38.285737+0000","last_peered":"2024-09-16T04:54:38.285737+0000","last_clean":"2024-09-16T04:54:38.285737+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:38.285737+0000","last_undegraded":"2024-09-16T04:54:38.285737+0000","last_fullsized":"2024-09-16T04:54:38.285737+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.023729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22802,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731356,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640484,"statfs":{"total":96636764160,"available":95887855616,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628244,"kb_used_data":964,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93743596,"statfs":{"total":96636764160,"available":95993442304,"internally_reserved":0,"allocated":987136,"data_stored":701534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":528384,"data_stored":528384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:46.911 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:47.205 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:47.206 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:47.218 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-16T04:54:46.827286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22802,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803816,"kb_used_data":2320,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683544,"statfs":{"total":386547056640,"available":383675949056,"internally_reserved":0,"allocated":2375680,"data_stored":1233272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-784,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.549109"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17194,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284666+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:38.284666+0000","last_peered":"2024-09-16T04:54:38.284666+0000","last_clean":"2024-09-16T04:54:38.284666+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:38.284666+0000","last_undegraded":"2024-09-16T04:54:38.284666+0000","last_fullsized":"2024-09-16T04:54:38.284666+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12800,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.286542+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:38.286542+0000","last_peered":"2024-09-16T04:54:38.286542+0000","last_clean":"2024-09-16T04:54:38.286542+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:38.286542+0000","last_undegraded":"2024-09-16T04:54:38.286542+0000","last_fullsized":"2024-09-16T04:54:38.286542+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.023708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":15456,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:45.016120+0000","last_change":"2024-09-16T04:54:36.736779+0000","last_active":"2024-09-16T04:54:45.016120+0000","last_peered":"2024-09-16T04:54:45.016120+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:45.016120+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1331,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-897}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17255,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285241+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:38.285241+0000","last_peered":"2024-09-16T04:54:38.285241+0000","last_clean":"2024-09-16T04:54:38.285241+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:38.285241+0000","last_undegraded":"2024-09-16T04:54:38.285241+0000","last_fullsized":"2024-09-16T04:54:38.285241+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:40.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12914,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.285601+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:38.285601+0000","last_peered":"2024-09-16T04:54:38.285601+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:38.285601+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16830,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284848+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:38.284848+0000","last_peered":"2024-09-16T04:54:38.284848+0000","last_clean":"2024-09-16T04:54:38.284848+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:38.284848+0000","last_undegraded":"2024-09-16T04:54:38.284848+0000","last_fullsized":"2024-09-16T04:54:38.284848+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17177,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.289049+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:38.289049+0000","last_peered":"2024-09-16T04:54:38.289049+0000","last_clean":"2024-09-16T04:54:38.289049+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:38.289049+0000","last_undegraded":"2024-09-16T04:54:38.289049+0000","last_fullsized":"2024-09-16T04:54:38.289049+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.205935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17388,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285673+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:38.285673+0000","last_peered":"2024-09-16T04:54:38.285673+0000","last_clean":"2024-09-16T04:54:38.285673+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:38.285673+0000","last_undegraded":"2024-09-16T04:54:38.285673+0000","last_fullsized":"2024-09-16T04:54:38.285673+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:45.038120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17552,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284772+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:38.284772+0000","last_peered":"2024-09-16T04:54:38.284772+0000","last_clean":"2024-09-16T04:54:38.284772+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:38.284772+0000","last_undegraded":"2024-09-16T04:54:38.284772+0000","last_fullsized":"2024-09-16T04:54:38.284772+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.862480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17296,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.287887+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:38.287887+0000","last_peered":"2024-09-16T04:54:38.287887+0000","last_clean":"2024-09-16T04:54:38.287887+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:38.287887+0000","last_undegraded":"2024-09-16T04:54:38.287887+0000","last_fullsized":"2024-09-16T04:54:38.287887+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:38.759937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12663,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.285036+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:38.285036+0000","last_peered":"2024-09-16T04:54:38.285036+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:38.285036+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12442,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.367021+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:38.367021+0000","last_peered":"2024-09-16T04:54:38.367021+0000","last_clean":"2024-09-16T04:54:38.367021+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:38.367021+0000","last_undegraded":"2024-09-16T04:54:38.367021+0000","last_fullsized":"2024-09-16T04:54:38.367021+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:44.030027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12656,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285318+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:38.285318+0000","last_peered":"2024-09-16T04:54:38.285318+0000","last_clean":"2024-09-16T04:54:38.285318+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:38.285318+0000","last_undegraded":"2024-09-16T04:54:38.285318+0000","last_fullsized":"2024-09-16T04:54:38.285318+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12955,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.286004+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:38.286004+0000","last_peered":"2024-09-16T04:54:38.286004+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:38.286004+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14961,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285505+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:38.285505+0000","last_peered":"2024-09-16T04:54:38.285505+0000","last_clean":"2024-09-16T04:54:38.285505+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:38.285505+0000","last_undegraded":"2024-09-16T04:54:38.285505+0000","last_fullsized":"2024-09-16T04:54:38.285505+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12431,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285737+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:38.285737+0000","last_peered":"2024-09-16T04:54:38.285737+0000","last_clean":"2024-09-16T04:54:38.285737+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:38.285737+0000","last_undegraded":"2024-09-16T04:54:38.285737+0000","last_fullsized":"2024-09-16T04:54:38.285737+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.023729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22802,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731356,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640484,"statfs":{"total":96636764160,"available":95887855616,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628244,"kb_used_data":964,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93743596,"statfs":{"total":96636764160,"available":95993442304,"internally_reserved":0,"allocated":987136,"data_stored":701534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":528384,"data_stored":528384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:47.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:54:47.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 15456, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:45.016120+0000', 'last_change': '2024-09-16T04:54:36.736779+0000', 'last_active': '2024-09-16T04:54:45.016120+0000', 'last_peered': '2024-09-16T04:54:45.016120+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:54:45.016120+0000', 'last_undegraded': '2024-09-16T04:41:27.554557+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 898, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1331, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -897}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:47.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:54:47.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12914, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:38.285601+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:54:38.285601+0000', 'last_peered': '2024-09-16T04:54:38.285601+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:54:38.285601+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:47.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:54:47.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12663, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:38.285036+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:54:38.285036+0000', 'last_peered': '2024-09-16T04:54:38.285036+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:54:38.285036+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:47.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:54:47.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12955, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:38.286004+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:54:38.286004+0000', 'last_peered': '2024-09-16T04:54:38.286004+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:54:38.286004+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:47.220 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:54:47.633 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:47.633 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7506/112 objects degraded (6701.786%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":899,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2871107584,"bytes_avail":383675949056,"bytes_total":386547056640,"degraded_objects":7506,"degraded_total":112,"degraded_ratio":67.017857142857139},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:54:47.647 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T04:54:47.647 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:47.963 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:47.963 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:47.976 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-16T04:54:46.827286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22802,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803816,"kb_used_data":2320,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683544,"statfs":{"total":386547056640,"available":383675949056,"internally_reserved":0,"allocated":2375680,"data_stored":1233272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-784,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.549109"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17194,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284666+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:38.284666+0000","last_peered":"2024-09-16T04:54:38.284666+0000","last_clean":"2024-09-16T04:54:38.284666+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:38.284666+0000","last_undegraded":"2024-09-16T04:54:38.284666+0000","last_fullsized":"2024-09-16T04:54:38.284666+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12800,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.286542+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:38.286542+0000","last_peered":"2024-09-16T04:54:38.286542+0000","last_clean":"2024-09-16T04:54:38.286542+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:38.286542+0000","last_undegraded":"2024-09-16T04:54:38.286542+0000","last_fullsized":"2024-09-16T04:54:38.286542+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.023708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":15456,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:45.016120+0000","last_change":"2024-09-16T04:54:36.736779+0000","last_active":"2024-09-16T04:54:45.016120+0000","last_peered":"2024-09-16T04:54:45.016120+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:45.016120+0000","last_undegraded":"2024-09-16T04:41:27.554557+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1331,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-897}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17255,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285241+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:38.285241+0000","last_peered":"2024-09-16T04:54:38.285241+0000","last_clean":"2024-09-16T04:54:38.285241+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:38.285241+0000","last_undegraded":"2024-09-16T04:54:38.285241+0000","last_fullsized":"2024-09-16T04:54:38.285241+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:40.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12914,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.285601+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:38.285601+0000","last_peered":"2024-09-16T04:54:38.285601+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:38.285601+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16830,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284848+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:38.284848+0000","last_peered":"2024-09-16T04:54:38.284848+0000","last_clean":"2024-09-16T04:54:38.284848+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:38.284848+0000","last_undegraded":"2024-09-16T04:54:38.284848+0000","last_fullsized":"2024-09-16T04:54:38.284848+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17177,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.289049+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:38.289049+0000","last_peered":"2024-09-16T04:54:38.289049+0000","last_clean":"2024-09-16T04:54:38.289049+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:38.289049+0000","last_undegraded":"2024-09-16T04:54:38.289049+0000","last_fullsized":"2024-09-16T04:54:38.289049+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.205935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17388,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285673+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:38.285673+0000","last_peered":"2024-09-16T04:54:38.285673+0000","last_clean":"2024-09-16T04:54:38.285673+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:38.285673+0000","last_undegraded":"2024-09-16T04:54:38.285673+0000","last_fullsized":"2024-09-16T04:54:38.285673+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:45.038120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17552,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284772+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:38.284772+0000","last_peered":"2024-09-16T04:54:38.284772+0000","last_clean":"2024-09-16T04:54:38.284772+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:38.284772+0000","last_undegraded":"2024-09-16T04:54:38.284772+0000","last_fullsized":"2024-09-16T04:54:38.284772+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.862480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17296,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.287887+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:38.287887+0000","last_peered":"2024-09-16T04:54:38.287887+0000","last_clean":"2024-09-16T04:54:38.287887+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:38.287887+0000","last_undegraded":"2024-09-16T04:54:38.287887+0000","last_fullsized":"2024-09-16T04:54:38.287887+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:38.759937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12663,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.285036+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:38.285036+0000","last_peered":"2024-09-16T04:54:38.285036+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:38.285036+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12442,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.367021+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:38.367021+0000","last_peered":"2024-09-16T04:54:38.367021+0000","last_clean":"2024-09-16T04:54:38.367021+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:38.367021+0000","last_undegraded":"2024-09-16T04:54:38.367021+0000","last_fullsized":"2024-09-16T04:54:38.367021+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:44.030027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12656,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285318+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:38.285318+0000","last_peered":"2024-09-16T04:54:38.285318+0000","last_clean":"2024-09-16T04:54:38.285318+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:38.285318+0000","last_undegraded":"2024-09-16T04:54:38.285318+0000","last_fullsized":"2024-09-16T04:54:38.285318+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12955,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.286004+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:38.286004+0000","last_peered":"2024-09-16T04:54:38.286004+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:38.286004+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14961,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285505+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:38.285505+0000","last_peered":"2024-09-16T04:54:38.285505+0000","last_clean":"2024-09-16T04:54:38.285505+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:38.285505+0000","last_undegraded":"2024-09-16T04:54:38.285505+0000","last_fullsized":"2024-09-16T04:54:38.285505+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12431,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.285737+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:38.285737+0000","last_peered":"2024-09-16T04:54:38.285737+0000","last_clean":"2024-09-16T04:54:38.285737+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:38.285737+0000","last_undegraded":"2024-09-16T04:54:38.285737+0000","last_fullsized":"2024-09-16T04:54:38.285737+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.023729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22802,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731356,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640484,"statfs":{"total":96636764160,"available":95887855616,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628244,"kb_used_data":964,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93743596,"statfs":{"total":96636764160,"available":95993442304,"internally_reserved":0,"allocated":987136,"data_stored":701534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":528384,"data_stored":528384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:50.978 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:51.280 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:51.280 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:51.293 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-16T04:54:50.827916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23700,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2812140,"kb_used_data":2328,"kb_used_omap":14,"kb_used_meta":2809713,"kb_avail":374675220,"statfs":{"total":386547056640,"available":383667425280,"internally_reserved":0,"allocated":2383872,"data_stored":1236451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2877146673},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1760,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001254"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17194,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284666+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:38.284666+0000","last_peered":"2024-09-16T04:54:38.284666+0000","last_clean":"2024-09-16T04:54:38.284666+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:38.284666+0000","last_undegraded":"2024-09-16T04:54:38.284666+0000","last_fullsized":"2024-09-16T04:54:38.284666+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12802,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314368+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:49.314368+0000","last_peered":"2024-09-16T04:54:49.314368+0000","last_clean":"2024-09-16T04:54:49.314368+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:49.314368+0000","last_undegraded":"2024-09-16T04:54:49.314368+0000","last_fullsized":"2024-09-16T04:54:49.314368+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17254,"reported_epoch":55,"state":"peering","last_fresh":"2024-09-16T04:54:49.107206+0000","last_change":"2024-09-16T04:54:49.320702+0000","last_active":"2024-09-16T04:54:49.320702+0000","last_peered":"2024-09-16T04:54:49.107206+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:49.107206+0000","last_undegraded":"2024-09-16T04:54:49.107206+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17257,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314190+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:49.314190+0000","last_peered":"2024-09-16T04:54:49.314190+0000","last_clean":"2024-09-16T04:54:49.314190+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:49.314190+0000","last_undegraded":"2024-09-16T04:54:49.314190+0000","last_fullsized":"2024-09-16T04:54:49.314190+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12914,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.285601+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:38.285601+0000","last_peered":"2024-09-16T04:54:38.285601+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:38.285601+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16830,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284848+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:38.284848+0000","last_peered":"2024-09-16T04:54:38.284848+0000","last_clean":"2024-09-16T04:54:38.284848+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:38.284848+0000","last_undegraded":"2024-09-16T04:54:38.284848+0000","last_fullsized":"2024-09-16T04:54:38.284848+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17177,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.289049+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:38.289049+0000","last_peered":"2024-09-16T04:54:38.289049+0000","last_clean":"2024-09-16T04:54:38.289049+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:38.289049+0000","last_undegraded":"2024-09-16T04:54:38.289049+0000","last_fullsized":"2024-09-16T04:54:38.289049+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.205935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17390,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314245+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:49.314245+0000","last_peered":"2024-09-16T04:54:49.314245+0000","last_clean":"2024-09-16T04:54:49.314245+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:49.314245+0000","last_undegraded":"2024-09-16T04:54:49.314245+0000","last_fullsized":"2024-09-16T04:54:49.314245+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17552,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284772+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:38.284772+0000","last_peered":"2024-09-16T04:54:38.284772+0000","last_clean":"2024-09-16T04:54:38.284772+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:38.284772+0000","last_undegraded":"2024-09-16T04:54:38.284772+0000","last_fullsized":"2024-09-16T04:54:38.284772+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.862480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17298,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.321218+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:49.321218+0000","last_peered":"2024-09-16T04:54:49.321218+0000","last_clean":"2024-09-16T04:54:49.321218+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:49.321218+0000","last_undegraded":"2024-09-16T04:54:49.321218+0000","last_fullsized":"2024-09-16T04:54:49.321218+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.679754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12663,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.285036+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:38.285036+0000","last_peered":"2024-09-16T04:54:38.285036+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:38.285036+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12444,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314243+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:49.314243+0000","last_peered":"2024-09-16T04:54:49.314243+0000","last_clean":"2024-09-16T04:54:49.314243+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:49.314243+0000","last_undegraded":"2024-09-16T04:54:49.314243+0000","last_fullsized":"2024-09-16T04:54:49.314243+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:49.958976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314487+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:49.314487+0000","last_peered":"2024-09-16T04:54:49.314487+0000","last_clean":"2024-09-16T04:54:49.314487+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:49.314487+0000","last_undegraded":"2024-09-16T04:54:49.314487+0000","last_fullsized":"2024-09-16T04:54:49.314487+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:49.958964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12955,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.286004+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:38.286004+0000","last_peered":"2024-09-16T04:54:38.286004+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:38.286004+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14963,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314290+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:49.314290+0000","last_peered":"2024-09-16T04:54:49.314290+0000","last_clean":"2024-09-16T04:54:49.314290+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:49.314290+0000","last_undegraded":"2024-09-16T04:54:49.314290+0000","last_fullsized":"2024-09-16T04:54:49.314290+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:49.958950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12433,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314388+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:49.314388+0000","last_peered":"2024-09-16T04:54:49.314388+0000","last_clean":"2024-09-16T04:54:49.314388+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:49.314388+0000","last_undegraded":"2024-09-16T04:54:49.314388+0000","last_fullsized":"2024-09-16T04:54:49.314388+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23700,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731360,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640480,"statfs":{"total":96636764160,"available":95887851520,"internally_reserved":0,"allocated":471040,"data_stored":180425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636564,"kb_used_data":964,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93735276,"statfs":{"total":96636764160,"available":95984922624,"internally_reserved":0,"allocated":987136,"data_stored":701534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":650833615},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:51.295 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:51.633 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:51.634 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:51.646 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-16T04:54:50.827916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23700,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2812140,"kb_used_data":2328,"kb_used_omap":14,"kb_used_meta":2809713,"kb_avail":374675220,"statfs":{"total":386547056640,"available":383667425280,"internally_reserved":0,"allocated":2383872,"data_stored":1236451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2877146673},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1760,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001254"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17194,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284666+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:38.284666+0000","last_peered":"2024-09-16T04:54:38.284666+0000","last_clean":"2024-09-16T04:54:38.284666+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:38.284666+0000","last_undegraded":"2024-09-16T04:54:38.284666+0000","last_fullsized":"2024-09-16T04:54:38.284666+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12802,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314368+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:49.314368+0000","last_peered":"2024-09-16T04:54:49.314368+0000","last_clean":"2024-09-16T04:54:49.314368+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:49.314368+0000","last_undegraded":"2024-09-16T04:54:49.314368+0000","last_fullsized":"2024-09-16T04:54:49.314368+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17254,"reported_epoch":55,"state":"peering","last_fresh":"2024-09-16T04:54:49.107206+0000","last_change":"2024-09-16T04:54:49.320702+0000","last_active":"2024-09-16T04:54:49.320702+0000","last_peered":"2024-09-16T04:54:49.107206+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:49.107206+0000","last_undegraded":"2024-09-16T04:54:49.107206+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17257,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314190+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:49.314190+0000","last_peered":"2024-09-16T04:54:49.314190+0000","last_clean":"2024-09-16T04:54:49.314190+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:49.314190+0000","last_undegraded":"2024-09-16T04:54:49.314190+0000","last_fullsized":"2024-09-16T04:54:49.314190+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12914,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.285601+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:38.285601+0000","last_peered":"2024-09-16T04:54:38.285601+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:38.285601+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16830,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284848+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:38.284848+0000","last_peered":"2024-09-16T04:54:38.284848+0000","last_clean":"2024-09-16T04:54:38.284848+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:38.284848+0000","last_undegraded":"2024-09-16T04:54:38.284848+0000","last_fullsized":"2024-09-16T04:54:38.284848+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17177,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.289049+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:38.289049+0000","last_peered":"2024-09-16T04:54:38.289049+0000","last_clean":"2024-09-16T04:54:38.289049+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:38.289049+0000","last_undegraded":"2024-09-16T04:54:38.289049+0000","last_fullsized":"2024-09-16T04:54:38.289049+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.205935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17390,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314245+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:49.314245+0000","last_peered":"2024-09-16T04:54:49.314245+0000","last_clean":"2024-09-16T04:54:49.314245+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:49.314245+0000","last_undegraded":"2024-09-16T04:54:49.314245+0000","last_fullsized":"2024-09-16T04:54:49.314245+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17552,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284772+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:38.284772+0000","last_peered":"2024-09-16T04:54:38.284772+0000","last_clean":"2024-09-16T04:54:38.284772+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:38.284772+0000","last_undegraded":"2024-09-16T04:54:38.284772+0000","last_fullsized":"2024-09-16T04:54:38.284772+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.862480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17298,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.321218+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:49.321218+0000","last_peered":"2024-09-16T04:54:49.321218+0000","last_clean":"2024-09-16T04:54:49.321218+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:49.321218+0000","last_undegraded":"2024-09-16T04:54:49.321218+0000","last_fullsized":"2024-09-16T04:54:49.321218+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.679754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12663,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.285036+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:38.285036+0000","last_peered":"2024-09-16T04:54:38.285036+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:38.285036+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12444,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314243+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:49.314243+0000","last_peered":"2024-09-16T04:54:49.314243+0000","last_clean":"2024-09-16T04:54:49.314243+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:49.314243+0000","last_undegraded":"2024-09-16T04:54:49.314243+0000","last_fullsized":"2024-09-16T04:54:49.314243+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:49.958976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314487+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:49.314487+0000","last_peered":"2024-09-16T04:54:49.314487+0000","last_clean":"2024-09-16T04:54:49.314487+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:49.314487+0000","last_undegraded":"2024-09-16T04:54:49.314487+0000","last_fullsized":"2024-09-16T04:54:49.314487+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:49.958964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12955,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.286004+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:38.286004+0000","last_peered":"2024-09-16T04:54:38.286004+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:38.286004+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14963,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314290+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:49.314290+0000","last_peered":"2024-09-16T04:54:49.314290+0000","last_clean":"2024-09-16T04:54:49.314290+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:49.314290+0000","last_undegraded":"2024-09-16T04:54:49.314290+0000","last_fullsized":"2024-09-16T04:54:49.314290+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:49.958950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12433,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314388+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:49.314388+0000","last_peered":"2024-09-16T04:54:49.314388+0000","last_clean":"2024-09-16T04:54:49.314388+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:49.314388+0000","last_undegraded":"2024-09-16T04:54:49.314388+0000","last_fullsized":"2024-09-16T04:54:49.314388+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23700,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731360,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640480,"statfs":{"total":96636764160,"available":95887851520,"internally_reserved":0,"allocated":471040,"data_stored":180425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636564,"kb_used_data":964,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93735276,"statfs":{"total":96636764160,"available":95984922624,"internally_reserved":0,"allocated":987136,"data_stored":701534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":650833615},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:51.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:54:51.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 17254, 'reported_epoch': 55, 'state': 'peering', 'last_fresh': '2024-09-16T04:54:49.107206+0000', 'last_change': '2024-09-16T04:54:49.320702+0000', 'last_active': '2024-09-16T04:54:49.320702+0000', 'last_peered': '2024-09-16T04:54:49.107206+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:54:49.107206+0000', 'last_undegraded': '2024-09-16T04:54:49.107206+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 56, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2229, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 1, 3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:51.647 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:54:51.647 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12914, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:38.285601+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:54:38.285601+0000', 'last_peered': '2024-09-16T04:54:38.285601+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:54:38.285601+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:51.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:54:51.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12663, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:38.285036+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:54:38.285036+0000', 'last_peered': '2024-09-16T04:54:38.285036+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:54:38.285036+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:51.648 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:54:51.648 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 12955, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:38.286004+0000', 'last_change': '2024-09-16T04:41:27.864144+0000', 'last_active': '2024-09-16T04:54:38.286004+0000', 'last_peered': '2024-09-16T04:54:38.286004+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:54:38.286004+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:51.648 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:54:52.039 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:52.039 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6608/112 objects degraded (5900.000%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":904,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"peering","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2879631360,"bytes_avail":383667425280,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":6608,"degraded_total":112,"degraded_ratio":59,"recovering_objects_per_sec":219,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1760,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:54:52.052 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:54:52.052 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:52.347 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:52.348 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:52.360 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-16T04:54:50.827916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23700,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2812140,"kb_used_data":2328,"kb_used_omap":14,"kb_used_meta":2809713,"kb_avail":374675220,"statfs":{"total":386547056640,"available":383667425280,"internally_reserved":0,"allocated":2383872,"data_stored":1236451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2877146673},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1760,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001254"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17194,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284666+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:38.284666+0000","last_peered":"2024-09-16T04:54:38.284666+0000","last_clean":"2024-09-16T04:54:38.284666+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:38.284666+0000","last_undegraded":"2024-09-16T04:54:38.284666+0000","last_fullsized":"2024-09-16T04:54:38.284666+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12802,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314368+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:49.314368+0000","last_peered":"2024-09-16T04:54:49.314368+0000","last_clean":"2024-09-16T04:54:49.314368+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:49.314368+0000","last_undegraded":"2024-09-16T04:54:49.314368+0000","last_fullsized":"2024-09-16T04:54:49.314368+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17254,"reported_epoch":55,"state":"peering","last_fresh":"2024-09-16T04:54:49.107206+0000","last_change":"2024-09-16T04:54:49.320702+0000","last_active":"2024-09-16T04:54:49.320702+0000","last_peered":"2024-09-16T04:54:49.107206+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:49.107206+0000","last_undegraded":"2024-09-16T04:54:49.107206+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17257,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314190+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:49.314190+0000","last_peered":"2024-09-16T04:54:49.314190+0000","last_clean":"2024-09-16T04:54:49.314190+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:49.314190+0000","last_undegraded":"2024-09-16T04:54:49.314190+0000","last_fullsized":"2024-09-16T04:54:49.314190+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12914,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.285601+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:38.285601+0000","last_peered":"2024-09-16T04:54:38.285601+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:38.285601+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16830,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284848+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:38.284848+0000","last_peered":"2024-09-16T04:54:38.284848+0000","last_clean":"2024-09-16T04:54:38.284848+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:38.284848+0000","last_undegraded":"2024-09-16T04:54:38.284848+0000","last_fullsized":"2024-09-16T04:54:38.284848+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17177,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.289049+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:38.289049+0000","last_peered":"2024-09-16T04:54:38.289049+0000","last_clean":"2024-09-16T04:54:38.289049+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:38.289049+0000","last_undegraded":"2024-09-16T04:54:38.289049+0000","last_fullsized":"2024-09-16T04:54:38.289049+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.205935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17390,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314245+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:49.314245+0000","last_peered":"2024-09-16T04:54:49.314245+0000","last_clean":"2024-09-16T04:54:49.314245+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:49.314245+0000","last_undegraded":"2024-09-16T04:54:49.314245+0000","last_fullsized":"2024-09-16T04:54:49.314245+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17552,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:54:38.284772+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:38.284772+0000","last_peered":"2024-09-16T04:54:38.284772+0000","last_clean":"2024-09-16T04:54:38.284772+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:38.284772+0000","last_undegraded":"2024-09-16T04:54:38.284772+0000","last_fullsized":"2024-09-16T04:54:38.284772+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:42.862480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17298,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.321218+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:49.321218+0000","last_peered":"2024-09-16T04:54:49.321218+0000","last_clean":"2024-09-16T04:54:49.321218+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:49.321218+0000","last_undegraded":"2024-09-16T04:54:49.321218+0000","last_fullsized":"2024-09-16T04:54:49.321218+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.679754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12663,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.285036+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:38.285036+0000","last_peered":"2024-09-16T04:54:38.285036+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:38.285036+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12444,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314243+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:49.314243+0000","last_peered":"2024-09-16T04:54:49.314243+0000","last_clean":"2024-09-16T04:54:49.314243+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:49.314243+0000","last_undegraded":"2024-09-16T04:54:49.314243+0000","last_fullsized":"2024-09-16T04:54:49.314243+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:49.958976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12658,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314487+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:49.314487+0000","last_peered":"2024-09-16T04:54:49.314487+0000","last_clean":"2024-09-16T04:54:49.314487+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:49.314487+0000","last_undegraded":"2024-09-16T04:54:49.314487+0000","last_fullsized":"2024-09-16T04:54:49.314487+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:49.958964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12955,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:38.286004+0000","last_change":"2024-09-16T04:41:27.864144+0000","last_active":"2024-09-16T04:54:38.286004+0000","last_peered":"2024-09-16T04:54:38.286004+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:38.286004+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14963,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314290+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:49.314290+0000","last_peered":"2024-09-16T04:54:49.314290+0000","last_clean":"2024-09-16T04:54:49.314290+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:49.314290+0000","last_undegraded":"2024-09-16T04:54:49.314290+0000","last_fullsized":"2024-09-16T04:54:49.314290+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:49.958950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12433,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.314388+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:49.314388+0000","last_peered":"2024-09-16T04:54:49.314388+0000","last_clean":"2024-09-16T04:54:49.314388+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:49.314388+0000","last_undegraded":"2024-09-16T04:54:49.314388+0000","last_fullsized":"2024-09-16T04:54:49.314388+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23700,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722140,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649700,"statfs":{"total":96636764160,"available":95897292800,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731360,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640480,"statfs":{"total":96636764160,"available":95887851520,"internally_reserved":0,"allocated":471040,"data_stored":180425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636564,"kb_used_data":964,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93735276,"statfs":{"total":96636764160,"available":95984922624,"internally_reserved":0,"allocated":987136,"data_stored":701534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":650833615},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722076,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649764,"statfs":{"total":96636764160,"available":95897358336,"internally_reserved":0,"allocated":462848,"data_stored":177246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:55.362 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:55.661 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:55.662 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:55.674 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-16T04:54:54.828502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24122,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803860,"kb_used_data":2368,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683500,"statfs":{"total":386547056640,"available":383675904000,"internally_reserved":0,"allocated":2424832,"data_stored":1247270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1320,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001216"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17198,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.317076+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:50.317076+0000","last_peered":"2024-09-16T04:54:50.317076+0000","last_clean":"2024-09-16T04:54:50.317076+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:50.317076+0000","last_undegraded":"2024-09-16T04:54:50.317076+0000","last_fullsized":"2024-09-16T04:54:50.317076+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12804,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316813+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:50.316813+0000","last_peered":"2024-09-16T04:54:50.316813+0000","last_clean":"2024-09-16T04:54:50.316813+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:50.316813+0000","last_undegraded":"2024-09-16T04:54:50.316813+0000","last_fullsized":"2024-09-16T04:54:50.316813+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17254,"reported_epoch":55,"state":"peering","last_fresh":"2024-09-16T04:54:49.107206+0000","last_change":"2024-09-16T04:54:49.320702+0000","last_active":"2024-09-16T04:54:49.320702+0000","last_peered":"2024-09-16T04:54:49.107206+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:49.107206+0000","last_undegraded":"2024-09-16T04:54:49.107206+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17259,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316517+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:50.316517+0000","last_peered":"2024-09-16T04:54:50.316517+0000","last_clean":"2024-09-16T04:54:50.316517+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:50.316517+0000","last_undegraded":"2024-09-16T04:54:50.316517+0000","last_fullsized":"2024-09-16T04:54:50.316517+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12918,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:50.317322+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:50.317322+0000","last_peered":"2024-09-16T04:54:50.317322+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:50.317322+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316928+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:50.316928+0000","last_peered":"2024-09-16T04:54:50.316928+0000","last_clean":"2024-09-16T04:54:50.316928+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:50.316928+0000","last_undegraded":"2024-09-16T04:54:50.316928+0000","last_fullsized":"2024-09-16T04:54:50.316928+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17181,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316261+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:50.316261+0000","last_peered":"2024-09-16T04:54:50.316261+0000","last_clean":"2024-09-16T04:54:50.316261+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:50.316261+0000","last_undegraded":"2024-09-16T04:54:50.316261+0000","last_fullsized":"2024-09-16T04:54:50.316261+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:54.131963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17392,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316890+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:50.316890+0000","last_peered":"2024-09-16T04:54:50.316890+0000","last_clean":"2024-09-16T04:54:50.316890+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:50.316890+0000","last_undegraded":"2024-09-16T04:54:50.316890+0000","last_fullsized":"2024-09-16T04:54:50.316890+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17556,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316836+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:50.316836+0000","last_peered":"2024-09-16T04:54:50.316836+0000","last_clean":"2024-09-16T04:54:50.316836+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:50.316836+0000","last_undegraded":"2024-09-16T04:54:50.316836+0000","last_fullsized":"2024-09-16T04:54:50.316836+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:52.937279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17298,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.321218+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:49.321218+0000","last_peered":"2024-09-16T04:54:49.321218+0000","last_clean":"2024-09-16T04:54:49.321218+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:49.321218+0000","last_undegraded":"2024-09-16T04:54:49.321218+0000","last_fullsized":"2024-09-16T04:54:49.321218+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.679754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12667,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:50.317122+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:50.317122+0000","last_peered":"2024-09-16T04:54:50.317122+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:50.317122+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12446,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316907+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:50.316907+0000","last_peered":"2024-09-16T04:54:50.316907+0000","last_clean":"2024-09-16T04:54:50.316907+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:50.316907+0000","last_undegraded":"2024-09-16T04:54:50.316907+0000","last_fullsized":"2024-09-16T04:54:50.316907+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12660,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316620+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:50.316620+0000","last_peered":"2024-09-16T04:54:50.316620+0000","last_clean":"2024-09-16T04:54:50.316620+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:50.316620+0000","last_undegraded":"2024-09-16T04:54:50.316620+0000","last_fullsized":"2024-09-16T04:54:50.316620+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":13804,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:51.923930+0000","last_change":"2024-09-16T04:54:49.108069+0000","last_active":"2024-09-16T04:54:51.923930+0000","last_peered":"2024-09-16T04:54:51.923930+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:51.923930+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1776}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14965,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316789+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:50.316789+0000","last_peered":"2024-09-16T04:54:50.316789+0000","last_clean":"2024-09-16T04:54:50.316789+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:50.316789+0000","last_undegraded":"2024-09-16T04:54:50.316789+0000","last_fullsized":"2024-09-16T04:54:50.316789+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12435,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316983+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:50.316983+0000","last_peered":"2024-09-16T04:54:50.316983+0000","last_clean":"2024-09-16T04:54:50.316983+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:50.316983+0000","last_undegraded":"2024-09-16T04:54:50.316983+0000","last_fullsized":"2024-09-16T04:54:50.316983+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24122,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722156,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649684,"statfs":{"total":96636764160,"available":95897276416,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731360,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640480,"statfs":{"total":96636764160,"available":95887851520,"internally_reserved":0,"allocated":471040,"data_stored":180425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628252,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93743588,"statfs":{"total":96636764160,"available":95993434112,"internally_reserved":0,"allocated":995328,"data_stored":699679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:55.675 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:55.987 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:55.988 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:56.000 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-16T04:54:54.828502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24122,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803860,"kb_used_data":2368,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683500,"statfs":{"total":386547056640,"available":383675904000,"internally_reserved":0,"allocated":2424832,"data_stored":1247270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1320,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001216"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17198,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.317076+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:50.317076+0000","last_peered":"2024-09-16T04:54:50.317076+0000","last_clean":"2024-09-16T04:54:50.317076+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:50.317076+0000","last_undegraded":"2024-09-16T04:54:50.317076+0000","last_fullsized":"2024-09-16T04:54:50.317076+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12804,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316813+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:50.316813+0000","last_peered":"2024-09-16T04:54:50.316813+0000","last_clean":"2024-09-16T04:54:50.316813+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:50.316813+0000","last_undegraded":"2024-09-16T04:54:50.316813+0000","last_fullsized":"2024-09-16T04:54:50.316813+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17254,"reported_epoch":55,"state":"peering","last_fresh":"2024-09-16T04:54:49.107206+0000","last_change":"2024-09-16T04:54:49.320702+0000","last_active":"2024-09-16T04:54:49.320702+0000","last_peered":"2024-09-16T04:54:49.107206+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:49.107206+0000","last_undegraded":"2024-09-16T04:54:49.107206+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17259,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316517+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:50.316517+0000","last_peered":"2024-09-16T04:54:50.316517+0000","last_clean":"2024-09-16T04:54:50.316517+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:50.316517+0000","last_undegraded":"2024-09-16T04:54:50.316517+0000","last_fullsized":"2024-09-16T04:54:50.316517+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12918,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:50.317322+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:50.317322+0000","last_peered":"2024-09-16T04:54:50.317322+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:50.317322+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316928+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:50.316928+0000","last_peered":"2024-09-16T04:54:50.316928+0000","last_clean":"2024-09-16T04:54:50.316928+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:50.316928+0000","last_undegraded":"2024-09-16T04:54:50.316928+0000","last_fullsized":"2024-09-16T04:54:50.316928+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17181,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316261+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:50.316261+0000","last_peered":"2024-09-16T04:54:50.316261+0000","last_clean":"2024-09-16T04:54:50.316261+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:50.316261+0000","last_undegraded":"2024-09-16T04:54:50.316261+0000","last_fullsized":"2024-09-16T04:54:50.316261+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:54.131963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17392,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316890+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:50.316890+0000","last_peered":"2024-09-16T04:54:50.316890+0000","last_clean":"2024-09-16T04:54:50.316890+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:50.316890+0000","last_undegraded":"2024-09-16T04:54:50.316890+0000","last_fullsized":"2024-09-16T04:54:50.316890+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17556,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316836+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:50.316836+0000","last_peered":"2024-09-16T04:54:50.316836+0000","last_clean":"2024-09-16T04:54:50.316836+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:50.316836+0000","last_undegraded":"2024-09-16T04:54:50.316836+0000","last_fullsized":"2024-09-16T04:54:50.316836+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:52.937279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17298,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.321218+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:49.321218+0000","last_peered":"2024-09-16T04:54:49.321218+0000","last_clean":"2024-09-16T04:54:49.321218+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:49.321218+0000","last_undegraded":"2024-09-16T04:54:49.321218+0000","last_fullsized":"2024-09-16T04:54:49.321218+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.679754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12667,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:50.317122+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:50.317122+0000","last_peered":"2024-09-16T04:54:50.317122+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:50.317122+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12446,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316907+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:50.316907+0000","last_peered":"2024-09-16T04:54:50.316907+0000","last_clean":"2024-09-16T04:54:50.316907+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:50.316907+0000","last_undegraded":"2024-09-16T04:54:50.316907+0000","last_fullsized":"2024-09-16T04:54:50.316907+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12660,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316620+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:50.316620+0000","last_peered":"2024-09-16T04:54:50.316620+0000","last_clean":"2024-09-16T04:54:50.316620+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:50.316620+0000","last_undegraded":"2024-09-16T04:54:50.316620+0000","last_fullsized":"2024-09-16T04:54:50.316620+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":13804,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:51.923930+0000","last_change":"2024-09-16T04:54:49.108069+0000","last_active":"2024-09-16T04:54:51.923930+0000","last_peered":"2024-09-16T04:54:51.923930+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:51.923930+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1776}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14965,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316789+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:50.316789+0000","last_peered":"2024-09-16T04:54:50.316789+0000","last_clean":"2024-09-16T04:54:50.316789+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:50.316789+0000","last_undegraded":"2024-09-16T04:54:50.316789+0000","last_fullsized":"2024-09-16T04:54:50.316789+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12435,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316983+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:50.316983+0000","last_peered":"2024-09-16T04:54:50.316983+0000","last_clean":"2024-09-16T04:54:50.316983+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:50.316983+0000","last_undegraded":"2024-09-16T04:54:50.316983+0000","last_fullsized":"2024-09-16T04:54:50.316983+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24122,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722156,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649684,"statfs":{"total":96636764160,"available":95897276416,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731360,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640480,"statfs":{"total":96636764160,"available":95887851520,"internally_reserved":0,"allocated":471040,"data_stored":180425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628252,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93743588,"statfs":{"total":96636764160,"available":95993434112,"internally_reserved":0,"allocated":995328,"data_stored":699679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:56.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-16T04:54:56.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12670", 'reported_seq': 17254, 'reported_epoch': 55, 'state': 'peering', 'last_fresh': '2024-09-16T04:54:49.107206+0000', 'last_change': '2024-09-16T04:54:49.320702+0000', 'last_active': '2024-09-16T04:54:49.320702+0000', 'last_peered': '2024-09-16T04:54:49.107206+0000', 'last_clean': '2024-09-16T04:40:23.119617+0000', 'last_became_active': '2024-09-16T04:41:27.855109+0000', 'last_became_peered': '2024-09-16T04:41:27.855109+0000', 'last_unstale': '2024-09-16T04:54:49.107206+0000', 'last_undegraded': '2024-09-16T04:54:49.107206+0000', 'last_fullsized': '2024-09-16T04:41:27.533169+0000', 'mapping_epoch': 56, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12669, 'num_write_kb': 50671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2229, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 1, 3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T04:54:56.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:54:56.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12918, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:50.317322+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:54:50.317322+0000', 'last_peered': '2024-09-16T04:54:50.317322+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:54:50.317322+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:56.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:54:56.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12667, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:50.317122+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:54:50.317122+0000', 'last_peered': '2024-09-16T04:54:50.317122+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:54:50.317122+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:56.001 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:54:56.001 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 13804, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:51.923930+0000', 'last_change': '2024-09-16T04:54:49.108069+0000', 'last_active': '2024-09-16T04:54:51.923930+0000', 'last_peered': '2024-09-16T04:54:51.923930+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:54:51.923930+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1778, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 460, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1776}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:54:56.002 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:54:56.366 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:56.366 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6186/112 objects degraded (5523.214%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":908,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2871152640,"bytes_avail":383675904000,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":6186,"degraded_total":112,"degraded_ratio":55.232142857142854,"recovering_objects_per_sec":164,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1320,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:54:56.380 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:54:56.380 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:54:56.693 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:54:56.694 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:54:56.705 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-16T04:54:54.828502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24122,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803860,"kb_used_data":2368,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683500,"statfs":{"total":386547056640,"available":383675904000,"internally_reserved":0,"allocated":2424832,"data_stored":1247270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1320,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001216"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17198,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.317076+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:50.317076+0000","last_peered":"2024-09-16T04:54:50.317076+0000","last_clean":"2024-09-16T04:54:50.317076+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:50.317076+0000","last_undegraded":"2024-09-16T04:54:50.317076+0000","last_fullsized":"2024-09-16T04:54:50.317076+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12804,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316813+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:50.316813+0000","last_peered":"2024-09-16T04:54:50.316813+0000","last_clean":"2024-09-16T04:54:50.316813+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:50.316813+0000","last_undegraded":"2024-09-16T04:54:50.316813+0000","last_fullsized":"2024-09-16T04:54:50.316813+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17254,"reported_epoch":55,"state":"peering","last_fresh":"2024-09-16T04:54:49.107206+0000","last_change":"2024-09-16T04:54:49.320702+0000","last_active":"2024-09-16T04:54:49.320702+0000","last_peered":"2024-09-16T04:54:49.107206+0000","last_clean":"2024-09-16T04:40:23.119617+0000","last_became_active":"2024-09-16T04:41:27.855109+0000","last_became_peered":"2024-09-16T04:41:27.855109+0000","last_unstale":"2024-09-16T04:54:49.107206+0000","last_undegraded":"2024-09-16T04:54:49.107206+0000","last_fullsized":"2024-09-16T04:41:27.533169+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17259,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316517+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:50.316517+0000","last_peered":"2024-09-16T04:54:50.316517+0000","last_clean":"2024-09-16T04:54:50.316517+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:50.316517+0000","last_undegraded":"2024-09-16T04:54:50.316517+0000","last_fullsized":"2024-09-16T04:54:50.316517+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12918,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:50.317322+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:50.317322+0000","last_peered":"2024-09-16T04:54:50.317322+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:50.317322+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316928+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:50.316928+0000","last_peered":"2024-09-16T04:54:50.316928+0000","last_clean":"2024-09-16T04:54:50.316928+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:50.316928+0000","last_undegraded":"2024-09-16T04:54:50.316928+0000","last_fullsized":"2024-09-16T04:54:50.316928+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17181,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316261+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:50.316261+0000","last_peered":"2024-09-16T04:54:50.316261+0000","last_clean":"2024-09-16T04:54:50.316261+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:50.316261+0000","last_undegraded":"2024-09-16T04:54:50.316261+0000","last_fullsized":"2024-09-16T04:54:50.316261+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:54.131963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17392,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316890+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:50.316890+0000","last_peered":"2024-09-16T04:54:50.316890+0000","last_clean":"2024-09-16T04:54:50.316890+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:50.316890+0000","last_undegraded":"2024-09-16T04:54:50.316890+0000","last_fullsized":"2024-09-16T04:54:50.316890+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17556,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316836+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:50.316836+0000","last_peered":"2024-09-16T04:54:50.316836+0000","last_clean":"2024-09-16T04:54:50.316836+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:50.316836+0000","last_undegraded":"2024-09-16T04:54:50.316836+0000","last_fullsized":"2024-09-16T04:54:50.316836+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:52.937279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17298,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T04:54:49.321218+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:49.321218+0000","last_peered":"2024-09-16T04:54:49.321218+0000","last_clean":"2024-09-16T04:54:49.321218+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:49.321218+0000","last_undegraded":"2024-09-16T04:54:49.321218+0000","last_fullsized":"2024-09-16T04:54:49.321218+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.679754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12667,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:50.317122+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:50.317122+0000","last_peered":"2024-09-16T04:54:50.317122+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:50.317122+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12446,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316907+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:50.316907+0000","last_peered":"2024-09-16T04:54:50.316907+0000","last_clean":"2024-09-16T04:54:50.316907+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:50.316907+0000","last_undegraded":"2024-09-16T04:54:50.316907+0000","last_fullsized":"2024-09-16T04:54:50.316907+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12660,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316620+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:50.316620+0000","last_peered":"2024-09-16T04:54:50.316620+0000","last_clean":"2024-09-16T04:54:50.316620+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:50.316620+0000","last_undegraded":"2024-09-16T04:54:50.316620+0000","last_fullsized":"2024-09-16T04:54:50.316620+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":13804,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:51.923930+0000","last_change":"2024-09-16T04:54:49.108069+0000","last_active":"2024-09-16T04:54:51.923930+0000","last_peered":"2024-09-16T04:54:51.923930+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:51.923930+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1776}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14965,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316789+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:50.316789+0000","last_peered":"2024-09-16T04:54:50.316789+0000","last_clean":"2024-09-16T04:54:50.316789+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:50.316789+0000","last_undegraded":"2024-09-16T04:54:50.316789+0000","last_fullsized":"2024-09-16T04:54:50.316789+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12435,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316983+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:50.316983+0000","last_peered":"2024-09-16T04:54:50.316983+0000","last_clean":"2024-09-16T04:54:50.316983+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:50.316983+0000","last_undegraded":"2024-09-16T04:54:50.316983+0000","last_fullsized":"2024-09-16T04:54:50.316983+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24122,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722156,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649684,"statfs":{"total":96636764160,"available":95897276416,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731360,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640480,"statfs":{"total":96636764160,"available":95887851520,"internally_reserved":0,"allocated":471040,"data_stored":180425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628252,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93743588,"statfs":{"total":96636764160,"available":95993434112,"internally_reserved":0,"allocated":995328,"data_stored":699679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:54:59.707 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:00.007 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:00.007 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:00.020 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-09-16T04:54:58.829088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24960,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803872,"kb_used_data":2376,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683488,"statfs":{"total":386547056640,"available":383675891712,"internally_reserved":0,"allocated":2433024,"data_stored":1250428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2158,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.516735"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17198,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.317076+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:50.317076+0000","last_peered":"2024-09-16T04:54:50.317076+0000","last_clean":"2024-09-16T04:54:50.317076+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:50.317076+0000","last_undegraded":"2024-09-16T04:54:50.317076+0000","last_fullsized":"2024-09-16T04:54:50.317076+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12804,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316813+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:50.316813+0000","last_peered":"2024-09-16T04:54:50.316813+0000","last_clean":"2024-09-16T04:54:50.316813+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:50.316813+0000","last_undegraded":"2024-09-16T04:54:50.316813+0000","last_fullsized":"2024-09-16T04:54:50.316813+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17264,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.394068+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:54:50.394068+0000","last_peered":"2024-09-16T04:54:50.394068+0000","last_clean":"2024-09-16T04:54:50.394068+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:54:50.394068+0000","last_undegraded":"2024-09-16T04:54:50.394068+0000","last_fullsized":"2024-09-16T04:54:50.394068+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17259,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316517+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:50.316517+0000","last_peered":"2024-09-16T04:54:50.316517+0000","last_clean":"2024-09-16T04:54:50.316517+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:50.316517+0000","last_undegraded":"2024-09-16T04:54:50.316517+0000","last_fullsized":"2024-09-16T04:54:50.316517+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12918,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:50.317322+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:50.317322+0000","last_peered":"2024-09-16T04:54:50.317322+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:50.317322+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316928+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:50.316928+0000","last_peered":"2024-09-16T04:54:50.316928+0000","last_clean":"2024-09-16T04:54:50.316928+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:50.316928+0000","last_undegraded":"2024-09-16T04:54:50.316928+0000","last_fullsized":"2024-09-16T04:54:50.316928+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17181,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316261+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:50.316261+0000","last_peered":"2024-09-16T04:54:50.316261+0000","last_clean":"2024-09-16T04:54:50.316261+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:50.316261+0000","last_undegraded":"2024-09-16T04:54:50.316261+0000","last_fullsized":"2024-09-16T04:54:50.316261+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:54.131963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17392,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316890+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:50.316890+0000","last_peered":"2024-09-16T04:54:50.316890+0000","last_clean":"2024-09-16T04:54:50.316890+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:50.316890+0000","last_undegraded":"2024-09-16T04:54:50.316890+0000","last_fullsized":"2024-09-16T04:54:50.316890+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17556,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316836+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:50.316836+0000","last_peered":"2024-09-16T04:54:50.316836+0000","last_clean":"2024-09-16T04:54:50.316836+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:50.316836+0000","last_undegraded":"2024-09-16T04:54:50.316836+0000","last_fullsized":"2024-09-16T04:54:50.316836+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:52.937279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17301,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.319863+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:50.319863+0000","last_peered":"2024-09-16T04:54:50.319863+0000","last_clean":"2024-09-16T04:54:50.319863+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:50.319863+0000","last_undegraded":"2024-09-16T04:54:50.319863+0000","last_fullsized":"2024-09-16T04:54:50.319863+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:55.750407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12667,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:50.317122+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:50.317122+0000","last_peered":"2024-09-16T04:54:50.317122+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:50.317122+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12446,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316907+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:50.316907+0000","last_peered":"2024-09-16T04:54:50.316907+0000","last_clean":"2024-09-16T04:54:50.316907+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:50.316907+0000","last_undegraded":"2024-09-16T04:54:50.316907+0000","last_fullsized":"2024-09-16T04:54:50.316907+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12660,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316620+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:50.316620+0000","last_peered":"2024-09-16T04:54:50.316620+0000","last_clean":"2024-09-16T04:54:50.316620+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:50.316620+0000","last_undegraded":"2024-09-16T04:54:50.316620+0000","last_fullsized":"2024-09-16T04:54:50.316620+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":15480,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:56.922910+0000","last_change":"2024-09-16T04:54:49.108069+0000","last_active":"2024-09-16T04:54:56.922910+0000","last_peered":"2024-09-16T04:54:56.922910+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:56.922910+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1298,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-938}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14965,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316789+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:50.316789+0000","last_peered":"2024-09-16T04:54:50.316789+0000","last_clean":"2024-09-16T04:54:50.316789+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:50.316789+0000","last_undegraded":"2024-09-16T04:54:50.316789+0000","last_fullsized":"2024-09-16T04:54:50.316789+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12435,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316983+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:50.316983+0000","last_peered":"2024-09-16T04:54:50.316983+0000","last_clean":"2024-09-16T04:54:50.316983+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:50.316983+0000","last_undegraded":"2024-09-16T04:54:50.316983+0000","last_fullsized":"2024-09-16T04:54:50.316983+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24960,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722156,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649684,"statfs":{"total":96636764160,"available":95897276416,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731372,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640468,"statfs":{"total":96636764160,"available":95887839232,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628252,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93743588,"statfs":{"total":96636764160,"available":95993434112,"internally_reserved":0,"allocated":995328,"data_stored":699679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:00.021 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:00.316 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:00.317 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:00.329 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-09-16T04:54:58.829088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24960,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803872,"kb_used_data":2376,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683488,"statfs":{"total":386547056640,"available":383675891712,"internally_reserved":0,"allocated":2433024,"data_stored":1250428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2158,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.516735"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17198,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.317076+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:50.317076+0000","last_peered":"2024-09-16T04:54:50.317076+0000","last_clean":"2024-09-16T04:54:50.317076+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:50.317076+0000","last_undegraded":"2024-09-16T04:54:50.317076+0000","last_fullsized":"2024-09-16T04:54:50.317076+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12804,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316813+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:50.316813+0000","last_peered":"2024-09-16T04:54:50.316813+0000","last_clean":"2024-09-16T04:54:50.316813+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:50.316813+0000","last_undegraded":"2024-09-16T04:54:50.316813+0000","last_fullsized":"2024-09-16T04:54:50.316813+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17264,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.394068+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:54:50.394068+0000","last_peered":"2024-09-16T04:54:50.394068+0000","last_clean":"2024-09-16T04:54:50.394068+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:54:50.394068+0000","last_undegraded":"2024-09-16T04:54:50.394068+0000","last_fullsized":"2024-09-16T04:54:50.394068+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17259,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316517+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:50.316517+0000","last_peered":"2024-09-16T04:54:50.316517+0000","last_clean":"2024-09-16T04:54:50.316517+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:50.316517+0000","last_undegraded":"2024-09-16T04:54:50.316517+0000","last_fullsized":"2024-09-16T04:54:50.316517+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12918,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:50.317322+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:50.317322+0000","last_peered":"2024-09-16T04:54:50.317322+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:50.317322+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316928+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:50.316928+0000","last_peered":"2024-09-16T04:54:50.316928+0000","last_clean":"2024-09-16T04:54:50.316928+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:50.316928+0000","last_undegraded":"2024-09-16T04:54:50.316928+0000","last_fullsized":"2024-09-16T04:54:50.316928+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17181,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316261+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:50.316261+0000","last_peered":"2024-09-16T04:54:50.316261+0000","last_clean":"2024-09-16T04:54:50.316261+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:50.316261+0000","last_undegraded":"2024-09-16T04:54:50.316261+0000","last_fullsized":"2024-09-16T04:54:50.316261+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:54.131963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17392,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316890+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:50.316890+0000","last_peered":"2024-09-16T04:54:50.316890+0000","last_clean":"2024-09-16T04:54:50.316890+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:50.316890+0000","last_undegraded":"2024-09-16T04:54:50.316890+0000","last_fullsized":"2024-09-16T04:54:50.316890+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17556,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316836+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:50.316836+0000","last_peered":"2024-09-16T04:54:50.316836+0000","last_clean":"2024-09-16T04:54:50.316836+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:50.316836+0000","last_undegraded":"2024-09-16T04:54:50.316836+0000","last_fullsized":"2024-09-16T04:54:50.316836+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:52.937279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17301,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.319863+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:50.319863+0000","last_peered":"2024-09-16T04:54:50.319863+0000","last_clean":"2024-09-16T04:54:50.319863+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:50.319863+0000","last_undegraded":"2024-09-16T04:54:50.319863+0000","last_fullsized":"2024-09-16T04:54:50.319863+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:55.750407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12667,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:50.317122+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:50.317122+0000","last_peered":"2024-09-16T04:54:50.317122+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:50.317122+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12446,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316907+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:50.316907+0000","last_peered":"2024-09-16T04:54:50.316907+0000","last_clean":"2024-09-16T04:54:50.316907+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:50.316907+0000","last_undegraded":"2024-09-16T04:54:50.316907+0000","last_fullsized":"2024-09-16T04:54:50.316907+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12660,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316620+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:50.316620+0000","last_peered":"2024-09-16T04:54:50.316620+0000","last_clean":"2024-09-16T04:54:50.316620+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:50.316620+0000","last_undegraded":"2024-09-16T04:54:50.316620+0000","last_fullsized":"2024-09-16T04:54:50.316620+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":15480,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:56.922910+0000","last_change":"2024-09-16T04:54:49.108069+0000","last_active":"2024-09-16T04:54:56.922910+0000","last_peered":"2024-09-16T04:54:56.922910+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:56.922910+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1298,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-938}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14965,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316789+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:50.316789+0000","last_peered":"2024-09-16T04:54:50.316789+0000","last_clean":"2024-09-16T04:54:50.316789+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:50.316789+0000","last_undegraded":"2024-09-16T04:54:50.316789+0000","last_fullsized":"2024-09-16T04:54:50.316789+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12435,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316983+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:50.316983+0000","last_peered":"2024-09-16T04:54:50.316983+0000","last_clean":"2024-09-16T04:54:50.316983+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:50.316983+0000","last_undegraded":"2024-09-16T04:54:50.316983+0000","last_fullsized":"2024-09-16T04:54:50.316983+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24960,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722156,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649684,"statfs":{"total":96636764160,"available":95897276416,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731372,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640468,"statfs":{"total":96636764160,"available":95887839232,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628252,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93743588,"statfs":{"total":96636764160,"available":95993434112,"internally_reserved":0,"allocated":995328,"data_stored":699679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:00.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:55:00.330 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12918, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:50.317322+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:54:50.317322+0000', 'last_peered': '2024-09-16T04:54:50.317322+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:54:50.317322+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:55:00.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:55:00.330 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12667, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:50.317122+0000', 'last_change': '2024-09-16T04:41:27.873131+0000', 'last_active': '2024-09-16T04:54:50.317122+0000', 'last_peered': '2024-09-16T04:54:50.317122+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:54:50.317122+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2152}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:55:00.330 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:55:00.331 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 15480, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:56.922910+0000', 'last_change': '2024-09-16T04:54:49.108069+0000', 'last_active': '2024-09-16T04:54:56.922910+0000', 'last_peered': '2024-09-16T04:54:56.922910+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:54:56.922910+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 940, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1298, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -938}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:55:00.331 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:55:00.747 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:00.747 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5348/112 objects degraded (4775.000%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":913,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2871164928,"bytes_avail":383675891712,"bytes_total":386547056640,"degraded_objects":5348,"degraded_total":112,"degraded_ratio":47.75,"recovering_objects_per_sec":226,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2158,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:55:00.761 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:55:00.762 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:01.065 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:01.066 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:01.081 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-09-16T04:55:00.829480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24960,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2812192,"kb_used_data":2376,"kb_used_omap":14,"kb_used_meta":2809713,"kb_avail":374675168,"statfs":{"total":386547056640,"available":383667372032,"internally_reserved":0,"allocated":2433024,"data_stored":1250428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2877146673},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1260,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.515141"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17198,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.317076+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:54:50.317076+0000","last_peered":"2024-09-16T04:54:50.317076+0000","last_clean":"2024-09-16T04:54:50.317076+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:54:50.317076+0000","last_undegraded":"2024-09-16T04:54:50.317076+0000","last_fullsized":"2024-09-16T04:54:50.317076+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12804,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316813+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:50.316813+0000","last_peered":"2024-09-16T04:54:50.316813+0000","last_clean":"2024-09-16T04:54:50.316813+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:50.316813+0000","last_undegraded":"2024-09-16T04:54:50.316813+0000","last_fullsized":"2024-09-16T04:54:50.316813+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17264,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.394068+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:54:50.394068+0000","last_peered":"2024-09-16T04:54:50.394068+0000","last_clean":"2024-09-16T04:54:50.394068+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:54:50.394068+0000","last_undegraded":"2024-09-16T04:54:50.394068+0000","last_fullsized":"2024-09-16T04:54:50.394068+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17259,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316517+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:50.316517+0000","last_peered":"2024-09-16T04:54:50.316517+0000","last_clean":"2024-09-16T04:54:50.316517+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:50.316517+0000","last_undegraded":"2024-09-16T04:54:50.316517+0000","last_fullsized":"2024-09-16T04:54:50.316517+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12918,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:50.317322+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:54:50.317322+0000","last_peered":"2024-09-16T04:54:50.317322+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:54:50.317322+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316928+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:54:50.316928+0000","last_peered":"2024-09-16T04:54:50.316928+0000","last_clean":"2024-09-16T04:54:50.316928+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:54:50.316928+0000","last_undegraded":"2024-09-16T04:54:50.316928+0000","last_fullsized":"2024-09-16T04:54:50.316928+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17181,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316261+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:50.316261+0000","last_peered":"2024-09-16T04:54:50.316261+0000","last_clean":"2024-09-16T04:54:50.316261+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:50.316261+0000","last_undegraded":"2024-09-16T04:54:50.316261+0000","last_fullsized":"2024-09-16T04:54:50.316261+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:54.131963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17392,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316890+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:50.316890+0000","last_peered":"2024-09-16T04:54:50.316890+0000","last_clean":"2024-09-16T04:54:50.316890+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:50.316890+0000","last_undegraded":"2024-09-16T04:54:50.316890+0000","last_fullsized":"2024-09-16T04:54:50.316890+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17556,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316836+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:54:50.316836+0000","last_peered":"2024-09-16T04:54:50.316836+0000","last_clean":"2024-09-16T04:54:50.316836+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:54:50.316836+0000","last_undegraded":"2024-09-16T04:54:50.316836+0000","last_fullsized":"2024-09-16T04:54:50.316836+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:52.937279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17301,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.319863+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:50.319863+0000","last_peered":"2024-09-16T04:54:50.319863+0000","last_clean":"2024-09-16T04:54:50.319863+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:50.319863+0000","last_undegraded":"2024-09-16T04:54:50.319863+0000","last_fullsized":"2024-09-16T04:54:50.319863+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:55.750407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12667,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:50.317122+0000","last_change":"2024-09-16T04:41:27.873131+0000","last_active":"2024-09-16T04:54:50.317122+0000","last_peered":"2024-09-16T04:54:50.317122+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:54:50.317122+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12446,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316907+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:50.316907+0000","last_peered":"2024-09-16T04:54:50.316907+0000","last_clean":"2024-09-16T04:54:50.316907+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:50.316907+0000","last_undegraded":"2024-09-16T04:54:50.316907+0000","last_fullsized":"2024-09-16T04:54:50.316907+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12660,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316620+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:50.316620+0000","last_peered":"2024-09-16T04:54:50.316620+0000","last_clean":"2024-09-16T04:54:50.316620+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:50.316620+0000","last_undegraded":"2024-09-16T04:54:50.316620+0000","last_fullsized":"2024-09-16T04:54:50.316620+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":15480,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:56.922910+0000","last_change":"2024-09-16T04:54:49.108069+0000","last_active":"2024-09-16T04:54:56.922910+0000","last_peered":"2024-09-16T04:54:56.922910+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:56.922910+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1298,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-938}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14965,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316789+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:50.316789+0000","last_peered":"2024-09-16T04:54:50.316789+0000","last_clean":"2024-09-16T04:54:50.316789+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:50.316789+0000","last_undegraded":"2024-09-16T04:54:50.316789+0000","last_fullsized":"2024-09-16T04:54:50.316789+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12435,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316983+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:50.316983+0000","last_peered":"2024-09-16T04:54:50.316983+0000","last_clean":"2024-09-16T04:54:50.316983+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:50.316983+0000","last_undegraded":"2024-09-16T04:54:50.316983+0000","last_fullsized":"2024-09-16T04:54:50.316983+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24960,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722156,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649684,"statfs":{"total":96636764160,"available":95897276416,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731372,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640468,"statfs":{"total":96636764160,"available":95887839232,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636572,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93735268,"statfs":{"total":96636764160,"available":95984914432,"internally_reserved":0,"allocated":995328,"data_stored":699679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":650833615},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:04.083 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:04.385 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:04.386 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:04.397 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2024-09-16T04:55:02.829773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25056,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2812196,"kb_used_data":2384,"kb_used_omap":14,"kb_used_meta":2809713,"kb_avail":374675164,"statfs":{"total":386547056640,"available":383667367936,"internally_reserved":0,"allocated":2441216,"data_stored":1253570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2877146673},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001270"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17200,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T04:55:01.348053+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:01.348053+0000","last_peered":"2024-09-16T04:55:01.348053+0000","last_clean":"2024-09-16T04:55:01.348053+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:01.348053+0000","last_undegraded":"2024-09-16T04:55:01.348053+0000","last_fullsized":"2024-09-16T04:55:01.348053+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12804,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316813+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:50.316813+0000","last_peered":"2024-09-16T04:54:50.316813+0000","last_clean":"2024-09-16T04:54:50.316813+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:50.316813+0000","last_undegraded":"2024-09-16T04:54:50.316813+0000","last_fullsized":"2024-09-16T04:54:50.316813+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17264,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.394068+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:54:50.394068+0000","last_peered":"2024-09-16T04:54:50.394068+0000","last_clean":"2024-09-16T04:54:50.394068+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:54:50.394068+0000","last_undegraded":"2024-09-16T04:54:50.394068+0000","last_fullsized":"2024-09-16T04:54:50.394068+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17259,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316517+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:50.316517+0000","last_peered":"2024-09-16T04:54:50.316517+0000","last_clean":"2024-09-16T04:54:50.316517+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:50.316517+0000","last_undegraded":"2024-09-16T04:54:50.316517+0000","last_fullsized":"2024-09-16T04:54:50.316517+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12921,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:01.348250+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:55:01.348250+0000","last_peered":"2024-09-16T04:55:01.348250+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:01.348250+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T04:55:01.347500+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:01.347500+0000","last_peered":"2024-09-16T04:55:01.347500+0000","last_clean":"2024-09-16T04:55:01.347500+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:01.347500+0000","last_undegraded":"2024-09-16T04:55:01.347500+0000","last_fullsized":"2024-09-16T04:55:01.347500+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17181,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316261+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:50.316261+0000","last_peered":"2024-09-16T04:54:50.316261+0000","last_clean":"2024-09-16T04:54:50.316261+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:50.316261+0000","last_undegraded":"2024-09-16T04:54:50.316261+0000","last_fullsized":"2024-09-16T04:54:50.316261+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:54.131963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17392,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316890+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:50.316890+0000","last_peered":"2024-09-16T04:54:50.316890+0000","last_clean":"2024-09-16T04:54:50.316890+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:50.316890+0000","last_undegraded":"2024-09-16T04:54:50.316890+0000","last_fullsized":"2024-09-16T04:54:50.316890+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17558,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T04:55:01.347403+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:01.347403+0000","last_peered":"2024-09-16T04:55:01.347403+0000","last_clean":"2024-09-16T04:55:01.347403+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:01.347403+0000","last_undegraded":"2024-09-16T04:55:01.347403+0000","last_fullsized":"2024-09-16T04:55:01.347403+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17301,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.319863+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:50.319863+0000","last_peered":"2024-09-16T04:54:50.319863+0000","last_clean":"2024-09-16T04:54:50.319863+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:50.319863+0000","last_undegraded":"2024-09-16T04:54:50.319863+0000","last_fullsized":"2024-09-16T04:54:50.319863+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:55.750407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12862,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:01.924675+0000","last_change":"2024-09-16T04:55:01.243932+0000","last_active":"2024-09-16T04:55:01.924675+0000","last_peered":"2024-09-16T04:55:01.924675+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:55:01.924675+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12446,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316907+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:50.316907+0000","last_peered":"2024-09-16T04:54:50.316907+0000","last_clean":"2024-09-16T04:54:50.316907+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:50.316907+0000","last_undegraded":"2024-09-16T04:54:50.316907+0000","last_fullsized":"2024-09-16T04:54:50.316907+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12660,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316620+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:50.316620+0000","last_peered":"2024-09-16T04:54:50.316620+0000","last_clean":"2024-09-16T04:54:50.316620+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:50.316620+0000","last_undegraded":"2024-09-16T04:54:50.316620+0000","last_fullsized":"2024-09-16T04:54:50.316620+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":15480,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:56.922910+0000","last_change":"2024-09-16T04:54:49.108069+0000","last_active":"2024-09-16T04:54:56.922910+0000","last_peered":"2024-09-16T04:54:56.922910+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:56.922910+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1298,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-938}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14965,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316789+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:50.316789+0000","last_peered":"2024-09-16T04:54:50.316789+0000","last_clean":"2024-09-16T04:54:50.316789+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:50.316789+0000","last_undegraded":"2024-09-16T04:54:50.316789+0000","last_fullsized":"2024-09-16T04:54:50.316789+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12435,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316983+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:50.316983+0000","last_peered":"2024-09-16T04:54:50.316983+0000","last_clean":"2024-09-16T04:54:50.316983+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:50.316983+0000","last_undegraded":"2024-09-16T04:54:50.316983+0000","last_fullsized":"2024-09-16T04:54:50.316983+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25056,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722160,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649680,"statfs":{"total":96636764160,"available":95897272320,"internally_reserved":0,"allocated":487424,"data_stored":186725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731372,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640468,"statfs":{"total":96636764160,"available":95887839232,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636572,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93735268,"statfs":{"total":96636764160,"available":95984914432,"internally_reserved":0,"allocated":995328,"data_stored":699679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":650833615},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:04.399 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:04.698 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:04.699 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:04.711 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2024-09-16T04:55:02.829773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25056,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2812196,"kb_used_data":2384,"kb_used_omap":14,"kb_used_meta":2809713,"kb_avail":374675164,"statfs":{"total":386547056640,"available":383667367936,"internally_reserved":0,"allocated":2441216,"data_stored":1253570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2877146673},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001270"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17200,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T04:55:01.348053+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:01.348053+0000","last_peered":"2024-09-16T04:55:01.348053+0000","last_clean":"2024-09-16T04:55:01.348053+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:01.348053+0000","last_undegraded":"2024-09-16T04:55:01.348053+0000","last_fullsized":"2024-09-16T04:55:01.348053+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12804,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316813+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:54:50.316813+0000","last_peered":"2024-09-16T04:54:50.316813+0000","last_clean":"2024-09-16T04:54:50.316813+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:54:50.316813+0000","last_undegraded":"2024-09-16T04:54:50.316813+0000","last_fullsized":"2024-09-16T04:54:50.316813+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17264,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.394068+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:54:50.394068+0000","last_peered":"2024-09-16T04:54:50.394068+0000","last_clean":"2024-09-16T04:54:50.394068+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:54:50.394068+0000","last_undegraded":"2024-09-16T04:54:50.394068+0000","last_fullsized":"2024-09-16T04:54:50.394068+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17259,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316517+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:54:50.316517+0000","last_peered":"2024-09-16T04:54:50.316517+0000","last_clean":"2024-09-16T04:54:50.316517+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:54:50.316517+0000","last_undegraded":"2024-09-16T04:54:50.316517+0000","last_fullsized":"2024-09-16T04:54:50.316517+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12921,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:01.348250+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:55:01.348250+0000","last_peered":"2024-09-16T04:55:01.348250+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:01.348250+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T04:55:01.347500+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:01.347500+0000","last_peered":"2024-09-16T04:55:01.347500+0000","last_clean":"2024-09-16T04:55:01.347500+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:01.347500+0000","last_undegraded":"2024-09-16T04:55:01.347500+0000","last_fullsized":"2024-09-16T04:55:01.347500+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17181,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316261+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:50.316261+0000","last_peered":"2024-09-16T04:54:50.316261+0000","last_clean":"2024-09-16T04:54:50.316261+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:50.316261+0000","last_undegraded":"2024-09-16T04:54:50.316261+0000","last_fullsized":"2024-09-16T04:54:50.316261+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:54.131963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17392,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316890+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:54:50.316890+0000","last_peered":"2024-09-16T04:54:50.316890+0000","last_clean":"2024-09-16T04:54:50.316890+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:54:50.316890+0000","last_undegraded":"2024-09-16T04:54:50.316890+0000","last_fullsized":"2024-09-16T04:54:50.316890+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:50.944172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17558,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T04:55:01.347403+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:01.347403+0000","last_peered":"2024-09-16T04:55:01.347403+0000","last_clean":"2024-09-16T04:55:01.347403+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:01.347403+0000","last_undegraded":"2024-09-16T04:55:01.347403+0000","last_fullsized":"2024-09-16T04:55:01.347403+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17301,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.319863+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:54:50.319863+0000","last_peered":"2024-09-16T04:54:50.319863+0000","last_clean":"2024-09-16T04:54:50.319863+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:54:50.319863+0000","last_undegraded":"2024-09-16T04:54:50.319863+0000","last_fullsized":"2024-09-16T04:54:50.319863+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:55.750407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12862,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:01.924675+0000","last_change":"2024-09-16T04:55:01.243932+0000","last_active":"2024-09-16T04:55:01.924675+0000","last_peered":"2024-09-16T04:55:01.924675+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:55:01.924675+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12446,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316907+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:54:50.316907+0000","last_peered":"2024-09-16T04:54:50.316907+0000","last_clean":"2024-09-16T04:54:50.316907+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:54:50.316907+0000","last_undegraded":"2024-09-16T04:54:50.316907+0000","last_fullsized":"2024-09-16T04:54:50.316907+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12660,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316620+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:54:50.316620+0000","last_peered":"2024-09-16T04:54:50.316620+0000","last_clean":"2024-09-16T04:54:50.316620+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:54:50.316620+0000","last_undegraded":"2024-09-16T04:54:50.316620+0000","last_fullsized":"2024-09-16T04:54:50.316620+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":15480,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:54:56.922910+0000","last_change":"2024-09-16T04:54:49.108069+0000","last_active":"2024-09-16T04:54:56.922910+0000","last_peered":"2024-09-16T04:54:56.922910+0000","last_clean":"2024-09-16T04:40:22.711859+0000","last_became_active":"2024-09-16T04:41:27.855591+0000","last_became_peered":"2024-09-16T04:41:27.855591+0000","last_unstale":"2024-09-16T04:54:56.922910+0000","last_undegraded":"2024-09-16T04:41:27.542080+0000","last_fullsized":"2024-09-16T04:41:27.484210+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1298,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-938}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14965,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316789+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:54:50.316789+0000","last_peered":"2024-09-16T04:54:50.316789+0000","last_clean":"2024-09-16T04:54:50.316789+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:54:50.316789+0000","last_undegraded":"2024-09-16T04:54:50.316789+0000","last_fullsized":"2024-09-16T04:54:50.316789+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12435,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316983+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:54:50.316983+0000","last_peered":"2024-09-16T04:54:50.316983+0000","last_clean":"2024-09-16T04:54:50.316983+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:54:50.316983+0000","last_undegraded":"2024-09-16T04:54:50.316983+0000","last_fullsized":"2024-09-16T04:54:50.316983+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:53.886370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25056,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722160,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649680,"statfs":{"total":96636764160,"available":95897272320,"internally_reserved":0,"allocated":487424,"data_stored":186725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731372,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640468,"statfs":{"total":96636764160,"available":95887839232,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636572,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":635579,"kb_avail":93735268,"statfs":{"total":96636764160,"available":95984914432,"internally_reserved":0,"allocated":995328,"data_stored":699679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":650833615},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:04.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:55:04.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12921, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:55:01.348250+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:55:01.348250+0000', 'last_peered': '2024-09-16T04:55:01.348250+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:55:01.348250+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:55:04.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:55:04.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 12862, 'reported_epoch': 58, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:55:01.924675+0000', 'last_change': '2024-09-16T04:55:01.243932+0000', 'last_active': '2024-09-16T04:55:01.924675+0000', 'last_peered': '2024-09-16T04:55:01.924675+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:55:01.924675+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2062, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 133, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:55:04.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-16T04:55:04.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12780", 'reported_seq': 15480, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:54:56.922910+0000', 'last_change': '2024-09-16T04:54:49.108069+0000', 'last_active': '2024-09-16T04:54:56.922910+0000', 'last_peered': '2024-09-16T04:54:56.922910+0000', 'last_clean': '2024-09-16T04:40:22.711859+0000', 'last_became_active': '2024-09-16T04:41:27.855591+0000', 'last_became_peered': '2024-09-16T04:41:27.855591+0000', 'last_unstale': '2024-09-16T04:54:56.922910+0000', 'last_undegraded': '2024-09-16T04:41:27.542080+0000', 'last_fullsized': '2024-09-16T04:41:27.484210+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 940, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12778, 'num_write_kb': 51124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1298, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -938}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:55:04.713 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:55:05.095 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:05.095 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5252/112 objects degraded (4689.286%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":917,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovering+undersized+degraded+remapped","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2879688704,"bytes_avail":383667367936,"bytes_total":386547056640,"degraded_objects":5252,"degraded_total":112,"degraded_ratio":46.892857142857146,"recovering_objects_per_sec":116,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":934,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:55:05.109 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:55:05.109 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:05.428 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:05.428 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:05.440 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-16T04:55:04.830126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23796,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803896,"kb_used_data":2404,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683464,"statfs":{"total":386547056640,"available":383675867136,"internally_reserved":0,"allocated":2461696,"data_stored":1253808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001340"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17200,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T04:55:01.348053+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:01.348053+0000","last_peered":"2024-09-16T04:55:01.348053+0000","last_clean":"2024-09-16T04:55:01.348053+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:01.348053+0000","last_undegraded":"2024-09-16T04:55:01.348053+0000","last_fullsized":"2024-09-16T04:55:01.348053+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12808,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.363433+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:02.363433+0000","last_peered":"2024-09-16T04:55:02.363433+0000","last_clean":"2024-09-16T04:55:02.363433+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:02.363433+0000","last_undegraded":"2024-09-16T04:55:02.363433+0000","last_fullsized":"2024-09-16T04:55:02.363433+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.015792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17268,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360251+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:02.360251+0000","last_peered":"2024-09-16T04:55:02.360251+0000","last_clean":"2024-09-16T04:55:02.360251+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:02.360251+0000","last_undegraded":"2024-09-16T04:55:02.360251+0000","last_fullsized":"2024-09-16T04:55:02.360251+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.832463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17263,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359706+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:02.359706+0000","last_peered":"2024-09-16T04:55:02.359706+0000","last_clean":"2024-09-16T04:55:02.359706+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:02.359706+0000","last_undegraded":"2024-09-16T04:55:02.359706+0000","last_fullsized":"2024-09-16T04:55:02.359706+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12921,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:01.348250+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:55:01.348250+0000","last_peered":"2024-09-16T04:55:01.348250+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:01.348250+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T04:55:01.347500+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:01.347500+0000","last_peered":"2024-09-16T04:55:01.347500+0000","last_clean":"2024-09-16T04:55:01.347500+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:01.347500+0000","last_undegraded":"2024-09-16T04:55:01.347500+0000","last_fullsized":"2024-09-16T04:55:01.347500+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17181,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T04:54:50.316261+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:54:50.316261+0000","last_peered":"2024-09-16T04:54:50.316261+0000","last_clean":"2024-09-16T04:54:50.316261+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:54:50.316261+0000","last_undegraded":"2024-09-16T04:54:50.316261+0000","last_fullsized":"2024-09-16T04:54:50.316261+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:54:54.131963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17396,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360275+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:02.360275+0000","last_peered":"2024-09-16T04:55:02.360275+0000","last_clean":"2024-09-16T04:55:02.360275+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:02.360275+0000","last_undegraded":"2024-09-16T04:55:02.360275+0000","last_fullsized":"2024-09-16T04:55:02.360275+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17558,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T04:55:01.347403+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:01.347403+0000","last_peered":"2024-09-16T04:55:01.347403+0000","last_clean":"2024-09-16T04:55:01.347403+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:01.347403+0000","last_undegraded":"2024-09-16T04:55:01.347403+0000","last_fullsized":"2024-09-16T04:55:01.347403+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17305,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360346+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:02.360346+0000","last_peered":"2024-09-16T04:55:02.360346+0000","last_clean":"2024-09-16T04:55:02.360346+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:02.360346+0000","last_undegraded":"2024-09-16T04:55:02.360346+0000","last_fullsized":"2024-09-16T04:55:02.360346+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.832435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12862,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:01.924675+0000","last_change":"2024-09-16T04:55:01.243932+0000","last_active":"2024-09-16T04:55:01.924675+0000","last_peered":"2024-09-16T04:55:01.924675+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:55:01.924675+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12450,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360751+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:02.360751+0000","last_peered":"2024-09-16T04:55:02.360751+0000","last_clean":"2024-09-16T04:55:02.360751+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:02.360751+0000","last_undegraded":"2024-09-16T04:55:02.360751+0000","last_fullsized":"2024-09-16T04:55:02.360751+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12664,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359861+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:02.359861+0000","last_peered":"2024-09-16T04:55:02.359861+0000","last_clean":"2024-09-16T04:55:02.359861+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:02.359861+0000","last_undegraded":"2024-09-16T04:55:02.359861+0000","last_fullsized":"2024-09-16T04:55:02.359861+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12932,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.447982+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:02.447982+0000","last_peered":"2024-09-16T04:55:02.447982+0000","last_clean":"2024-09-16T04:55:02.447982+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:02.447982+0000","last_undegraded":"2024-09-16T04:55:02.447982+0000","last_fullsized":"2024-09-16T04:55:02.447982+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:07.937744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14969,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360154+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:02.360154+0000","last_peered":"2024-09-16T04:55:02.360154+0000","last_clean":"2024-09-16T04:55:02.360154+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:02.360154+0000","last_undegraded":"2024-09-16T04:55:02.360154+0000","last_fullsized":"2024-09-16T04:55:02.360154+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12439,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360378+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:02.360378+0000","last_peered":"2024-09-16T04:55:02.360378+0000","last_clean":"2024-09-16T04:55:02.360378+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:02.360378+0000","last_undegraded":"2024-09-16T04:55:02.360378+0000","last_fullsized":"2024-09-16T04:55:02.360378+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.015805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23796,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722160,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649680,"statfs":{"total":96636764160,"available":95897272320,"internally_reserved":0,"allocated":487424,"data_stored":186725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731388,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640452,"statfs":{"total":96636764160,"available":95887822848,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628256,"kb_used_data":976,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93743584,"statfs":{"total":96636764160,"available":95993430016,"internally_reserved":0,"allocated":999424,"data_stored":693654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722092,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649748,"statfs":{"total":96636764160,"available":95897341952,"internally_reserved":0,"allocated":479232,"data_stored":183583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":507904,"data_stored":507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:08.442 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:08.782 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:08.859 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:08.860 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-09-16T04:55:06.830494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23796,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803912,"kb_used_data":2420,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683448,"statfs":{"total":386547056640,"available":383675850752,"internally_reserved":0,"allocated":2478080,"data_stored":1260071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001405"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17200,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T04:55:01.348053+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:01.348053+0000","last_peered":"2024-09-16T04:55:01.348053+0000","last_clean":"2024-09-16T04:55:01.348053+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:01.348053+0000","last_undegraded":"2024-09-16T04:55:01.348053+0000","last_fullsized":"2024-09-16T04:55:01.348053+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12808,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.363433+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:02.363433+0000","last_peered":"2024-09-16T04:55:02.363433+0000","last_clean":"2024-09-16T04:55:02.363433+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:02.363433+0000","last_undegraded":"2024-09-16T04:55:02.363433+0000","last_fullsized":"2024-09-16T04:55:02.363433+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.015792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17268,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360251+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:02.360251+0000","last_peered":"2024-09-16T04:55:02.360251+0000","last_clean":"2024-09-16T04:55:02.360251+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:02.360251+0000","last_undegraded":"2024-09-16T04:55:02.360251+0000","last_fullsized":"2024-09-16T04:55:02.360251+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.832463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17263,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359706+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:02.359706+0000","last_peered":"2024-09-16T04:55:02.359706+0000","last_clean":"2024-09-16T04:55:02.359706+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:02.359706+0000","last_undegraded":"2024-09-16T04:55:02.359706+0000","last_fullsized":"2024-09-16T04:55:02.359706+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12921,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:01.348250+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:55:01.348250+0000","last_peered":"2024-09-16T04:55:01.348250+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:01.348250+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T04:55:01.347500+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:01.347500+0000","last_peered":"2024-09-16T04:55:01.347500+0000","last_clean":"2024-09-16T04:55:01.347500+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:01.347500+0000","last_undegraded":"2024-09-16T04:55:01.347500+0000","last_fullsized":"2024-09-16T04:55:01.347500+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17185,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.362550+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:02.362550+0000","last_peered":"2024-09-16T04:55:02.362550+0000","last_clean":"2024-09-16T04:55:02.362550+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:02.362550+0000","last_undegraded":"2024-09-16T04:55:02.362550+0000","last_fullsized":"2024-09-16T04:55:02.362550+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.198475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17396,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360275+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:02.360275+0000","last_peered":"2024-09-16T04:55:02.360275+0000","last_clean":"2024-09-16T04:55:02.360275+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:02.360275+0000","last_undegraded":"2024-09-16T04:55:02.360275+0000","last_fullsized":"2024-09-16T04:55:02.360275+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17558,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T04:55:01.347403+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:01.347403+0000","last_peered":"2024-09-16T04:55:01.347403+0000","last_clean":"2024-09-16T04:55:01.347403+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:01.347403+0000","last_undegraded":"2024-09-16T04:55:01.347403+0000","last_fullsized":"2024-09-16T04:55:01.347403+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17305,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360346+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:02.360346+0000","last_peered":"2024-09-16T04:55:02.360346+0000","last_clean":"2024-09-16T04:55:02.360346+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:02.360346+0000","last_undegraded":"2024-09-16T04:55:02.360346+0000","last_fullsized":"2024-09-16T04:55:02.360346+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.832435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":12862,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:01.924675+0000","last_change":"2024-09-16T04:55:01.243932+0000","last_active":"2024-09-16T04:55:01.924675+0000","last_peered":"2024-09-16T04:55:01.924675+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:55:01.924675+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12450,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360751+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:02.360751+0000","last_peered":"2024-09-16T04:55:02.360751+0000","last_clean":"2024-09-16T04:55:02.360751+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:02.360751+0000","last_undegraded":"2024-09-16T04:55:02.360751+0000","last_fullsized":"2024-09-16T04:55:02.360751+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12664,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359861+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:02.359861+0000","last_peered":"2024-09-16T04:55:02.359861+0000","last_clean":"2024-09-16T04:55:02.359861+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:02.359861+0000","last_undegraded":"2024-09-16T04:55:02.359861+0000","last_fullsized":"2024-09-16T04:55:02.359861+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12932,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.447982+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:02.447982+0000","last_peered":"2024-09-16T04:55:02.447982+0000","last_clean":"2024-09-16T04:55:02.447982+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:02.447982+0000","last_undegraded":"2024-09-16T04:55:02.447982+0000","last_fullsized":"2024-09-16T04:55:02.447982+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:07.937744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14969,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360154+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:02.360154+0000","last_peered":"2024-09-16T04:55:02.360154+0000","last_clean":"2024-09-16T04:55:02.360154+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:02.360154+0000","last_undegraded":"2024-09-16T04:55:02.360154+0000","last_fullsized":"2024-09-16T04:55:02.360154+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12439,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360378+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:02.360378+0000","last_peered":"2024-09-16T04:55:02.360378+0000","last_clean":"2024-09-16T04:55:02.360378+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:02.360378+0000","last_undegraded":"2024-09-16T04:55:02.360378+0000","last_fullsized":"2024-09-16T04:55:02.360378+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.015805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23796,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722160,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649680,"statfs":{"total":96636764160,"available":95897272320,"internally_reserved":0,"allocated":487424,"data_stored":186725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731388,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640452,"statfs":{"total":96636764160,"available":95887822848,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628256,"kb_used_data":976,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93743584,"statfs":{"total":96636764160,"available":95993430016,"internally_reserved":0,"allocated":999424,"data_stored":693654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":507904,"data_stored":507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:08.862 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:09.164 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:09.165 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:09.177 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-16T04:55:08.830784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24616,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803924,"kb_used_data":2428,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683436,"statfs":{"total":386547056640,"available":383675838464,"internally_reserved":0,"allocated":2486272,"data_stored":1263192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-344,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001304"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17202,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359077+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:02.359077+0000","last_peered":"2024-09-16T04:55:02.359077+0000","last_clean":"2024-09-16T04:55:02.359077+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:02.359077+0000","last_undegraded":"2024-09-16T04:55:02.359077+0000","last_fullsized":"2024-09-16T04:55:02.359077+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12808,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.363433+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:02.363433+0000","last_peered":"2024-09-16T04:55:02.363433+0000","last_clean":"2024-09-16T04:55:02.363433+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:02.363433+0000","last_undegraded":"2024-09-16T04:55:02.363433+0000","last_fullsized":"2024-09-16T04:55:02.363433+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.015792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17268,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360251+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:02.360251+0000","last_peered":"2024-09-16T04:55:02.360251+0000","last_clean":"2024-09-16T04:55:02.360251+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:02.360251+0000","last_undegraded":"2024-09-16T04:55:02.360251+0000","last_fullsized":"2024-09-16T04:55:02.360251+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.832463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17263,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359706+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:02.359706+0000","last_peered":"2024-09-16T04:55:02.359706+0000","last_clean":"2024-09-16T04:55:02.359706+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:02.359706+0000","last_undegraded":"2024-09-16T04:55:02.359706+0000","last_fullsized":"2024-09-16T04:55:02.359706+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12923,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:02.358627+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:55:02.358627+0000","last_peered":"2024-09-16T04:55:02.358627+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:02.358627+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16838,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359067+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:02.359067+0000","last_peered":"2024-09-16T04:55:02.359067+0000","last_clean":"2024-09-16T04:55:02.359067+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:02.359067+0000","last_undegraded":"2024-09-16T04:55:02.359067+0000","last_fullsized":"2024-09-16T04:55:02.359067+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17185,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.362550+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:02.362550+0000","last_peered":"2024-09-16T04:55:02.362550+0000","last_clean":"2024-09-16T04:55:02.362550+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:02.362550+0000","last_undegraded":"2024-09-16T04:55:02.362550+0000","last_fullsized":"2024-09-16T04:55:02.362550+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.198475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17396,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360275+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:02.360275+0000","last_peered":"2024-09-16T04:55:02.360275+0000","last_clean":"2024-09-16T04:55:02.360275+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:02.360275+0000","last_undegraded":"2024-09-16T04:55:02.360275+0000","last_fullsized":"2024-09-16T04:55:02.360275+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17560,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.358956+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:02.358956+0000","last_peered":"2024-09-16T04:55:02.358956+0000","last_clean":"2024-09-16T04:55:02.358956+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:02.358956+0000","last_undegraded":"2024-09-16T04:55:02.358956+0000","last_fullsized":"2024-09-16T04:55:02.358956+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17305,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360346+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:02.360346+0000","last_peered":"2024-09-16T04:55:02.360346+0000","last_clean":"2024-09-16T04:55:02.360346+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:02.360346+0000","last_undegraded":"2024-09-16T04:55:02.360346+0000","last_fullsized":"2024-09-16T04:55:02.360346+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.832435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":14504,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:06.924836+0000","last_change":"2024-09-16T04:55:01.243932+0000","last_active":"2024-09-16T04:55:06.924836+0000","last_peered":"2024-09-16T04:55:06.924836+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:55:06.924836+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":953,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12450,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360751+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:02.360751+0000","last_peered":"2024-09-16T04:55:02.360751+0000","last_clean":"2024-09-16T04:55:02.360751+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:02.360751+0000","last_undegraded":"2024-09-16T04:55:02.360751+0000","last_fullsized":"2024-09-16T04:55:02.360751+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12664,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359861+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:02.359861+0000","last_peered":"2024-09-16T04:55:02.359861+0000","last_clean":"2024-09-16T04:55:02.359861+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:02.359861+0000","last_undegraded":"2024-09-16T04:55:02.359861+0000","last_fullsized":"2024-09-16T04:55:02.359861+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12932,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.447982+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:02.447982+0000","last_peered":"2024-09-16T04:55:02.447982+0000","last_clean":"2024-09-16T04:55:02.447982+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:02.447982+0000","last_undegraded":"2024-09-16T04:55:02.447982+0000","last_fullsized":"2024-09-16T04:55:02.447982+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:07.937744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14969,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360154+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:02.360154+0000","last_peered":"2024-09-16T04:55:02.360154+0000","last_clean":"2024-09-16T04:55:02.360154+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:02.360154+0000","last_undegraded":"2024-09-16T04:55:02.360154+0000","last_fullsized":"2024-09-16T04:55:02.360154+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12439,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360378+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:02.360378+0000","last_peered":"2024-09-16T04:55:02.360378+0000","last_clean":"2024-09-16T04:55:02.360378+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:02.360378+0000","last_undegraded":"2024-09-16T04:55:02.360378+0000","last_fullsized":"2024-09-16T04:55:02.360378+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.015805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24616,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722172,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649668,"statfs":{"total":96636764160,"available":95897260032,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731388,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640452,"statfs":{"total":96636764160,"available":95887822848,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628256,"kb_used_data":976,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93743584,"statfs":{"total":96636764160,"available":95993430016,"internally_reserved":0,"allocated":999424,"data_stored":693654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":507904,"data_stored":507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:09.179 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:55:09.179 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12923, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:55:02.358627+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:55:02.358627+0000', 'last_peered': '2024-09-16T04:55:02.358627+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:55:02.358627+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:55:09.180 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:55:09.180 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 14504, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:55:06.924836+0000', 'last_change': '2024-09-16T04:55:01.243932+0000', 'last_active': '2024-09-16T04:55:06.924836+0000', 'last_peered': '2024-09-16T04:55:06.924836+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:55:06.924836+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1242, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 953, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1239}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:55:09.180 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:55:09.565 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:09.565 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3492/112 objects degraded (3117.857%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":921,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2871218176,"bytes_avail":383675838464,"bytes_total":386547056640,"degraded_objects":3492,"degraded_total":112,"degraded_ratio":31.178571428571427},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:55:09.578 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T04:55:09.579 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:09.868 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:09.869 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:09.881 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-16T04:55:08.830784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24616,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803924,"kb_used_data":2428,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683436,"statfs":{"total":386547056640,"available":383675838464,"internally_reserved":0,"allocated":2486272,"data_stored":1263192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-344,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001304"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17202,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359077+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:02.359077+0000","last_peered":"2024-09-16T04:55:02.359077+0000","last_clean":"2024-09-16T04:55:02.359077+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:02.359077+0000","last_undegraded":"2024-09-16T04:55:02.359077+0000","last_fullsized":"2024-09-16T04:55:02.359077+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12808,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.363433+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:02.363433+0000","last_peered":"2024-09-16T04:55:02.363433+0000","last_clean":"2024-09-16T04:55:02.363433+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:02.363433+0000","last_undegraded":"2024-09-16T04:55:02.363433+0000","last_fullsized":"2024-09-16T04:55:02.363433+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.015792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17268,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360251+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:02.360251+0000","last_peered":"2024-09-16T04:55:02.360251+0000","last_clean":"2024-09-16T04:55:02.360251+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:02.360251+0000","last_undegraded":"2024-09-16T04:55:02.360251+0000","last_fullsized":"2024-09-16T04:55:02.360251+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.832463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17263,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359706+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:02.359706+0000","last_peered":"2024-09-16T04:55:02.359706+0000","last_clean":"2024-09-16T04:55:02.359706+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:02.359706+0000","last_undegraded":"2024-09-16T04:55:02.359706+0000","last_fullsized":"2024-09-16T04:55:02.359706+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12923,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:02.358627+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:55:02.358627+0000","last_peered":"2024-09-16T04:55:02.358627+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:02.358627+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16838,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359067+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:02.359067+0000","last_peered":"2024-09-16T04:55:02.359067+0000","last_clean":"2024-09-16T04:55:02.359067+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:02.359067+0000","last_undegraded":"2024-09-16T04:55:02.359067+0000","last_fullsized":"2024-09-16T04:55:02.359067+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17185,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.362550+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:02.362550+0000","last_peered":"2024-09-16T04:55:02.362550+0000","last_clean":"2024-09-16T04:55:02.362550+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:02.362550+0000","last_undegraded":"2024-09-16T04:55:02.362550+0000","last_fullsized":"2024-09-16T04:55:02.362550+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.198475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17396,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360275+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:02.360275+0000","last_peered":"2024-09-16T04:55:02.360275+0000","last_clean":"2024-09-16T04:55:02.360275+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:02.360275+0000","last_undegraded":"2024-09-16T04:55:02.360275+0000","last_fullsized":"2024-09-16T04:55:02.360275+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17560,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.358956+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:02.358956+0000","last_peered":"2024-09-16T04:55:02.358956+0000","last_clean":"2024-09-16T04:55:02.358956+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:02.358956+0000","last_undegraded":"2024-09-16T04:55:02.358956+0000","last_fullsized":"2024-09-16T04:55:02.358956+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17305,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360346+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:02.360346+0000","last_peered":"2024-09-16T04:55:02.360346+0000","last_clean":"2024-09-16T04:55:02.360346+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:02.360346+0000","last_undegraded":"2024-09-16T04:55:02.360346+0000","last_fullsized":"2024-09-16T04:55:02.360346+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.832435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":14504,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:06.924836+0000","last_change":"2024-09-16T04:55:01.243932+0000","last_active":"2024-09-16T04:55:06.924836+0000","last_peered":"2024-09-16T04:55:06.924836+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:55:06.924836+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":953,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12450,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360751+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:02.360751+0000","last_peered":"2024-09-16T04:55:02.360751+0000","last_clean":"2024-09-16T04:55:02.360751+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:02.360751+0000","last_undegraded":"2024-09-16T04:55:02.360751+0000","last_fullsized":"2024-09-16T04:55:02.360751+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12664,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359861+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:02.359861+0000","last_peered":"2024-09-16T04:55:02.359861+0000","last_clean":"2024-09-16T04:55:02.359861+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:02.359861+0000","last_undegraded":"2024-09-16T04:55:02.359861+0000","last_fullsized":"2024-09-16T04:55:02.359861+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12932,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.447982+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:02.447982+0000","last_peered":"2024-09-16T04:55:02.447982+0000","last_clean":"2024-09-16T04:55:02.447982+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:02.447982+0000","last_undegraded":"2024-09-16T04:55:02.447982+0000","last_fullsized":"2024-09-16T04:55:02.447982+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:07.937744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14969,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360154+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:02.360154+0000","last_peered":"2024-09-16T04:55:02.360154+0000","last_clean":"2024-09-16T04:55:02.360154+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:02.360154+0000","last_undegraded":"2024-09-16T04:55:02.360154+0000","last_fullsized":"2024-09-16T04:55:02.360154+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12439,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360378+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:02.360378+0000","last_peered":"2024-09-16T04:55:02.360378+0000","last_clean":"2024-09-16T04:55:02.360378+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:02.360378+0000","last_undegraded":"2024-09-16T04:55:02.360378+0000","last_fullsized":"2024-09-16T04:55:02.360378+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.015805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24616,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722172,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649668,"statfs":{"total":96636764160,"available":95897260032,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731388,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640452,"statfs":{"total":96636764160,"available":95887822848,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628256,"kb_used_data":976,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93743584,"statfs":{"total":96636764160,"available":95993430016,"internally_reserved":0,"allocated":999424,"data_stored":693654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":507904,"data_stored":507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:12.883 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:13.179 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:13.180 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:13.192 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-16T04:55:12.831446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25610,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2808084,"kb_used_data":2428,"kb_used_omap":14,"kb_used_meta":2805553,"kb_avail":374679276,"statfs":{"total":386547056640,"available":383671578624,"internally_reserved":0,"allocated":2486272,"data_stored":1263192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2872886833},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.472994"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17202,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359077+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:02.359077+0000","last_peered":"2024-09-16T04:55:02.359077+0000","last_clean":"2024-09-16T04:55:02.359077+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:02.359077+0000","last_undegraded":"2024-09-16T04:55:02.359077+0000","last_fullsized":"2024-09-16T04:55:02.359077+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12808,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.363433+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:02.363433+0000","last_peered":"2024-09-16T04:55:02.363433+0000","last_clean":"2024-09-16T04:55:02.363433+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:02.363433+0000","last_undegraded":"2024-09-16T04:55:02.363433+0000","last_fullsized":"2024-09-16T04:55:02.363433+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.015792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17268,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360251+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:02.360251+0000","last_peered":"2024-09-16T04:55:02.360251+0000","last_clean":"2024-09-16T04:55:02.360251+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:02.360251+0000","last_undegraded":"2024-09-16T04:55:02.360251+0000","last_fullsized":"2024-09-16T04:55:02.360251+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.832463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17264,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359771+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:02.359771+0000","last_peered":"2024-09-16T04:55:02.359771+0000","last_clean":"2024-09-16T04:55:02.359771+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:02.359771+0000","last_undegraded":"2024-09-16T04:55:02.359771+0000","last_fullsized":"2024-09-16T04:55:02.359771+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:12.954217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12923,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:02.358627+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:55:02.358627+0000","last_peered":"2024-09-16T04:55:02.358627+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:02.358627+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16838,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359067+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:02.359067+0000","last_peered":"2024-09-16T04:55:02.359067+0000","last_clean":"2024-09-16T04:55:02.359067+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:02.359067+0000","last_undegraded":"2024-09-16T04:55:02.359067+0000","last_fullsized":"2024-09-16T04:55:02.359067+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17186,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.362602+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:02.362602+0000","last_peered":"2024-09-16T04:55:02.362602+0000","last_clean":"2024-09-16T04:55:02.362602+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:02.362602+0000","last_undegraded":"2024-09-16T04:55:02.362602+0000","last_fullsized":"2024-09-16T04:55:02.362602+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.245580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17396,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360275+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:02.360275+0000","last_peered":"2024-09-16T04:55:02.360275+0000","last_clean":"2024-09-16T04:55:02.360275+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:02.360275+0000","last_undegraded":"2024-09-16T04:55:02.360275+0000","last_fullsized":"2024-09-16T04:55:02.360275+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17560,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.358956+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:02.358956+0000","last_peered":"2024-09-16T04:55:02.358956+0000","last_clean":"2024-09-16T04:55:02.358956+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:02.358956+0000","last_undegraded":"2024-09-16T04:55:02.358956+0000","last_fullsized":"2024-09-16T04:55:02.358956+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17305,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360346+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:02.360346+0000","last_peered":"2024-09-16T04:55:02.360346+0000","last_clean":"2024-09-16T04:55:02.360346+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:02.360346+0000","last_undegraded":"2024-09-16T04:55:02.360346+0000","last_fullsized":"2024-09-16T04:55:02.360346+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.832435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":16492,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:11.927864+0000","last_change":"2024-09-16T04:55:01.243932+0000","last_active":"2024-09-16T04:55:11.927864+0000","last_peered":"2024-09-16T04:55:11.927864+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:55:11.927864+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12450,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360751+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:02.360751+0000","last_peered":"2024-09-16T04:55:02.360751+0000","last_clean":"2024-09-16T04:55:02.360751+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:02.360751+0000","last_undegraded":"2024-09-16T04:55:02.360751+0000","last_fullsized":"2024-09-16T04:55:02.360751+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12664,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359861+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:02.359861+0000","last_peered":"2024-09-16T04:55:02.359861+0000","last_clean":"2024-09-16T04:55:02.359861+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:02.359861+0000","last_undegraded":"2024-09-16T04:55:02.359861+0000","last_fullsized":"2024-09-16T04:55:02.359861+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12932,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.447982+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:02.447982+0000","last_peered":"2024-09-16T04:55:02.447982+0000","last_clean":"2024-09-16T04:55:02.447982+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:02.447982+0000","last_undegraded":"2024-09-16T04:55:02.447982+0000","last_fullsized":"2024-09-16T04:55:02.447982+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:07.937744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14969,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360154+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:02.360154+0000","last_peered":"2024-09-16T04:55:02.360154+0000","last_clean":"2024-09-16T04:55:02.360154+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:02.360154+0000","last_undegraded":"2024-09-16T04:55:02.360154+0000","last_fullsized":"2024-09-16T04:55:02.360154+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12439,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360378+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:02.360378+0000","last_peered":"2024-09-16T04:55:02.360378+0000","last_clean":"2024-09-16T04:55:02.360378+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:02.360378+0000","last_undegraded":"2024-09-16T04:55:02.360378+0000","last_fullsized":"2024-09-16T04:55:02.360378+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.015805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25610,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722172,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649668,"statfs":{"total":96636764160,"available":95897260032,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731388,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640452,"statfs":{"total":96636764160,"available":95887822848,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632416,"kb_used_data":976,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739424,"statfs":{"total":96636764160,"available":95989170176,"internally_reserved":0,"allocated":999424,"data_stored":693654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646573775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":507904,"data_stored":507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:13.194 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:13.518 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:13.519 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:13.531 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":514,"stamp":"2024-09-16T04:55:13.401031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25610,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2808084,"kb_used_data":2428,"kb_used_omap":14,"kb_used_meta":2805553,"kb_avail":374679276,"statfs":{"total":386547056640,"available":383671578624,"internally_reserved":0,"allocated":2486272,"data_stored":1263192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2872886833},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.571258"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17202,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359077+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:02.359077+0000","last_peered":"2024-09-16T04:55:02.359077+0000","last_clean":"2024-09-16T04:55:02.359077+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:02.359077+0000","last_undegraded":"2024-09-16T04:55:02.359077+0000","last_fullsized":"2024-09-16T04:55:02.359077+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12808,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.363433+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:02.363433+0000","last_peered":"2024-09-16T04:55:02.363433+0000","last_clean":"2024-09-16T04:55:02.363433+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:02.363433+0000","last_undegraded":"2024-09-16T04:55:02.363433+0000","last_fullsized":"2024-09-16T04:55:02.363433+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.015792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17268,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360251+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:02.360251+0000","last_peered":"2024-09-16T04:55:02.360251+0000","last_clean":"2024-09-16T04:55:02.360251+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:02.360251+0000","last_undegraded":"2024-09-16T04:55:02.360251+0000","last_fullsized":"2024-09-16T04:55:02.360251+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.832463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17264,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359771+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:02.359771+0000","last_peered":"2024-09-16T04:55:02.359771+0000","last_clean":"2024-09-16T04:55:02.359771+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:02.359771+0000","last_undegraded":"2024-09-16T04:55:02.359771+0000","last_fullsized":"2024-09-16T04:55:02.359771+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:12.954217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12923,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:02.358627+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:55:02.358627+0000","last_peered":"2024-09-16T04:55:02.358627+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:02.358627+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16838,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359067+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:02.359067+0000","last_peered":"2024-09-16T04:55:02.359067+0000","last_clean":"2024-09-16T04:55:02.359067+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:02.359067+0000","last_undegraded":"2024-09-16T04:55:02.359067+0000","last_fullsized":"2024-09-16T04:55:02.359067+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17186,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.362602+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:02.362602+0000","last_peered":"2024-09-16T04:55:02.362602+0000","last_clean":"2024-09-16T04:55:02.362602+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:02.362602+0000","last_undegraded":"2024-09-16T04:55:02.362602+0000","last_fullsized":"2024-09-16T04:55:02.362602+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.245580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17396,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360275+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:02.360275+0000","last_peered":"2024-09-16T04:55:02.360275+0000","last_clean":"2024-09-16T04:55:02.360275+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:02.360275+0000","last_undegraded":"2024-09-16T04:55:02.360275+0000","last_fullsized":"2024-09-16T04:55:02.360275+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17560,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.358956+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:02.358956+0000","last_peered":"2024-09-16T04:55:02.358956+0000","last_clean":"2024-09-16T04:55:02.358956+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:02.358956+0000","last_undegraded":"2024-09-16T04:55:02.358956+0000","last_fullsized":"2024-09-16T04:55:02.358956+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17305,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360346+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:02.360346+0000","last_peered":"2024-09-16T04:55:02.360346+0000","last_clean":"2024-09-16T04:55:02.360346+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:02.360346+0000","last_undegraded":"2024-09-16T04:55:02.360346+0000","last_fullsized":"2024-09-16T04:55:02.360346+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.832435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":16492,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:11.927864+0000","last_change":"2024-09-16T04:55:01.243932+0000","last_active":"2024-09-16T04:55:11.927864+0000","last_peered":"2024-09-16T04:55:11.927864+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:55:11.927864+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12450,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360751+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:02.360751+0000","last_peered":"2024-09-16T04:55:02.360751+0000","last_clean":"2024-09-16T04:55:02.360751+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:02.360751+0000","last_undegraded":"2024-09-16T04:55:02.360751+0000","last_fullsized":"2024-09-16T04:55:02.360751+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12664,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359861+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:02.359861+0000","last_peered":"2024-09-16T04:55:02.359861+0000","last_clean":"2024-09-16T04:55:02.359861+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:02.359861+0000","last_undegraded":"2024-09-16T04:55:02.359861+0000","last_fullsized":"2024-09-16T04:55:02.359861+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12932,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.447982+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:02.447982+0000","last_peered":"2024-09-16T04:55:02.447982+0000","last_clean":"2024-09-16T04:55:02.447982+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:02.447982+0000","last_undegraded":"2024-09-16T04:55:02.447982+0000","last_fullsized":"2024-09-16T04:55:02.447982+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:07.937744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14969,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360154+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:02.360154+0000","last_peered":"2024-09-16T04:55:02.360154+0000","last_clean":"2024-09-16T04:55:02.360154+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:02.360154+0000","last_undegraded":"2024-09-16T04:55:02.360154+0000","last_fullsized":"2024-09-16T04:55:02.360154+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12439,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360378+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:02.360378+0000","last_peered":"2024-09-16T04:55:02.360378+0000","last_clean":"2024-09-16T04:55:02.360378+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:02.360378+0000","last_undegraded":"2024-09-16T04:55:02.360378+0000","last_fullsized":"2024-09-16T04:55:02.360378+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.015805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25610,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722172,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649668,"statfs":{"total":96636764160,"available":95897260032,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731388,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640452,"statfs":{"total":96636764160,"available":95887822848,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632416,"kb_used_data":976,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739424,"statfs":{"total":96636764160,"available":95989170176,"internally_reserved":0,"allocated":999424,"data_stored":693654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646573775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":507904,"data_stored":507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:13.532 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:55:13.532 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12923, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:55:02.358627+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:55:02.358627+0000', 'last_peered': '2024-09-16T04:55:02.358627+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:55:02.358627+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:55:13.533 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:55:13.533 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 16492, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:55:11.927864+0000', 'last_change': '2024-09-16T04:55:01.243932+0000', 'last_active': '2024-09-16T04:55:11.927864+0000', 'last_peered': '2024-09-16T04:55:11.927864+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:55:11.927864+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1947, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -245}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:55:13.533 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:55:13.896 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:13.896 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2498/112 objects degraded (2230.357%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":926,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":60,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2875478016,"bytes_avail":383671578624,"bytes_total":386547056640,"degraded_objects":2498,"degraded_total":112,"degraded_ratio":22.303571428571427,"recovering_objects_per_sec":52,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":554,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:55:13.910 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:55:13.910 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:14.217 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:14.217 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:14.229 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":514,"stamp":"2024-09-16T04:55:13.401031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25610,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2808084,"kb_used_data":2428,"kb_used_omap":14,"kb_used_meta":2805553,"kb_avail":374679276,"statfs":{"total":386547056640,"available":383671578624,"internally_reserved":0,"allocated":2486272,"data_stored":1263192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2872886833},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.571258"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17202,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359077+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:02.359077+0000","last_peered":"2024-09-16T04:55:02.359077+0000","last_clean":"2024-09-16T04:55:02.359077+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:02.359077+0000","last_undegraded":"2024-09-16T04:55:02.359077+0000","last_fullsized":"2024-09-16T04:55:02.359077+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12808,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.363433+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:02.363433+0000","last_peered":"2024-09-16T04:55:02.363433+0000","last_clean":"2024-09-16T04:55:02.363433+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:02.363433+0000","last_undegraded":"2024-09-16T04:55:02.363433+0000","last_fullsized":"2024-09-16T04:55:02.363433+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.015792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17268,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360251+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:02.360251+0000","last_peered":"2024-09-16T04:55:02.360251+0000","last_clean":"2024-09-16T04:55:02.360251+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:02.360251+0000","last_undegraded":"2024-09-16T04:55:02.360251+0000","last_fullsized":"2024-09-16T04:55:02.360251+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.832463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17264,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359771+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:02.359771+0000","last_peered":"2024-09-16T04:55:02.359771+0000","last_clean":"2024-09-16T04:55:02.359771+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:02.359771+0000","last_undegraded":"2024-09-16T04:55:02.359771+0000","last_fullsized":"2024-09-16T04:55:02.359771+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:12.954217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12923,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:02.358627+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:55:02.358627+0000","last_peered":"2024-09-16T04:55:02.358627+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:02.358627+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16838,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359067+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:02.359067+0000","last_peered":"2024-09-16T04:55:02.359067+0000","last_clean":"2024-09-16T04:55:02.359067+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:02.359067+0000","last_undegraded":"2024-09-16T04:55:02.359067+0000","last_fullsized":"2024-09-16T04:55:02.359067+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17186,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.362602+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:02.362602+0000","last_peered":"2024-09-16T04:55:02.362602+0000","last_clean":"2024-09-16T04:55:02.362602+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:02.362602+0000","last_undegraded":"2024-09-16T04:55:02.362602+0000","last_fullsized":"2024-09-16T04:55:02.362602+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.245580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17396,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360275+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:02.360275+0000","last_peered":"2024-09-16T04:55:02.360275+0000","last_clean":"2024-09-16T04:55:02.360275+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:02.360275+0000","last_undegraded":"2024-09-16T04:55:02.360275+0000","last_fullsized":"2024-09-16T04:55:02.360275+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17560,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.358956+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:02.358956+0000","last_peered":"2024-09-16T04:55:02.358956+0000","last_clean":"2024-09-16T04:55:02.358956+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:02.358956+0000","last_undegraded":"2024-09-16T04:55:02.358956+0000","last_fullsized":"2024-09-16T04:55:02.358956+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17305,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360346+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:02.360346+0000","last_peered":"2024-09-16T04:55:02.360346+0000","last_clean":"2024-09-16T04:55:02.360346+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:02.360346+0000","last_undegraded":"2024-09-16T04:55:02.360346+0000","last_fullsized":"2024-09-16T04:55:02.360346+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:05.832435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":16492,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:11.927864+0000","last_change":"2024-09-16T04:55:01.243932+0000","last_active":"2024-09-16T04:55:11.927864+0000","last_peered":"2024-09-16T04:55:11.927864+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:55:11.927864+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12450,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360751+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:02.360751+0000","last_peered":"2024-09-16T04:55:02.360751+0000","last_clean":"2024-09-16T04:55:02.360751+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:02.360751+0000","last_undegraded":"2024-09-16T04:55:02.360751+0000","last_fullsized":"2024-09-16T04:55:02.360751+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12664,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359861+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:02.359861+0000","last_peered":"2024-09-16T04:55:02.359861+0000","last_clean":"2024-09-16T04:55:02.359861+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:02.359861+0000","last_undegraded":"2024-09-16T04:55:02.359861+0000","last_fullsized":"2024-09-16T04:55:02.359861+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12932,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.447982+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:02.447982+0000","last_peered":"2024-09-16T04:55:02.447982+0000","last_clean":"2024-09-16T04:55:02.447982+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:02.447982+0000","last_undegraded":"2024-09-16T04:55:02.447982+0000","last_fullsized":"2024-09-16T04:55:02.447982+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:07.937744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14969,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360154+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:02.360154+0000","last_peered":"2024-09-16T04:55:02.360154+0000","last_clean":"2024-09-16T04:55:02.360154+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:02.360154+0000","last_undegraded":"2024-09-16T04:55:02.360154+0000","last_fullsized":"2024-09-16T04:55:02.360154+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:06.970818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12439,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.360378+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:02.360378+0000","last_peered":"2024-09-16T04:55:02.360378+0000","last_clean":"2024-09-16T04:55:02.360378+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:02.360378+0000","last_undegraded":"2024-09-16T04:55:02.360378+0000","last_fullsized":"2024-09-16T04:55:02.360378+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.015805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25610,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722172,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649668,"statfs":{"total":96636764160,"available":95897260032,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731388,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640452,"statfs":{"total":96636764160,"available":95887822848,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632416,"kb_used_data":976,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739424,"statfs":{"total":96636764160,"available":95989170176,"internally_reserved":0,"allocated":999424,"data_stored":693654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646573775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722108,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649732,"statfs":{"total":96636764160,"available":95897325568,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":507904,"data_stored":507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:17.231 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:17.539 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:17.539 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:17.551 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-16T04:55:16.832198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25610,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799784,"kb_used_data":2444,"kb_used_omap":14,"kb_used_meta":2797233,"kb_avail":374687576,"statfs":{"total":386547056640,"available":383680077824,"internally_reserved":0,"allocated":2502656,"data_stored":1264294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2864367153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":994,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001413"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17202,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359077+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:02.359077+0000","last_peered":"2024-09-16T04:55:02.359077+0000","last_clean":"2024-09-16T04:55:02.359077+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:02.359077+0000","last_undegraded":"2024-09-16T04:55:02.359077+0000","last_fullsized":"2024-09-16T04:55:02.359077+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406667+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:14.406667+0000","last_peered":"2024-09-16T04:55:14.406667+0000","last_clean":"2024-09-16T04:55:14.406667+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:14.406667+0000","last_undegraded":"2024-09-16T04:55:14.406667+0000","last_fullsized":"2024-09-16T04:55:14.406667+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17270,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T04:55:13.402659+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:13.402659+0000","last_peered":"2024-09-16T04:55:13.402659+0000","last_clean":"2024-09-16T04:55:13.402659+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:13.402659+0000","last_undegraded":"2024-09-16T04:55:13.402659+0000","last_fullsized":"2024-09-16T04:55:13.402659+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.742016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17268,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12923,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:02.358627+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:55:02.358627+0000","last_peered":"2024-09-16T04:55:02.358627+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:02.358627+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16838,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359067+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:02.359067+0000","last_peered":"2024-09-16T04:55:02.359067+0000","last_clean":"2024-09-16T04:55:02.359067+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:02.359067+0000","last_undegraded":"2024-09-16T04:55:02.359067+0000","last_fullsized":"2024-09-16T04:55:02.359067+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17190,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408911+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:14.408911+0000","last_peered":"2024-09-16T04:55:14.408911+0000","last_clean":"2024-09-16T04:55:14.408911+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:14.408911+0000","last_undegraded":"2024-09-16T04:55:14.408911+0000","last_fullsized":"2024-09-16T04:55:14.408911+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.245580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17400,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406720+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:14.406720+0000","last_peered":"2024-09-16T04:55:14.406720+0000","last_clean":"2024-09-16T04:55:14.406720+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:14.406720+0000","last_undegraded":"2024-09-16T04:55:14.406720+0000","last_fullsized":"2024-09-16T04:55:14.406720+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17560,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.358956+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:02.358956+0000","last_peered":"2024-09-16T04:55:02.358956+0000","last_clean":"2024-09-16T04:55:02.358956+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:02.358956+0000","last_undegraded":"2024-09-16T04:55:02.358956+0000","last_fullsized":"2024-09-16T04:55:02.358956+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17307,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T04:55:13.402769+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:13.402769+0000","last_peered":"2024-09-16T04:55:13.402769+0000","last_clean":"2024-09-16T04:55:13.402769+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:13.402769+0000","last_undegraded":"2024-09-16T04:55:13.402769+0000","last_fullsized":"2024-09-16T04:55:13.402769+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.741986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":16492,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:11.927864+0000","last_change":"2024-09-16T04:55:01.243932+0000","last_active":"2024-09-16T04:55:11.927864+0000","last_peered":"2024-09-16T04:55:11.927864+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:55:11.927864+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12454,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.407035+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:14.407035+0000","last_peered":"2024-09-16T04:55:14.407035+0000","last_clean":"2024-09-16T04:55:14.407035+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:14.407035+0000","last_undegraded":"2024-09-16T04:55:14.407035+0000","last_fullsized":"2024-09-16T04:55:14.407035+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12668,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406548+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:14.406548+0000","last_peered":"2024-09-16T04:55:14.406548+0000","last_clean":"2024-09-16T04:55:14.406548+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:14.406548+0000","last_undegraded":"2024-09-16T04:55:14.406548+0000","last_fullsized":"2024-09-16T04:55:14.406548+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12936,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408766+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:14.408766+0000","last_peered":"2024-09-16T04:55:14.408766+0000","last_clean":"2024-09-16T04:55:14.408766+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:14.408766+0000","last_undegraded":"2024-09-16T04:55:14.408766+0000","last_fullsized":"2024-09-16T04:55:14.408766+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14974,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406670+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:14.406670+0000","last_peered":"2024-09-16T04:55:14.406670+0000","last_clean":"2024-09-16T04:55:14.406670+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:14.406670+0000","last_undegraded":"2024-09-16T04:55:14.406670+0000","last_fullsized":"2024-09-16T04:55:14.406670+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12443,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406816+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:14.406816+0000","last_peered":"2024-09-16T04:55:14.406816+0000","last_clean":"2024-09-16T04:55:14.406816+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:14.406816+0000","last_undegraded":"2024-09-16T04:55:14.406816+0000","last_fullsized":"2024-09-16T04:55:14.406816+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25610,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":512000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722172,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649668,"statfs":{"total":96636764160,"available":95897260032,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731388,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640452,"statfs":{"total":96636764160,"available":95887822848,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624100,"kb_used_data":976,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93747740,"statfs":{"total":96636764160,"available":95997685760,"internally_reserved":0,"allocated":999424,"data_stored":688567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":638054095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":499712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:17.553 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:17.860 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-16T04:55:17.859+0000 7fbfea283640 0 --2- 172.21.15.152:0/2612052465 >> [v2:172.21.15.152:3301/0,v1:172.21.15.152:6790/0] conn(0x7fbfe412d280 0x7fbfe41216f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T04:55:18.052 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:18.053 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:18.065 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-16T04:55:16.832198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25610,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799784,"kb_used_data":2444,"kb_used_omap":14,"kb_used_meta":2797233,"kb_avail":374687576,"statfs":{"total":386547056640,"available":383680077824,"internally_reserved":0,"allocated":2502656,"data_stored":1264294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2864367153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":994,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001413"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17202,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359077+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:02.359077+0000","last_peered":"2024-09-16T04:55:02.359077+0000","last_clean":"2024-09-16T04:55:02.359077+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:02.359077+0000","last_undegraded":"2024-09-16T04:55:02.359077+0000","last_fullsized":"2024-09-16T04:55:02.359077+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406667+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:14.406667+0000","last_peered":"2024-09-16T04:55:14.406667+0000","last_clean":"2024-09-16T04:55:14.406667+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:14.406667+0000","last_undegraded":"2024-09-16T04:55:14.406667+0000","last_fullsized":"2024-09-16T04:55:14.406667+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17270,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T04:55:13.402659+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:13.402659+0000","last_peered":"2024-09-16T04:55:13.402659+0000","last_clean":"2024-09-16T04:55:13.402659+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:13.402659+0000","last_undegraded":"2024-09-16T04:55:13.402659+0000","last_fullsized":"2024-09-16T04:55:13.402659+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.742016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17268,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12923,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:02.358627+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:55:02.358627+0000","last_peered":"2024-09-16T04:55:02.358627+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:02.358627+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16838,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359067+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:02.359067+0000","last_peered":"2024-09-16T04:55:02.359067+0000","last_clean":"2024-09-16T04:55:02.359067+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:02.359067+0000","last_undegraded":"2024-09-16T04:55:02.359067+0000","last_fullsized":"2024-09-16T04:55:02.359067+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17190,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408911+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:14.408911+0000","last_peered":"2024-09-16T04:55:14.408911+0000","last_clean":"2024-09-16T04:55:14.408911+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:14.408911+0000","last_undegraded":"2024-09-16T04:55:14.408911+0000","last_fullsized":"2024-09-16T04:55:14.408911+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.245580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17400,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406720+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:14.406720+0000","last_peered":"2024-09-16T04:55:14.406720+0000","last_clean":"2024-09-16T04:55:14.406720+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:14.406720+0000","last_undegraded":"2024-09-16T04:55:14.406720+0000","last_fullsized":"2024-09-16T04:55:14.406720+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17560,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.358956+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:02.358956+0000","last_peered":"2024-09-16T04:55:02.358956+0000","last_clean":"2024-09-16T04:55:02.358956+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:02.358956+0000","last_undegraded":"2024-09-16T04:55:02.358956+0000","last_fullsized":"2024-09-16T04:55:02.358956+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17307,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T04:55:13.402769+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:13.402769+0000","last_peered":"2024-09-16T04:55:13.402769+0000","last_clean":"2024-09-16T04:55:13.402769+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:13.402769+0000","last_undegraded":"2024-09-16T04:55:13.402769+0000","last_fullsized":"2024-09-16T04:55:13.402769+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.741986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":16492,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:11.927864+0000","last_change":"2024-09-16T04:55:01.243932+0000","last_active":"2024-09-16T04:55:11.927864+0000","last_peered":"2024-09-16T04:55:11.927864+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:55:11.927864+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12454,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.407035+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:14.407035+0000","last_peered":"2024-09-16T04:55:14.407035+0000","last_clean":"2024-09-16T04:55:14.407035+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:14.407035+0000","last_undegraded":"2024-09-16T04:55:14.407035+0000","last_fullsized":"2024-09-16T04:55:14.407035+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12668,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406548+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:14.406548+0000","last_peered":"2024-09-16T04:55:14.406548+0000","last_clean":"2024-09-16T04:55:14.406548+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:14.406548+0000","last_undegraded":"2024-09-16T04:55:14.406548+0000","last_fullsized":"2024-09-16T04:55:14.406548+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12936,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408766+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:14.408766+0000","last_peered":"2024-09-16T04:55:14.408766+0000","last_clean":"2024-09-16T04:55:14.408766+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:14.408766+0000","last_undegraded":"2024-09-16T04:55:14.408766+0000","last_fullsized":"2024-09-16T04:55:14.408766+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14974,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406670+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:14.406670+0000","last_peered":"2024-09-16T04:55:14.406670+0000","last_clean":"2024-09-16T04:55:14.406670+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:14.406670+0000","last_undegraded":"2024-09-16T04:55:14.406670+0000","last_fullsized":"2024-09-16T04:55:14.406670+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12443,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406816+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:14.406816+0000","last_peered":"2024-09-16T04:55:14.406816+0000","last_clean":"2024-09-16T04:55:14.406816+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:14.406816+0000","last_undegraded":"2024-09-16T04:55:14.406816+0000","last_fullsized":"2024-09-16T04:55:14.406816+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25610,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":512000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722172,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649668,"statfs":{"total":96636764160,"available":95897260032,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731388,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640452,"statfs":{"total":96636764160,"available":95887822848,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624100,"kb_used_data":976,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93747740,"statfs":{"total":96636764160,"available":95997685760,"internally_reserved":0,"allocated":999424,"data_stored":688567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":638054095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":499712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:18.066 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:55:18.066 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 12923, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:55:02.358627+0000', 'last_change': '2024-09-16T04:41:27.900794+0000', 'last_active': '2024-09-16T04:55:02.358627+0000', 'last_peered': '2024-09-16T04:55:02.358627+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:55:02.358627+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:55:18.066 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-16T04:55:18.066 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'12488", 'reported_seq': 16492, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:55:11.927864+0000', 'last_change': '2024-09-16T04:55:01.243932+0000', 'last_active': '2024-09-16T04:55:11.927864+0000', 'last_peered': '2024-09-16T04:55:11.927864+0000', 'last_clean': '2024-09-16T04:40:23.274621+0000', 'last_became_active': '2024-09-16T04:41:27.856379+0000', 'last_became_peered': '2024-09-16T04:41:27.856379+0000', 'last_unstale': '2024-09-16T04:55:11.927864+0000', 'last_undegraded': '2024-09-16T04:41:27.541103+0000', 'last_fullsized': '2024-09-16T04:41:27.483034+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20370, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 248, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12485, 'num_write_kb': 49948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1947, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -245}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:55:18.067 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:55:18.429 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:18.429 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2498/112 objects degraded (2230.357%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":930,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2866978816,"bytes_avail":383680077824,"bytes_total":386547056640,"degraded_objects":2498,"degraded_total":112,"degraded_ratio":22.303571428571427,"recovering_objects_per_sec":124,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":994,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:55:18.442 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:55:18.443 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:18.739 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:18.740 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:18.752 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-16T04:55:16.832198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25610,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799784,"kb_used_data":2444,"kb_used_omap":14,"kb_used_meta":2797233,"kb_avail":374687576,"statfs":{"total":386547056640,"available":383680077824,"internally_reserved":0,"allocated":2502656,"data_stored":1264294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2864367153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":994,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001413"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17202,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359077+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:02.359077+0000","last_peered":"2024-09-16T04:55:02.359077+0000","last_clean":"2024-09-16T04:55:02.359077+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:02.359077+0000","last_undegraded":"2024-09-16T04:55:02.359077+0000","last_fullsized":"2024-09-16T04:55:02.359077+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406667+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:14.406667+0000","last_peered":"2024-09-16T04:55:14.406667+0000","last_clean":"2024-09-16T04:55:14.406667+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:14.406667+0000","last_undegraded":"2024-09-16T04:55:14.406667+0000","last_fullsized":"2024-09-16T04:55:14.406667+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17270,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T04:55:13.402659+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:13.402659+0000","last_peered":"2024-09-16T04:55:13.402659+0000","last_clean":"2024-09-16T04:55:13.402659+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:13.402659+0000","last_undegraded":"2024-09-16T04:55:13.402659+0000","last_fullsized":"2024-09-16T04:55:13.402659+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.742016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17268,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":12923,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:02.358627+0000","last_change":"2024-09-16T04:41:27.900794+0000","last_active":"2024-09-16T04:55:02.358627+0000","last_peered":"2024-09-16T04:55:02.358627+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:02.358627+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16838,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.359067+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:02.359067+0000","last_peered":"2024-09-16T04:55:02.359067+0000","last_clean":"2024-09-16T04:55:02.359067+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:02.359067+0000","last_undegraded":"2024-09-16T04:55:02.359067+0000","last_fullsized":"2024-09-16T04:55:02.359067+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17190,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408911+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:14.408911+0000","last_peered":"2024-09-16T04:55:14.408911+0000","last_clean":"2024-09-16T04:55:14.408911+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:14.408911+0000","last_undegraded":"2024-09-16T04:55:14.408911+0000","last_fullsized":"2024-09-16T04:55:14.408911+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.245580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17400,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406720+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:14.406720+0000","last_peered":"2024-09-16T04:55:14.406720+0000","last_clean":"2024-09-16T04:55:14.406720+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:14.406720+0000","last_undegraded":"2024-09-16T04:55:14.406720+0000","last_fullsized":"2024-09-16T04:55:14.406720+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17560,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:55:02.358956+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:02.358956+0000","last_peered":"2024-09-16T04:55:02.358956+0000","last_clean":"2024-09-16T04:55:02.358956+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:02.358956+0000","last_undegraded":"2024-09-16T04:55:02.358956+0000","last_fullsized":"2024-09-16T04:55:02.358956+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:04.918994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17307,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T04:55:13.402769+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:13.402769+0000","last_peered":"2024-09-16T04:55:13.402769+0000","last_clean":"2024-09-16T04:55:13.402769+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:13.402769+0000","last_undegraded":"2024-09-16T04:55:13.402769+0000","last_fullsized":"2024-09-16T04:55:13.402769+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.741986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":16492,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:11.927864+0000","last_change":"2024-09-16T04:55:01.243932+0000","last_active":"2024-09-16T04:55:11.927864+0000","last_peered":"2024-09-16T04:55:11.927864+0000","last_clean":"2024-09-16T04:40:23.274621+0000","last_became_active":"2024-09-16T04:41:27.856379+0000","last_became_peered":"2024-09-16T04:41:27.856379+0000","last_unstale":"2024-09-16T04:55:11.927864+0000","last_undegraded":"2024-09-16T04:41:27.541103+0000","last_fullsized":"2024-09-16T04:41:27.483034+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12454,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.407035+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:14.407035+0000","last_peered":"2024-09-16T04:55:14.407035+0000","last_clean":"2024-09-16T04:55:14.407035+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:14.407035+0000","last_undegraded":"2024-09-16T04:55:14.407035+0000","last_fullsized":"2024-09-16T04:55:14.407035+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12668,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406548+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:14.406548+0000","last_peered":"2024-09-16T04:55:14.406548+0000","last_clean":"2024-09-16T04:55:14.406548+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:14.406548+0000","last_undegraded":"2024-09-16T04:55:14.406548+0000","last_fullsized":"2024-09-16T04:55:14.406548+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12936,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408766+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:14.408766+0000","last_peered":"2024-09-16T04:55:14.408766+0000","last_clean":"2024-09-16T04:55:14.408766+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:14.408766+0000","last_undegraded":"2024-09-16T04:55:14.408766+0000","last_fullsized":"2024-09-16T04:55:14.408766+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14974,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406670+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:14.406670+0000","last_peered":"2024-09-16T04:55:14.406670+0000","last_clean":"2024-09-16T04:55:14.406670+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:14.406670+0000","last_undegraded":"2024-09-16T04:55:14.406670+0000","last_fullsized":"2024-09-16T04:55:14.406670+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12443,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406816+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:14.406816+0000","last_peered":"2024-09-16T04:55:14.406816+0000","last_clean":"2024-09-16T04:55:14.406816+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:14.406816+0000","last_undegraded":"2024-09-16T04:55:14.406816+0000","last_fullsized":"2024-09-16T04:55:14.406816+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25610,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":512000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722172,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649668,"statfs":{"total":96636764160,"available":95897260032,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731388,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640452,"statfs":{"total":96636764160,"available":95887822848,"internally_reserved":0,"allocated":495616,"data_stored":189846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624100,"kb_used_data":976,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93747740,"statfs":{"total":96636764160,"available":95997685760,"internally_reserved":0,"allocated":999424,"data_stored":688567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":638054095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":499712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:21.754 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:22.050 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:22.050 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:22.062 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":519,"stamp":"2024-09-16T04:55:20.832872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26454,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2808140,"kb_used_data":2484,"kb_used_omap":14,"kb_used_meta":2805553,"kb_avail":374679220,"statfs":{"total":386547056640,"available":383671521280,"internally_reserved":0,"allocated":2543616,"data_stored":1279756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2872886833},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001425"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17206,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406667+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:14.406667+0000","last_peered":"2024-09-16T04:55:14.406667+0000","last_clean":"2024-09-16T04:55:14.406667+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:14.406667+0000","last_undegraded":"2024-09-16T04:55:14.406667+0000","last_fullsized":"2024-09-16T04:55:14.406667+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17272,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.405452+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:14.405452+0000","last_peered":"2024-09-16T04:55:14.405452+0000","last_clean":"2024-09-16T04:55:14.405452+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:14.405452+0000","last_undegraded":"2024-09-16T04:55:14.405452+0000","last_fullsized":"2024-09-16T04:55:14.405452+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.742016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17268,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":14120,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:16.926413+0000","last_change":"2024-09-16T04:55:12.900563+0000","last_active":"2024-09-16T04:55:16.926413+0000","last_peered":"2024-09-16T04:55:16.926413+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:16.926413+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1652}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16842,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406147+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:14.406147+0000","last_peered":"2024-09-16T04:55:14.406147+0000","last_clean":"2024-09-16T04:55:14.406147+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:14.406147+0000","last_undegraded":"2024-09-16T04:55:14.406147+0000","last_fullsized":"2024-09-16T04:55:14.406147+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17190,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408911+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:14.408911+0000","last_peered":"2024-09-16T04:55:14.408911+0000","last_clean":"2024-09-16T04:55:14.408911+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:14.408911+0000","last_undegraded":"2024-09-16T04:55:14.408911+0000","last_fullsized":"2024-09-16T04:55:14.408911+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.245580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17400,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406720+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:14.406720+0000","last_peered":"2024-09-16T04:55:14.406720+0000","last_clean":"2024-09-16T04:55:14.406720+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:14.406720+0000","last_undegraded":"2024-09-16T04:55:14.406720+0000","last_fullsized":"2024-09-16T04:55:14.406720+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17564,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406104+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:14.406104+0000","last_peered":"2024-09-16T04:55:14.406104+0000","last_clean":"2024-09-16T04:55:14.406104+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:14.406104+0000","last_undegraded":"2024-09-16T04:55:14.406104+0000","last_fullsized":"2024-09-16T04:55:14.406104+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.724511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17309,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.405618+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:14.405618+0000","last_peered":"2024-09-16T04:55:14.405618+0000","last_clean":"2024-09-16T04:55:14.405618+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:14.405618+0000","last_undegraded":"2024-09-16T04:55:14.405618+0000","last_fullsized":"2024-09-16T04:55:14.405618+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.741986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":17000,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.473593+0000","last_change":"2024-09-16T04:55:14.473593+0000","last_active":"2024-09-16T04:55:14.473593+0000","last_peered":"2024-09-16T04:55:14.473593+0000","last_clean":"2024-09-16T04:55:14.473593+0000","last_became_active":"2024-09-16T04:55:14.409783+0000","last_became_peered":"2024-09-16T04:55:14.409783+0000","last_unstale":"2024-09-16T04:55:14.473593+0000","last_undegraded":"2024-09-16T04:55:14.473593+0000","last_fullsized":"2024-09-16T04:55:14.473593+0000","mapping_epoch":60,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12454,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.407035+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:14.407035+0000","last_peered":"2024-09-16T04:55:14.407035+0000","last_clean":"2024-09-16T04:55:14.407035+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:14.407035+0000","last_undegraded":"2024-09-16T04:55:14.407035+0000","last_fullsized":"2024-09-16T04:55:14.407035+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12668,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406548+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:14.406548+0000","last_peered":"2024-09-16T04:55:14.406548+0000","last_clean":"2024-09-16T04:55:14.406548+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:14.406548+0000","last_undegraded":"2024-09-16T04:55:14.406548+0000","last_fullsized":"2024-09-16T04:55:14.406548+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12936,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408766+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:14.408766+0000","last_peered":"2024-09-16T04:55:14.408766+0000","last_clean":"2024-09-16T04:55:14.408766+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:14.408766+0000","last_undegraded":"2024-09-16T04:55:14.408766+0000","last_fullsized":"2024-09-16T04:55:14.408766+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14974,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406670+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:14.406670+0000","last_peered":"2024-09-16T04:55:14.406670+0000","last_clean":"2024-09-16T04:55:14.406670+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:14.406670+0000","last_undegraded":"2024-09-16T04:55:14.406670+0000","last_fullsized":"2024-09-16T04:55:14.406670+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12443,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406816+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:14.406816+0000","last_peered":"2024-09-16T04:55:14.406816+0000","last_clean":"2024-09-16T04:55:14.406816+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:14.406816+0000","last_undegraded":"2024-09-16T04:55:14.406816+0000","last_fullsized":"2024-09-16T04:55:14.406816+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26454,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":512000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722188,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649652,"statfs":{"total":96636764160,"available":95897243648,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731404,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640436,"statfs":{"total":96636764160,"available":95887806464,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632424,"kb_used_data":984,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739416,"statfs":{"total":96636764160,"available":95989161984,"internally_reserved":0,"allocated":1007616,"data_stored":691651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646573775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":499712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:22.063 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:22.369 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:22.369 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:22.382 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":519,"stamp":"2024-09-16T04:55:20.832872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26454,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2808140,"kb_used_data":2484,"kb_used_omap":14,"kb_used_meta":2805553,"kb_avail":374679220,"statfs":{"total":386547056640,"available":383671521280,"internally_reserved":0,"allocated":2543616,"data_stored":1279756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2872886833},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001425"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17206,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406667+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:14.406667+0000","last_peered":"2024-09-16T04:55:14.406667+0000","last_clean":"2024-09-16T04:55:14.406667+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:14.406667+0000","last_undegraded":"2024-09-16T04:55:14.406667+0000","last_fullsized":"2024-09-16T04:55:14.406667+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17272,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.405452+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:14.405452+0000","last_peered":"2024-09-16T04:55:14.405452+0000","last_clean":"2024-09-16T04:55:14.405452+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:14.405452+0000","last_undegraded":"2024-09-16T04:55:14.405452+0000","last_fullsized":"2024-09-16T04:55:14.405452+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.742016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17268,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":14120,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:16.926413+0000","last_change":"2024-09-16T04:55:12.900563+0000","last_active":"2024-09-16T04:55:16.926413+0000","last_peered":"2024-09-16T04:55:16.926413+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:16.926413+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1652}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16842,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406147+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:14.406147+0000","last_peered":"2024-09-16T04:55:14.406147+0000","last_clean":"2024-09-16T04:55:14.406147+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:14.406147+0000","last_undegraded":"2024-09-16T04:55:14.406147+0000","last_fullsized":"2024-09-16T04:55:14.406147+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17190,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408911+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:14.408911+0000","last_peered":"2024-09-16T04:55:14.408911+0000","last_clean":"2024-09-16T04:55:14.408911+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:14.408911+0000","last_undegraded":"2024-09-16T04:55:14.408911+0000","last_fullsized":"2024-09-16T04:55:14.408911+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.245580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17400,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406720+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:14.406720+0000","last_peered":"2024-09-16T04:55:14.406720+0000","last_clean":"2024-09-16T04:55:14.406720+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:14.406720+0000","last_undegraded":"2024-09-16T04:55:14.406720+0000","last_fullsized":"2024-09-16T04:55:14.406720+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17564,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406104+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:14.406104+0000","last_peered":"2024-09-16T04:55:14.406104+0000","last_clean":"2024-09-16T04:55:14.406104+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:14.406104+0000","last_undegraded":"2024-09-16T04:55:14.406104+0000","last_fullsized":"2024-09-16T04:55:14.406104+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.724511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17309,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.405618+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:14.405618+0000","last_peered":"2024-09-16T04:55:14.405618+0000","last_clean":"2024-09-16T04:55:14.405618+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:14.405618+0000","last_undegraded":"2024-09-16T04:55:14.405618+0000","last_fullsized":"2024-09-16T04:55:14.405618+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.741986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":17000,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.473593+0000","last_change":"2024-09-16T04:55:14.473593+0000","last_active":"2024-09-16T04:55:14.473593+0000","last_peered":"2024-09-16T04:55:14.473593+0000","last_clean":"2024-09-16T04:55:14.473593+0000","last_became_active":"2024-09-16T04:55:14.409783+0000","last_became_peered":"2024-09-16T04:55:14.409783+0000","last_unstale":"2024-09-16T04:55:14.473593+0000","last_undegraded":"2024-09-16T04:55:14.473593+0000","last_fullsized":"2024-09-16T04:55:14.473593+0000","mapping_epoch":60,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12454,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.407035+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:14.407035+0000","last_peered":"2024-09-16T04:55:14.407035+0000","last_clean":"2024-09-16T04:55:14.407035+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:14.407035+0000","last_undegraded":"2024-09-16T04:55:14.407035+0000","last_fullsized":"2024-09-16T04:55:14.407035+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12668,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406548+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:14.406548+0000","last_peered":"2024-09-16T04:55:14.406548+0000","last_clean":"2024-09-16T04:55:14.406548+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:14.406548+0000","last_undegraded":"2024-09-16T04:55:14.406548+0000","last_fullsized":"2024-09-16T04:55:14.406548+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12936,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408766+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:14.408766+0000","last_peered":"2024-09-16T04:55:14.408766+0000","last_clean":"2024-09-16T04:55:14.408766+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:14.408766+0000","last_undegraded":"2024-09-16T04:55:14.408766+0000","last_fullsized":"2024-09-16T04:55:14.408766+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14974,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406670+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:14.406670+0000","last_peered":"2024-09-16T04:55:14.406670+0000","last_clean":"2024-09-16T04:55:14.406670+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:14.406670+0000","last_undegraded":"2024-09-16T04:55:14.406670+0000","last_fullsized":"2024-09-16T04:55:14.406670+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12443,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406816+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:14.406816+0000","last_peered":"2024-09-16T04:55:14.406816+0000","last_clean":"2024-09-16T04:55:14.406816+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:14.406816+0000","last_undegraded":"2024-09-16T04:55:14.406816+0000","last_fullsized":"2024-09-16T04:55:14.406816+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26454,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":512000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722188,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649652,"statfs":{"total":96636764160,"available":95897243648,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731404,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640436,"statfs":{"total":96636764160,"available":95887806464,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632424,"kb_used_data":984,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739416,"statfs":{"total":96636764160,"available":95989161984,"internally_reserved":0,"allocated":1007616,"data_stored":691651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646573775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":499712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:22.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:55:22.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 14120, 'reported_epoch': 61, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:55:16.926413+0000', 'last_change': '2024-09-16T04:55:12.900563+0000', 'last_active': '2024-09-16T04:55:16.926413+0000', 'last_peered': '2024-09-16T04:55:16.926413+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:55:16.926413+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1654, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 633, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1652}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:55:22.383 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:55:22.754 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:22.754 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1654/112 objects degraded (1476.786%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":935,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2875535360,"bytes_avail":383671521280,"bytes_total":386547056640,"degraded_objects":1654,"degraded_total":112,"degraded_ratio":14.767857142857142,"recovering_objects_per_sec":105,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":844,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:55:22.768 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:55:22.768 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:23.071 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:23.072 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:23.085 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-09-16T04:55:22.833163+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27339,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2808140,"kb_used_data":2484,"kb_used_omap":14,"kb_used_meta":2805553,"kb_avail":374679220,"statfs":{"total":386547056640,"available":383671521280,"internally_reserved":0,"allocated":2543616,"data_stored":1279756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2872886833},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1729,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1729,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.432131"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17206,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406667+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:14.406667+0000","last_peered":"2024-09-16T04:55:14.406667+0000","last_clean":"2024-09-16T04:55:14.406667+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:14.406667+0000","last_undegraded":"2024-09-16T04:55:14.406667+0000","last_fullsized":"2024-09-16T04:55:14.406667+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17272,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.405452+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:14.405452+0000","last_peered":"2024-09-16T04:55:14.405452+0000","last_clean":"2024-09-16T04:55:14.405452+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:14.405452+0000","last_undegraded":"2024-09-16T04:55:14.405452+0000","last_fullsized":"2024-09-16T04:55:14.405452+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.742016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17268,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":15890,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:21.929107+0000","last_change":"2024-09-16T04:55:12.900563+0000","last_active":"2024-09-16T04:55:21.929107+0000","last_peered":"2024-09-16T04:55:21.929107+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:21.929107+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1518,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-767}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16842,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406147+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:14.406147+0000","last_peered":"2024-09-16T04:55:14.406147+0000","last_clean":"2024-09-16T04:55:14.406147+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:14.406147+0000","last_undegraded":"2024-09-16T04:55:14.406147+0000","last_fullsized":"2024-09-16T04:55:14.406147+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17190,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408911+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:14.408911+0000","last_peered":"2024-09-16T04:55:14.408911+0000","last_clean":"2024-09-16T04:55:14.408911+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:14.408911+0000","last_undegraded":"2024-09-16T04:55:14.408911+0000","last_fullsized":"2024-09-16T04:55:14.408911+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.245580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17400,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406720+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:14.406720+0000","last_peered":"2024-09-16T04:55:14.406720+0000","last_clean":"2024-09-16T04:55:14.406720+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:14.406720+0000","last_undegraded":"2024-09-16T04:55:14.406720+0000","last_fullsized":"2024-09-16T04:55:14.406720+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17564,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406104+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:14.406104+0000","last_peered":"2024-09-16T04:55:14.406104+0000","last_clean":"2024-09-16T04:55:14.406104+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:14.406104+0000","last_undegraded":"2024-09-16T04:55:14.406104+0000","last_fullsized":"2024-09-16T04:55:14.406104+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.724511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17309,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.405618+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:14.405618+0000","last_peered":"2024-09-16T04:55:14.405618+0000","last_clean":"2024-09-16T04:55:14.405618+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:14.405618+0000","last_undegraded":"2024-09-16T04:55:14.405618+0000","last_fullsized":"2024-09-16T04:55:14.405618+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.741986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":17000,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.473593+0000","last_change":"2024-09-16T04:55:14.473593+0000","last_active":"2024-09-16T04:55:14.473593+0000","last_peered":"2024-09-16T04:55:14.473593+0000","last_clean":"2024-09-16T04:55:14.473593+0000","last_became_active":"2024-09-16T04:55:14.409783+0000","last_became_peered":"2024-09-16T04:55:14.409783+0000","last_unstale":"2024-09-16T04:55:14.473593+0000","last_undegraded":"2024-09-16T04:55:14.473593+0000","last_fullsized":"2024-09-16T04:55:14.473593+0000","mapping_epoch":60,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12454,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.407035+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:14.407035+0000","last_peered":"2024-09-16T04:55:14.407035+0000","last_clean":"2024-09-16T04:55:14.407035+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:14.407035+0000","last_undegraded":"2024-09-16T04:55:14.407035+0000","last_fullsized":"2024-09-16T04:55:14.407035+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12668,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406548+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:14.406548+0000","last_peered":"2024-09-16T04:55:14.406548+0000","last_clean":"2024-09-16T04:55:14.406548+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:14.406548+0000","last_undegraded":"2024-09-16T04:55:14.406548+0000","last_fullsized":"2024-09-16T04:55:14.406548+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12936,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408766+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:14.408766+0000","last_peered":"2024-09-16T04:55:14.408766+0000","last_clean":"2024-09-16T04:55:14.408766+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:14.408766+0000","last_undegraded":"2024-09-16T04:55:14.408766+0000","last_fullsized":"2024-09-16T04:55:14.408766+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14974,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406670+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:14.406670+0000","last_peered":"2024-09-16T04:55:14.406670+0000","last_clean":"2024-09-16T04:55:14.406670+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:14.406670+0000","last_undegraded":"2024-09-16T04:55:14.406670+0000","last_fullsized":"2024-09-16T04:55:14.406670+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12443,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406816+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:14.406816+0000","last_peered":"2024-09-16T04:55:14.406816+0000","last_clean":"2024-09-16T04:55:14.406816+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:14.406816+0000","last_undegraded":"2024-09-16T04:55:14.406816+0000","last_fullsized":"2024-09-16T04:55:14.406816+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27339,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":512000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722188,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649652,"statfs":{"total":96636764160,"available":95897243648,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731404,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640436,"statfs":{"total":96636764160,"available":95887806464,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632424,"kb_used_data":984,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93739416,"statfs":{"total":96636764160,"available":95989161984,"internally_reserved":0,"allocated":1007616,"data_stored":691651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":646573775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":499712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:26.087 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:26.373 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:26.373 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:26.385 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2024-09-16T04:55:25.435426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27339,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799820,"kb_used_data":2484,"kb_used_omap":14,"kb_used_meta":2797233,"kb_avail":374687540,"statfs":{"total":386547056640,"available":383680040960,"internally_reserved":0,"allocated":2543616,"data_stored":1279756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2864367153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1729,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1729,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.603636"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17206,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406667+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:14.406667+0000","last_peered":"2024-09-16T04:55:14.406667+0000","last_clean":"2024-09-16T04:55:14.406667+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:14.406667+0000","last_undegraded":"2024-09-16T04:55:14.406667+0000","last_fullsized":"2024-09-16T04:55:14.406667+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17272,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.405452+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:14.405452+0000","last_peered":"2024-09-16T04:55:14.405452+0000","last_clean":"2024-09-16T04:55:14.405452+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:14.405452+0000","last_undegraded":"2024-09-16T04:55:14.405452+0000","last_fullsized":"2024-09-16T04:55:14.405452+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.742016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17268,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":15890,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:21.929107+0000","last_change":"2024-09-16T04:55:12.900563+0000","last_active":"2024-09-16T04:55:21.929107+0000","last_peered":"2024-09-16T04:55:21.929107+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:21.929107+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1518,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-767}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16842,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406147+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:14.406147+0000","last_peered":"2024-09-16T04:55:14.406147+0000","last_clean":"2024-09-16T04:55:14.406147+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:14.406147+0000","last_undegraded":"2024-09-16T04:55:14.406147+0000","last_fullsized":"2024-09-16T04:55:14.406147+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17190,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408911+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:14.408911+0000","last_peered":"2024-09-16T04:55:14.408911+0000","last_clean":"2024-09-16T04:55:14.408911+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:14.408911+0000","last_undegraded":"2024-09-16T04:55:14.408911+0000","last_fullsized":"2024-09-16T04:55:14.408911+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.245580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17400,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406720+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:14.406720+0000","last_peered":"2024-09-16T04:55:14.406720+0000","last_clean":"2024-09-16T04:55:14.406720+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:14.406720+0000","last_undegraded":"2024-09-16T04:55:14.406720+0000","last_fullsized":"2024-09-16T04:55:14.406720+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17564,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406104+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:14.406104+0000","last_peered":"2024-09-16T04:55:14.406104+0000","last_clean":"2024-09-16T04:55:14.406104+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:14.406104+0000","last_undegraded":"2024-09-16T04:55:14.406104+0000","last_fullsized":"2024-09-16T04:55:14.406104+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.724511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17309,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.405618+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:14.405618+0000","last_peered":"2024-09-16T04:55:14.405618+0000","last_clean":"2024-09-16T04:55:14.405618+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:14.405618+0000","last_undegraded":"2024-09-16T04:55:14.405618+0000","last_fullsized":"2024-09-16T04:55:14.405618+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.741986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":17000,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.473593+0000","last_change":"2024-09-16T04:55:14.473593+0000","last_active":"2024-09-16T04:55:14.473593+0000","last_peered":"2024-09-16T04:55:14.473593+0000","last_clean":"2024-09-16T04:55:14.473593+0000","last_became_active":"2024-09-16T04:55:14.409783+0000","last_became_peered":"2024-09-16T04:55:14.409783+0000","last_unstale":"2024-09-16T04:55:14.473593+0000","last_undegraded":"2024-09-16T04:55:14.473593+0000","last_fullsized":"2024-09-16T04:55:14.473593+0000","mapping_epoch":60,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12454,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.407035+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:14.407035+0000","last_peered":"2024-09-16T04:55:14.407035+0000","last_clean":"2024-09-16T04:55:14.407035+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:14.407035+0000","last_undegraded":"2024-09-16T04:55:14.407035+0000","last_fullsized":"2024-09-16T04:55:14.407035+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12668,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406548+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:14.406548+0000","last_peered":"2024-09-16T04:55:14.406548+0000","last_clean":"2024-09-16T04:55:14.406548+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:14.406548+0000","last_undegraded":"2024-09-16T04:55:14.406548+0000","last_fullsized":"2024-09-16T04:55:14.406548+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12936,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408766+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:14.408766+0000","last_peered":"2024-09-16T04:55:14.408766+0000","last_clean":"2024-09-16T04:55:14.408766+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:14.408766+0000","last_undegraded":"2024-09-16T04:55:14.408766+0000","last_fullsized":"2024-09-16T04:55:14.408766+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14974,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406670+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:14.406670+0000","last_peered":"2024-09-16T04:55:14.406670+0000","last_clean":"2024-09-16T04:55:14.406670+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:14.406670+0000","last_undegraded":"2024-09-16T04:55:14.406670+0000","last_fullsized":"2024-09-16T04:55:14.406670+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12443,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406816+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:14.406816+0000","last_peered":"2024-09-16T04:55:14.406816+0000","last_clean":"2024-09-16T04:55:14.406816+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:14.406816+0000","last_undegraded":"2024-09-16T04:55:14.406816+0000","last_fullsized":"2024-09-16T04:55:14.406816+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27339,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":512000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722188,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649652,"statfs":{"total":96636764160,"available":95897243648,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731404,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640436,"statfs":{"total":96636764160,"available":95887806464,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624104,"kb_used_data":984,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93747736,"statfs":{"total":96636764160,"available":95997681664,"internally_reserved":0,"allocated":1007616,"data_stored":691651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":638054095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":499712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:26.386 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:26.676 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:26.677 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:26.691 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2024-09-16T04:55:26.441583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27339,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799820,"kb_used_data":2484,"kb_used_omap":14,"kb_used_meta":2797233,"kb_avail":374687540,"statfs":{"total":386547056640,"available":383680040960,"internally_reserved":0,"allocated":2543616,"data_stored":1279756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2864367153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1729,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1729,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.609384"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17206,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406667+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:14.406667+0000","last_peered":"2024-09-16T04:55:14.406667+0000","last_clean":"2024-09-16T04:55:14.406667+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:14.406667+0000","last_undegraded":"2024-09-16T04:55:14.406667+0000","last_fullsized":"2024-09-16T04:55:14.406667+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17272,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.405452+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:14.405452+0000","last_peered":"2024-09-16T04:55:14.405452+0000","last_clean":"2024-09-16T04:55:14.405452+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:14.405452+0000","last_undegraded":"2024-09-16T04:55:14.405452+0000","last_fullsized":"2024-09-16T04:55:14.405452+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.742016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17268,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":15890,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:21.929107+0000","last_change":"2024-09-16T04:55:12.900563+0000","last_active":"2024-09-16T04:55:21.929107+0000","last_peered":"2024-09-16T04:55:21.929107+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:21.929107+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1518,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-767}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16842,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406147+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:14.406147+0000","last_peered":"2024-09-16T04:55:14.406147+0000","last_clean":"2024-09-16T04:55:14.406147+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:14.406147+0000","last_undegraded":"2024-09-16T04:55:14.406147+0000","last_fullsized":"2024-09-16T04:55:14.406147+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17192,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T04:55:25.442654+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:25.442654+0000","last_peered":"2024-09-16T04:55:25.442654+0000","last_clean":"2024-09-16T04:55:25.442654+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:25.442654+0000","last_undegraded":"2024-09-16T04:55:25.442654+0000","last_fullsized":"2024-09-16T04:55:25.442654+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17400,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406720+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:14.406720+0000","last_peered":"2024-09-16T04:55:14.406720+0000","last_clean":"2024-09-16T04:55:14.406720+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:14.406720+0000","last_undegraded":"2024-09-16T04:55:14.406720+0000","last_fullsized":"2024-09-16T04:55:14.406720+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17564,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406104+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:14.406104+0000","last_peered":"2024-09-16T04:55:14.406104+0000","last_clean":"2024-09-16T04:55:14.406104+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:14.406104+0000","last_undegraded":"2024-09-16T04:55:14.406104+0000","last_fullsized":"2024-09-16T04:55:14.406104+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.724511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17309,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.405618+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:14.405618+0000","last_peered":"2024-09-16T04:55:14.405618+0000","last_clean":"2024-09-16T04:55:14.405618+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:14.405618+0000","last_undegraded":"2024-09-16T04:55:14.405618+0000","last_fullsized":"2024-09-16T04:55:14.405618+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.741986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":17000,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.473593+0000","last_change":"2024-09-16T04:55:14.473593+0000","last_active":"2024-09-16T04:55:14.473593+0000","last_peered":"2024-09-16T04:55:14.473593+0000","last_clean":"2024-09-16T04:55:14.473593+0000","last_became_active":"2024-09-16T04:55:14.409783+0000","last_became_peered":"2024-09-16T04:55:14.409783+0000","last_unstale":"2024-09-16T04:55:14.473593+0000","last_undegraded":"2024-09-16T04:55:14.473593+0000","last_fullsized":"2024-09-16T04:55:14.473593+0000","mapping_epoch":60,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12454,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.407035+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:14.407035+0000","last_peered":"2024-09-16T04:55:14.407035+0000","last_clean":"2024-09-16T04:55:14.407035+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:14.407035+0000","last_undegraded":"2024-09-16T04:55:14.407035+0000","last_fullsized":"2024-09-16T04:55:14.407035+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12668,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406548+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:14.406548+0000","last_peered":"2024-09-16T04:55:14.406548+0000","last_clean":"2024-09-16T04:55:14.406548+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:14.406548+0000","last_undegraded":"2024-09-16T04:55:14.406548+0000","last_fullsized":"2024-09-16T04:55:14.406548+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12936,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408766+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:14.408766+0000","last_peered":"2024-09-16T04:55:14.408766+0000","last_clean":"2024-09-16T04:55:14.408766+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:14.408766+0000","last_undegraded":"2024-09-16T04:55:14.408766+0000","last_fullsized":"2024-09-16T04:55:14.408766+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14974,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406670+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:14.406670+0000","last_peered":"2024-09-16T04:55:14.406670+0000","last_clean":"2024-09-16T04:55:14.406670+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:14.406670+0000","last_undegraded":"2024-09-16T04:55:14.406670+0000","last_fullsized":"2024-09-16T04:55:14.406670+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12443,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406816+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:14.406816+0000","last_peered":"2024-09-16T04:55:14.406816+0000","last_clean":"2024-09-16T04:55:14.406816+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:14.406816+0000","last_undegraded":"2024-09-16T04:55:14.406816+0000","last_fullsized":"2024-09-16T04:55:14.406816+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27339,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":512000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722188,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649652,"statfs":{"total":96636764160,"available":95897243648,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731404,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640436,"statfs":{"total":96636764160,"available":95887806464,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624104,"kb_used_data":984,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93747736,"statfs":{"total":96636764160,"available":95997681664,"internally_reserved":0,"allocated":1007616,"data_stored":691651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":638054095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":499712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:26.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-16T04:55:26.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12739", 'reported_seq': 15890, 'reported_epoch': 61, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T04:55:21.929107+0000', 'last_change': '2024-09-16T04:55:12.900563+0000', 'last_active': '2024-09-16T04:55:21.929107+0000', 'last_peered': '2024-09-16T04:55:21.929107+0000', 'last_clean': '2024-09-16T04:40:22.402437+0000', 'last_became_active': '2024-09-16T04:41:27.881637+0000', 'last_became_peered': '2024-09-16T04:41:27.881637+0000', 'last_unstale': '2024-09-16T04:55:21.929107+0000', 'last_undegraded': '2024-09-16T04:41:27.558514+0000', 'last_fullsized': '2024-09-16T04:41:27.482650+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'last_clean_scrub_stamp': '2024-09-16T04:40:11.606968+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 769, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12737, 'num_write_kb': 50957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1518, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -767}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T04:55:26.693 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T04:55:27.032 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:27.032 INFO:teuthology.orchestra.run.smithi152.stdout:{"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 769/112 objects degraded (686.607%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":939,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":63,"num_osds":4,"num_up_osds":4,"osd_up_since":1726461686,"num_in_osds":4,"osd_in_since":1726461589,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":2867015680,"bytes_avail":383680040960,"bytes_total":386547056640,"degraded_objects":769,"degraded_total":112,"degraded_ratio":6.8660714285714288,"recovering_objects_per_sec":165,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1729,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T04:39:47:671911+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T04:48:18.702612+0000","services":{}},"progress_events":{}} 2024-09-16T04:55:27.044 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-16T04:55:27.045 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:27.317 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:27.318 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:27.330 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-09-16T04:55:26.833800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27339,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799820,"kb_used_data":2484,"kb_used_omap":14,"kb_used_meta":2797233,"kb_avail":374687540,"statfs":{"total":386547056640,"available":383680040960,"internally_reserved":0,"allocated":2543616,"data_stored":1279756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2864367153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-885,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":885,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001228"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17206,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406667+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:14.406667+0000","last_peered":"2024-09-16T04:55:14.406667+0000","last_clean":"2024-09-16T04:55:14.406667+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:14.406667+0000","last_undegraded":"2024-09-16T04:55:14.406667+0000","last_fullsized":"2024-09-16T04:55:14.406667+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17272,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.405452+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:14.405452+0000","last_peered":"2024-09-16T04:55:14.405452+0000","last_clean":"2024-09-16T04:55:14.405452+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:14.405452+0000","last_undegraded":"2024-09-16T04:55:14.405452+0000","last_fullsized":"2024-09-16T04:55:14.405452+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.742016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17268,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":15890,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T04:55:21.929107+0000","last_change":"2024-09-16T04:55:12.900563+0000","last_active":"2024-09-16T04:55:21.929107+0000","last_peered":"2024-09-16T04:55:21.929107+0000","last_clean":"2024-09-16T04:40:22.402437+0000","last_became_active":"2024-09-16T04:41:27.881637+0000","last_became_peered":"2024-09-16T04:41:27.881637+0000","last_unstale":"2024-09-16T04:55:21.929107+0000","last_undegraded":"2024-09-16T04:41:27.558514+0000","last_fullsized":"2024-09-16T04:41:27.482650+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":50957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1518,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-767}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16842,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406147+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:14.406147+0000","last_peered":"2024-09-16T04:55:14.406147+0000","last_clean":"2024-09-16T04:55:14.406147+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:14.406147+0000","last_undegraded":"2024-09-16T04:55:14.406147+0000","last_fullsized":"2024-09-16T04:55:14.406147+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17192,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T04:55:25.442654+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:25.442654+0000","last_peered":"2024-09-16T04:55:25.442654+0000","last_clean":"2024-09-16T04:55:25.442654+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:25.442654+0000","last_undegraded":"2024-09-16T04:55:25.442654+0000","last_fullsized":"2024-09-16T04:55:25.442654+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17400,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406720+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:14.406720+0000","last_peered":"2024-09-16T04:55:14.406720+0000","last_clean":"2024-09-16T04:55:14.406720+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:14.406720+0000","last_undegraded":"2024-09-16T04:55:14.406720+0000","last_fullsized":"2024-09-16T04:55:14.406720+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17564,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406104+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:14.406104+0000","last_peered":"2024-09-16T04:55:14.406104+0000","last_clean":"2024-09-16T04:55:14.406104+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:14.406104+0000","last_undegraded":"2024-09-16T04:55:14.406104+0000","last_fullsized":"2024-09-16T04:55:14.406104+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.724511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17309,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.405618+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:14.405618+0000","last_peered":"2024-09-16T04:55:14.405618+0000","last_clean":"2024-09-16T04:55:14.405618+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:14.405618+0000","last_undegraded":"2024-09-16T04:55:14.405618+0000","last_fullsized":"2024-09-16T04:55:14.405618+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.741986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":17000,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.473593+0000","last_change":"2024-09-16T04:55:14.473593+0000","last_active":"2024-09-16T04:55:14.473593+0000","last_peered":"2024-09-16T04:55:14.473593+0000","last_clean":"2024-09-16T04:55:14.473593+0000","last_became_active":"2024-09-16T04:55:14.409783+0000","last_became_peered":"2024-09-16T04:55:14.409783+0000","last_unstale":"2024-09-16T04:55:14.473593+0000","last_undegraded":"2024-09-16T04:55:14.473593+0000","last_fullsized":"2024-09-16T04:55:14.473593+0000","mapping_epoch":60,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":49948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12454,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.407035+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:14.407035+0000","last_peered":"2024-09-16T04:55:14.407035+0000","last_clean":"2024-09-16T04:55:14.407035+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:14.407035+0000","last_undegraded":"2024-09-16T04:55:14.407035+0000","last_fullsized":"2024-09-16T04:55:14.407035+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12668,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406548+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:14.406548+0000","last_peered":"2024-09-16T04:55:14.406548+0000","last_clean":"2024-09-16T04:55:14.406548+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:14.406548+0000","last_undegraded":"2024-09-16T04:55:14.406548+0000","last_fullsized":"2024-09-16T04:55:14.406548+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12936,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408766+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:14.408766+0000","last_peered":"2024-09-16T04:55:14.408766+0000","last_clean":"2024-09-16T04:55:14.408766+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:14.408766+0000","last_undegraded":"2024-09-16T04:55:14.408766+0000","last_fullsized":"2024-09-16T04:55:14.408766+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14974,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406670+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:14.406670+0000","last_peered":"2024-09-16T04:55:14.406670+0000","last_clean":"2024-09-16T04:55:14.406670+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:14.406670+0000","last_undegraded":"2024-09-16T04:55:14.406670+0000","last_fullsized":"2024-09-16T04:55:14.406670+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12443,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406816+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:14.406816+0000","last_peered":"2024-09-16T04:55:14.406816+0000","last_clean":"2024-09-16T04:55:14.406816+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:14.406816+0000","last_undegraded":"2024-09-16T04:55:14.406816+0000","last_fullsized":"2024-09-16T04:55:14.406816+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201459,"num_write_kb":805931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27339,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":512000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722188,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649652,"statfs":{"total":96636764160,"available":95897243648,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731404,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640436,"statfs":{"total":96636764160,"available":95887806464,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624104,"kb_used_data":984,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93747736,"statfs":{"total":96636764160,"available":95997681664,"internally_reserved":0,"allocated":1007616,"data_stored":691651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":638054095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":499712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:27.695 INFO:tasks.ceph.osd.1.smithi152.stderr:2024-09-16T04:55:27.694+0000 7f4f8e433640 -1 osd.1 pg_epoch: 63 pg[1.4s3( v 37'12752 (31'2700,37'12752] local-lis/les=34/35 n=4 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [3,0,2,1]p3(0) r=3 lpr=34 luod=0'0 crt=37'12752 mlcod 37'12752 active mbc={}] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-16T04:55:30.331 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:30.630 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:30.631 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:30.642 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":525,"stamp":"2024-09-16T04:55:28.834179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":156170,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176237,"num_write_kb":705026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23626,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799852,"kb_used_data":2516,"kb_used_omap":14,"kb_used_meta":2797233,"kb_avail":374687508,"statfs":{"total":386547056640,"available":383680008192,"internally_reserved":0,"allocated":2576384,"data_stored":1291986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2864367153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33950,"num_objects":-5,"num_object_clones":0,"num_object_copies":-20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-25222,"num_write_kb":-100905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2828,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001307"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17210,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444531+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:26.444531+0000","last_peered":"2024-09-16T04:55:26.444531+0000","last_clean":"2024-09-16T04:55:26.444531+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:26.444531+0000","last_undegraded":"2024-09-16T04:55:26.444531+0000","last_fullsized":"2024-09-16T04:55:26.444531+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406667+0000","last_change":"2024-09-16T04:53:40.146094+0000","last_active":"2024-09-16T04:55:14.406667+0000","last_peered":"2024-09-16T04:55:14.406667+0000","last_clean":"2024-09-16T04:55:14.406667+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:14.406667+0000","last_undegraded":"2024-09-16T04:55:14.406667+0000","last_fullsized":"2024-09-16T04:55:14.406667+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":50692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17276,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443422+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:26.443422+0000","last_peered":"2024-09-16T04:55:26.443422+0000","last_clean":"2024-09-16T04:55:26.443422+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:26.443422+0000","last_undegraded":"2024-09-16T04:55:26.443422+0000","last_fullsized":"2024-09-16T04:55:26.443422+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.633529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17268,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406433+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:14.406433+0000","last_peered":"2024-09-16T04:55:14.406433+0000","last_clean":"2024-09-16T04:55:14.406433+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:14.406433+0000","last_undegraded":"2024-09-16T04:55:14.406433+0000","last_fullsized":"2024-09-16T04:55:14.406433+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":17448,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.684639+0000","last_change":"2024-09-16T04:55:26.684663+0000","last_active":"2024-09-16T04:55:26.684639+0000","last_peered":"2024-09-16T04:55:26.684639+0000","last_clean":"2024-09-16T04:55:26.684639+0000","last_became_active":"2024-09-16T04:55:26.446437+0000","last_became_peered":"2024-09-16T04:55:26.446437+0000","last_unstale":"2024-09-16T04:55:26.684639+0000","last_undegraded":"2024-09-16T04:55:26.684639+0000","last_fullsized":"2024-09-16T04:55:26.684639+0000","mapping_epoch":62,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12739","last_scrub_stamp":"2024-09-16T04:55:26.684554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:26.684554+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:29:17.114250+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16846,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443392+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:26.443392+0000","last_peered":"2024-09-16T04:55:26.443392+0000","last_clean":"2024-09-16T04:55:26.443392+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:26.443392+0000","last_undegraded":"2024-09-16T04:55:26.443392+0000","last_fullsized":"2024-09-16T04:55:26.443392+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17192,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T04:55:25.442654+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:25.442654+0000","last_peered":"2024-09-16T04:55:25.442654+0000","last_clean":"2024-09-16T04:55:25.442654+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:25.442654+0000","last_undegraded":"2024-09-16T04:55:25.442654+0000","last_fullsized":"2024-09-16T04:55:25.442654+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17400,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406720+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:14.406720+0000","last_peered":"2024-09-16T04:55:14.406720+0000","last_clean":"2024-09-16T04:55:14.406720+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:14.406720+0000","last_undegraded":"2024-09-16T04:55:14.406720+0000","last_fullsized":"2024-09-16T04:55:14.406720+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17569,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443342+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:26.443342+0000","last_peered":"2024-09-16T04:55:26.443342+0000","last_clean":"2024-09-16T04:55:26.443342+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:26.443342+0000","last_undegraded":"2024-09-16T04:55:26.443342+0000","last_fullsized":"2024-09-16T04:55:26.443342+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:27.665374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17313,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443527+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:26.443527+0000","last_peered":"2024-09-16T04:55:26.443527+0000","last_clean":"2024-09-16T04:55:26.443527+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:26.443527+0000","last_undegraded":"2024-09-16T04:55:26.443527+0000","last_fullsized":"2024-09-16T04:55:26.443527+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.633504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":17012,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443089+0000","last_change":"2024-09-16T04:55:25.698837+0000","last_active":"2024-09-16T04:55:26.443089+0000","last_peered":"2024-09-16T04:55:26.443089+0000","last_clean":"2024-09-16T04:55:26.443089+0000","last_became_active":"2024-09-16T04:55:14.409783+0000","last_became_peered":"2024-09-16T04:55:14.409783+0000","last_unstale":"2024-09-16T04:55:26.443089+0000","last_undegraded":"2024-09-16T04:55:26.443089+0000","last_fullsized":"2024-09-16T04:55:26.443089+0000","mapping_epoch":60,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12488","last_scrub_stamp":"2024-09-16T04:55:25.698717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:25.698717+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:52.582394+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12454,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.407035+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:14.407035+0000","last_peered":"2024-09-16T04:55:14.407035+0000","last_clean":"2024-09-16T04:55:14.407035+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:14.407035+0000","last_undegraded":"2024-09-16T04:55:14.407035+0000","last_fullsized":"2024-09-16T04:55:14.407035+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12668,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406548+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:14.406548+0000","last_peered":"2024-09-16T04:55:14.406548+0000","last_clean":"2024-09-16T04:55:14.406548+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:14.406548+0000","last_undegraded":"2024-09-16T04:55:14.406548+0000","last_fullsized":"2024-09-16T04:55:14.406548+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12936,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.408766+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:14.408766+0000","last_peered":"2024-09-16T04:55:14.408766+0000","last_clean":"2024-09-16T04:55:14.408766+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:14.408766+0000","last_undegraded":"2024-09-16T04:55:14.408766+0000","last_fullsized":"2024-09-16T04:55:14.408766+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14974,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406670+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:14.406670+0000","last_peered":"2024-09-16T04:55:14.406670+0000","last_clean":"2024-09-16T04:55:14.406670+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:14.406670+0000","last_undegraded":"2024-09-16T04:55:14.406670+0000","last_fullsized":"2024-09-16T04:55:14.406670+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:17.988132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12443,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T04:55:14.406816+0000","last_change":"2024-09-16T04:54:03.225269+0000","last_active":"2024-09-16T04:55:14.406816+0000","last_peered":"2024-09-16T04:55:14.406816+0000","last_clean":"2024-09-16T04:55:14.406816+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:14.406816+0000","last_undegraded":"2024-09-16T04:55:14.406816+0000","last_fullsized":"2024-09-16T04:55:14.406816+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:15.913855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12303,"num_write_kb":49220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":156170,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176237,"num_write_kb":705026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23626,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":512000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722204,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649636,"statfs":{"total":96636764160,"available":95897227264,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731420,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640420,"statfs":{"total":96636764160,"available":95887790080,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624104,"kb_used_data":984,"kb_used_omap":4,"kb_used_meta":623099,"kb_avail":93747736,"statfs":{"total":96636764160,"available":95997681664,"internally_reserved":0,"allocated":1007616,"data_stored":691651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":638054095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":499712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:30.644 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-16T04:55:30.644 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-16T04:55:30.644 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:30.920 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:30.921 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:30.933 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-16T04:55:30.834665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":129010,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151264,"num_write_kb":605114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19168,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803544,"kb_used_data":2048,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683816,"statfs":{"total":386547056640,"available":383676227584,"internally_reserved":0,"allocated":2097152,"data_stored":802485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-61110,"num_objects":-9,"num_object_clones":0,"num_object_copies":-36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-50195,"num_write_kb":-200817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001502"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17210,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444531+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:26.444531+0000","last_peered":"2024-09-16T04:55:26.444531+0000","last_clean":"2024-09-16T04:55:26.444531+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:26.444531+0000","last_undegraded":"2024-09-16T04:55:26.444531+0000","last_fullsized":"2024-09-16T04:55:26.444531+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12824,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:28.031628+0000","last_change":"2024-09-16T04:55:28.031672+0000","last_active":"2024-09-16T04:55:28.031628+0000","last_peered":"2024-09-16T04:55:28.031628+0000","last_clean":"2024-09-16T04:55:28.031628+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:28.031628+0000","last_undegraded":"2024-09-16T04:55:28.031628+0000","last_fullsized":"2024-09-16T04:55:28.031628+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12672","last_scrub_stamp":"2024-09-16T04:55:28.031507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:28.031507+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:06.569559+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17276,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443422+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:26.443422+0000","last_peered":"2024-09-16T04:55:26.443422+0000","last_clean":"2024-09-16T04:55:26.443422+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:26.443422+0000","last_undegraded":"2024-09-16T04:55:26.443422+0000","last_fullsized":"2024-09-16T04:55:26.443422+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.633529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17272,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443911+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:26.443911+0000","last_peered":"2024-09-16T04:55:26.443911+0000","last_clean":"2024-09-16T04:55:26.443911+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:26.443911+0000","last_undegraded":"2024-09-16T04:55:26.443911+0000","last_fullsized":"2024-09-16T04:55:26.443911+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":17448,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.684639+0000","last_change":"2024-09-16T04:55:26.684663+0000","last_active":"2024-09-16T04:55:26.684639+0000","last_peered":"2024-09-16T04:55:26.684639+0000","last_clean":"2024-09-16T04:55:26.684639+0000","last_became_active":"2024-09-16T04:55:26.446437+0000","last_became_peered":"2024-09-16T04:55:26.446437+0000","last_unstale":"2024-09-16T04:55:26.684639+0000","last_undegraded":"2024-09-16T04:55:26.684639+0000","last_fullsized":"2024-09-16T04:55:26.684639+0000","mapping_epoch":62,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12739","last_scrub_stamp":"2024-09-16T04:55:26.684554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:26.684554+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:29:17.114250+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16846,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443392+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:26.443392+0000","last_peered":"2024-09-16T04:55:26.443392+0000","last_clean":"2024-09-16T04:55:26.443392+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:26.443392+0000","last_undegraded":"2024-09-16T04:55:26.443392+0000","last_fullsized":"2024-09-16T04:55:26.443392+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17192,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T04:55:25.442654+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:25.442654+0000","last_peered":"2024-09-16T04:55:25.442654+0000","last_clean":"2024-09-16T04:55:25.442654+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:25.442654+0000","last_undegraded":"2024-09-16T04:55:25.442654+0000","last_fullsized":"2024-09-16T04:55:25.442654+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17404,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444330+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:26.444330+0000","last_peered":"2024-09-16T04:55:26.444330+0000","last_clean":"2024-09-16T04:55:26.444330+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:26.444330+0000","last_undegraded":"2024-09-16T04:55:26.444330+0000","last_fullsized":"2024-09-16T04:55:26.444330+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17569,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443342+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:26.443342+0000","last_peered":"2024-09-16T04:55:26.443342+0000","last_clean":"2024-09-16T04:55:26.443342+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:26.443342+0000","last_undegraded":"2024-09-16T04:55:26.443342+0000","last_fullsized":"2024-09-16T04:55:26.443342+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:27.665374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17313,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443527+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:26.443527+0000","last_peered":"2024-09-16T04:55:26.443527+0000","last_clean":"2024-09-16T04:55:26.443527+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:26.443527+0000","last_undegraded":"2024-09-16T04:55:26.443527+0000","last_fullsized":"2024-09-16T04:55:26.443527+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.633504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":17012,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443089+0000","last_change":"2024-09-16T04:55:25.698837+0000","last_active":"2024-09-16T04:55:26.443089+0000","last_peered":"2024-09-16T04:55:26.443089+0000","last_clean":"2024-09-16T04:55:26.443089+0000","last_became_active":"2024-09-16T04:55:14.409783+0000","last_became_peered":"2024-09-16T04:55:14.409783+0000","last_unstale":"2024-09-16T04:55:26.443089+0000","last_undegraded":"2024-09-16T04:55:26.443089+0000","last_fullsized":"2024-09-16T04:55:26.443089+0000","mapping_epoch":60,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12488","last_scrub_stamp":"2024-09-16T04:55:25.698717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:25.698717+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:52.582394+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12458,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444644+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:26.444644+0000","last_peered":"2024-09-16T04:55:26.444644+0000","last_clean":"2024-09-16T04:55:26.444644+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:26.444644+0000","last_undegraded":"2024-09-16T04:55:26.444644+0000","last_fullsized":"2024-09-16T04:55:26.444644+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12672,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444025+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:26.444025+0000","last_peered":"2024-09-16T04:55:26.444025+0000","last_clean":"2024-09-16T04:55:26.444025+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:26.444025+0000","last_undegraded":"2024-09-16T04:55:26.444025+0000","last_fullsized":"2024-09-16T04:55:26.444025+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12940,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444666+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:26.444666+0000","last_peered":"2024-09-16T04:55:26.444666+0000","last_clean":"2024-09-16T04:55:26.444666+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:26.444666+0000","last_undegraded":"2024-09-16T04:55:26.444666+0000","last_fullsized":"2024-09-16T04:55:26.444666+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14978,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444208+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:26.444208+0000","last_peered":"2024-09-16T04:55:26.444208+0000","last_clean":"2024-09-16T04:55:26.444208+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:26.444208+0000","last_undegraded":"2024-09-16T04:55:26.444208+0000","last_fullsized":"2024-09-16T04:55:26.444208+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12455,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:28.998608+0000","last_change":"2024-09-16T04:55:28.998653+0000","last_active":"2024-09-16T04:55:28.998608+0000","last_peered":"2024-09-16T04:55:28.998608+0000","last_clean":"2024-09-16T04:55:28.998608+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:28.998608+0000","last_undegraded":"2024-09-16T04:55:28.998608+0000","last_fullsized":"2024-09-16T04:55:28.998608+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12305","last_scrub_stamp":"2024-09-16T04:55:28.998471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:28.998471+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:53.082759+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":129010,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151264,"num_write_kb":605114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19168,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722204,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649636,"statfs":{"total":96636764160,"available":95897227264,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731420,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640420,"statfs":{"total":96636764160,"available":95887790080,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627796,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93744044,"statfs":{"total":96636764160,"available":95993901056,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:30.934 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:31.211 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:31.211 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:31.223 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-16T04:55:30.834665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":129010,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151264,"num_write_kb":605114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19168,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803544,"kb_used_data":2048,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683816,"statfs":{"total":386547056640,"available":383676227584,"internally_reserved":0,"allocated":2097152,"data_stored":802485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-61110,"num_objects":-9,"num_object_clones":0,"num_object_copies":-36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-50195,"num_write_kb":-200817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001502"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17210,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444531+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:26.444531+0000","last_peered":"2024-09-16T04:55:26.444531+0000","last_clean":"2024-09-16T04:55:26.444531+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:26.444531+0000","last_undegraded":"2024-09-16T04:55:26.444531+0000","last_fullsized":"2024-09-16T04:55:26.444531+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12824,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:28.031628+0000","last_change":"2024-09-16T04:55:28.031672+0000","last_active":"2024-09-16T04:55:28.031628+0000","last_peered":"2024-09-16T04:55:28.031628+0000","last_clean":"2024-09-16T04:55:28.031628+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:28.031628+0000","last_undegraded":"2024-09-16T04:55:28.031628+0000","last_fullsized":"2024-09-16T04:55:28.031628+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12672","last_scrub_stamp":"2024-09-16T04:55:28.031507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:28.031507+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:06.569559+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17276,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443422+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:26.443422+0000","last_peered":"2024-09-16T04:55:26.443422+0000","last_clean":"2024-09-16T04:55:26.443422+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:26.443422+0000","last_undegraded":"2024-09-16T04:55:26.443422+0000","last_fullsized":"2024-09-16T04:55:26.443422+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.633529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17272,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443911+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:26.443911+0000","last_peered":"2024-09-16T04:55:26.443911+0000","last_clean":"2024-09-16T04:55:26.443911+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:26.443911+0000","last_undegraded":"2024-09-16T04:55:26.443911+0000","last_fullsized":"2024-09-16T04:55:26.443911+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":17448,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.684639+0000","last_change":"2024-09-16T04:55:26.684663+0000","last_active":"2024-09-16T04:55:26.684639+0000","last_peered":"2024-09-16T04:55:26.684639+0000","last_clean":"2024-09-16T04:55:26.684639+0000","last_became_active":"2024-09-16T04:55:26.446437+0000","last_became_peered":"2024-09-16T04:55:26.446437+0000","last_unstale":"2024-09-16T04:55:26.684639+0000","last_undegraded":"2024-09-16T04:55:26.684639+0000","last_fullsized":"2024-09-16T04:55:26.684639+0000","mapping_epoch":62,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12739","last_scrub_stamp":"2024-09-16T04:55:26.684554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:26.684554+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:29:17.114250+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16846,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443392+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:26.443392+0000","last_peered":"2024-09-16T04:55:26.443392+0000","last_clean":"2024-09-16T04:55:26.443392+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:26.443392+0000","last_undegraded":"2024-09-16T04:55:26.443392+0000","last_fullsized":"2024-09-16T04:55:26.443392+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17192,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T04:55:25.442654+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:25.442654+0000","last_peered":"2024-09-16T04:55:25.442654+0000","last_clean":"2024-09-16T04:55:25.442654+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:25.442654+0000","last_undegraded":"2024-09-16T04:55:25.442654+0000","last_fullsized":"2024-09-16T04:55:25.442654+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17404,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444330+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:26.444330+0000","last_peered":"2024-09-16T04:55:26.444330+0000","last_clean":"2024-09-16T04:55:26.444330+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:26.444330+0000","last_undegraded":"2024-09-16T04:55:26.444330+0000","last_fullsized":"2024-09-16T04:55:26.444330+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17569,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443342+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:26.443342+0000","last_peered":"2024-09-16T04:55:26.443342+0000","last_clean":"2024-09-16T04:55:26.443342+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:26.443342+0000","last_undegraded":"2024-09-16T04:55:26.443342+0000","last_fullsized":"2024-09-16T04:55:26.443342+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:27.665374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17313,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443527+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:26.443527+0000","last_peered":"2024-09-16T04:55:26.443527+0000","last_clean":"2024-09-16T04:55:26.443527+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:26.443527+0000","last_undegraded":"2024-09-16T04:55:26.443527+0000","last_fullsized":"2024-09-16T04:55:26.443527+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.633504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":17012,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443089+0000","last_change":"2024-09-16T04:55:25.698837+0000","last_active":"2024-09-16T04:55:26.443089+0000","last_peered":"2024-09-16T04:55:26.443089+0000","last_clean":"2024-09-16T04:55:26.443089+0000","last_became_active":"2024-09-16T04:55:14.409783+0000","last_became_peered":"2024-09-16T04:55:14.409783+0000","last_unstale":"2024-09-16T04:55:26.443089+0000","last_undegraded":"2024-09-16T04:55:26.443089+0000","last_fullsized":"2024-09-16T04:55:26.443089+0000","mapping_epoch":60,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12488","last_scrub_stamp":"2024-09-16T04:55:25.698717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:25.698717+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:52.582394+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12458,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444644+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:26.444644+0000","last_peered":"2024-09-16T04:55:26.444644+0000","last_clean":"2024-09-16T04:55:26.444644+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:26.444644+0000","last_undegraded":"2024-09-16T04:55:26.444644+0000","last_fullsized":"2024-09-16T04:55:26.444644+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12672,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444025+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:26.444025+0000","last_peered":"2024-09-16T04:55:26.444025+0000","last_clean":"2024-09-16T04:55:26.444025+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:26.444025+0000","last_undegraded":"2024-09-16T04:55:26.444025+0000","last_fullsized":"2024-09-16T04:55:26.444025+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12940,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444666+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:26.444666+0000","last_peered":"2024-09-16T04:55:26.444666+0000","last_clean":"2024-09-16T04:55:26.444666+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:26.444666+0000","last_undegraded":"2024-09-16T04:55:26.444666+0000","last_fullsized":"2024-09-16T04:55:26.444666+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14978,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444208+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:26.444208+0000","last_peered":"2024-09-16T04:55:26.444208+0000","last_clean":"2024-09-16T04:55:26.444208+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:26.444208+0000","last_undegraded":"2024-09-16T04:55:26.444208+0000","last_fullsized":"2024-09-16T04:55:26.444208+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12455,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:28.998608+0000","last_change":"2024-09-16T04:55:28.998653+0000","last_active":"2024-09-16T04:55:28.998608+0000","last_peered":"2024-09-16T04:55:28.998608+0000","last_clean":"2024-09-16T04:55:28.998608+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:28.998608+0000","last_undegraded":"2024-09-16T04:55:28.998608+0000","last_fullsized":"2024-09-16T04:55:28.998608+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12305","last_scrub_stamp":"2024-09-16T04:55:28.998471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:28.998471+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:53.082759+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":129010,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151264,"num_write_kb":605114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19168,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722204,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649636,"statfs":{"total":96636764160,"available":95897227264,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731420,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640420,"statfs":{"total":96636764160,"available":95887790080,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627796,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93744044,"statfs":{"total":96636764160,"available":95993901056,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:31.224 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-16T04:55:31.225 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T04:55:31.238 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:55:31.238 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:31.514 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:31.515 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:31.527 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-16T04:55:30.834665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":129010,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151264,"num_write_kb":605114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19168,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803544,"kb_used_data":2048,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683816,"statfs":{"total":386547056640,"available":383676227584,"internally_reserved":0,"allocated":2097152,"data_stored":802485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-61110,"num_objects":-9,"num_object_clones":0,"num_object_copies":-36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-50195,"num_write_kb":-200817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001502"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17210,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444531+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:26.444531+0000","last_peered":"2024-09-16T04:55:26.444531+0000","last_clean":"2024-09-16T04:55:26.444531+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:26.444531+0000","last_undegraded":"2024-09-16T04:55:26.444531+0000","last_fullsized":"2024-09-16T04:55:26.444531+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12824,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:28.031628+0000","last_change":"2024-09-16T04:55:28.031672+0000","last_active":"2024-09-16T04:55:28.031628+0000","last_peered":"2024-09-16T04:55:28.031628+0000","last_clean":"2024-09-16T04:55:28.031628+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:28.031628+0000","last_undegraded":"2024-09-16T04:55:28.031628+0000","last_fullsized":"2024-09-16T04:55:28.031628+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12672","last_scrub_stamp":"2024-09-16T04:55:28.031507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:28.031507+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:06.569559+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17276,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443422+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:26.443422+0000","last_peered":"2024-09-16T04:55:26.443422+0000","last_clean":"2024-09-16T04:55:26.443422+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:26.443422+0000","last_undegraded":"2024-09-16T04:55:26.443422+0000","last_fullsized":"2024-09-16T04:55:26.443422+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.633529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17272,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443911+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:26.443911+0000","last_peered":"2024-09-16T04:55:26.443911+0000","last_clean":"2024-09-16T04:55:26.443911+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:26.443911+0000","last_undegraded":"2024-09-16T04:55:26.443911+0000","last_fullsized":"2024-09-16T04:55:26.443911+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":17448,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.684639+0000","last_change":"2024-09-16T04:55:26.684663+0000","last_active":"2024-09-16T04:55:26.684639+0000","last_peered":"2024-09-16T04:55:26.684639+0000","last_clean":"2024-09-16T04:55:26.684639+0000","last_became_active":"2024-09-16T04:55:26.446437+0000","last_became_peered":"2024-09-16T04:55:26.446437+0000","last_unstale":"2024-09-16T04:55:26.684639+0000","last_undegraded":"2024-09-16T04:55:26.684639+0000","last_fullsized":"2024-09-16T04:55:26.684639+0000","mapping_epoch":62,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12739","last_scrub_stamp":"2024-09-16T04:55:26.684554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:26.684554+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:29:17.114250+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16846,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443392+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:26.443392+0000","last_peered":"2024-09-16T04:55:26.443392+0000","last_clean":"2024-09-16T04:55:26.443392+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:26.443392+0000","last_undegraded":"2024-09-16T04:55:26.443392+0000","last_fullsized":"2024-09-16T04:55:26.443392+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17192,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T04:55:25.442654+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:25.442654+0000","last_peered":"2024-09-16T04:55:25.442654+0000","last_clean":"2024-09-16T04:55:25.442654+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:25.442654+0000","last_undegraded":"2024-09-16T04:55:25.442654+0000","last_fullsized":"2024-09-16T04:55:25.442654+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17404,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444330+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:26.444330+0000","last_peered":"2024-09-16T04:55:26.444330+0000","last_clean":"2024-09-16T04:55:26.444330+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:26.444330+0000","last_undegraded":"2024-09-16T04:55:26.444330+0000","last_fullsized":"2024-09-16T04:55:26.444330+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17569,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443342+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:26.443342+0000","last_peered":"2024-09-16T04:55:26.443342+0000","last_clean":"2024-09-16T04:55:26.443342+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:26.443342+0000","last_undegraded":"2024-09-16T04:55:26.443342+0000","last_fullsized":"2024-09-16T04:55:26.443342+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:27.665374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17313,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443527+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:26.443527+0000","last_peered":"2024-09-16T04:55:26.443527+0000","last_clean":"2024-09-16T04:55:26.443527+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:26.443527+0000","last_undegraded":"2024-09-16T04:55:26.443527+0000","last_fullsized":"2024-09-16T04:55:26.443527+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.633504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":17012,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443089+0000","last_change":"2024-09-16T04:55:25.698837+0000","last_active":"2024-09-16T04:55:26.443089+0000","last_peered":"2024-09-16T04:55:26.443089+0000","last_clean":"2024-09-16T04:55:26.443089+0000","last_became_active":"2024-09-16T04:55:14.409783+0000","last_became_peered":"2024-09-16T04:55:14.409783+0000","last_unstale":"2024-09-16T04:55:26.443089+0000","last_undegraded":"2024-09-16T04:55:26.443089+0000","last_fullsized":"2024-09-16T04:55:26.443089+0000","mapping_epoch":60,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12488","last_scrub_stamp":"2024-09-16T04:55:25.698717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:25.698717+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:52.582394+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12458,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444644+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:26.444644+0000","last_peered":"2024-09-16T04:55:26.444644+0000","last_clean":"2024-09-16T04:55:26.444644+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:26.444644+0000","last_undegraded":"2024-09-16T04:55:26.444644+0000","last_fullsized":"2024-09-16T04:55:26.444644+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12672,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444025+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:26.444025+0000","last_peered":"2024-09-16T04:55:26.444025+0000","last_clean":"2024-09-16T04:55:26.444025+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:26.444025+0000","last_undegraded":"2024-09-16T04:55:26.444025+0000","last_fullsized":"2024-09-16T04:55:26.444025+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12940,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444666+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:26.444666+0000","last_peered":"2024-09-16T04:55:26.444666+0000","last_clean":"2024-09-16T04:55:26.444666+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:26.444666+0000","last_undegraded":"2024-09-16T04:55:26.444666+0000","last_fullsized":"2024-09-16T04:55:26.444666+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14978,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444208+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:26.444208+0000","last_peered":"2024-09-16T04:55:26.444208+0000","last_clean":"2024-09-16T04:55:26.444208+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:26.444208+0000","last_undegraded":"2024-09-16T04:55:26.444208+0000","last_fullsized":"2024-09-16T04:55:26.444208+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12455,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:28.998608+0000","last_change":"2024-09-16T04:55:28.998653+0000","last_active":"2024-09-16T04:55:28.998608+0000","last_peered":"2024-09-16T04:55:28.998608+0000","last_clean":"2024-09-16T04:55:28.998608+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:28.998608+0000","last_undegraded":"2024-09-16T04:55:28.998608+0000","last_fullsized":"2024-09-16T04:55:28.998608+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12305","last_scrub_stamp":"2024-09-16T04:55:28.998471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:28.998471+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:53.082759+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":129010,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151264,"num_write_kb":605114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19168,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722204,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649636,"statfs":{"total":96636764160,"available":95897227264,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731420,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640420,"statfs":{"total":96636764160,"available":95887790080,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627796,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93744044,"statfs":{"total":96636764160,"available":95993901056,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:31.528 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:31.805 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:31.806 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:31.818 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-16T04:55:30.834665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":129010,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151264,"num_write_kb":605114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19168,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803544,"kb_used_data":2048,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683816,"statfs":{"total":386547056640,"available":383676227584,"internally_reserved":0,"allocated":2097152,"data_stored":802485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-61110,"num_objects":-9,"num_object_clones":0,"num_object_copies":-36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-50195,"num_write_kb":-200817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001502"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17210,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444531+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:26.444531+0000","last_peered":"2024-09-16T04:55:26.444531+0000","last_clean":"2024-09-16T04:55:26.444531+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:26.444531+0000","last_undegraded":"2024-09-16T04:55:26.444531+0000","last_fullsized":"2024-09-16T04:55:26.444531+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12824,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:28.031628+0000","last_change":"2024-09-16T04:55:28.031672+0000","last_active":"2024-09-16T04:55:28.031628+0000","last_peered":"2024-09-16T04:55:28.031628+0000","last_clean":"2024-09-16T04:55:28.031628+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:28.031628+0000","last_undegraded":"2024-09-16T04:55:28.031628+0000","last_fullsized":"2024-09-16T04:55:28.031628+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12672","last_scrub_stamp":"2024-09-16T04:55:28.031507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:28.031507+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:06.569559+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17276,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443422+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:26.443422+0000","last_peered":"2024-09-16T04:55:26.443422+0000","last_clean":"2024-09-16T04:55:26.443422+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:26.443422+0000","last_undegraded":"2024-09-16T04:55:26.443422+0000","last_fullsized":"2024-09-16T04:55:26.443422+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.633529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17272,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443911+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:26.443911+0000","last_peered":"2024-09-16T04:55:26.443911+0000","last_clean":"2024-09-16T04:55:26.443911+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:26.443911+0000","last_undegraded":"2024-09-16T04:55:26.443911+0000","last_fullsized":"2024-09-16T04:55:26.443911+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":17448,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.684639+0000","last_change":"2024-09-16T04:55:26.684663+0000","last_active":"2024-09-16T04:55:26.684639+0000","last_peered":"2024-09-16T04:55:26.684639+0000","last_clean":"2024-09-16T04:55:26.684639+0000","last_became_active":"2024-09-16T04:55:26.446437+0000","last_became_peered":"2024-09-16T04:55:26.446437+0000","last_unstale":"2024-09-16T04:55:26.684639+0000","last_undegraded":"2024-09-16T04:55:26.684639+0000","last_fullsized":"2024-09-16T04:55:26.684639+0000","mapping_epoch":62,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12739","last_scrub_stamp":"2024-09-16T04:55:26.684554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:26.684554+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:29:17.114250+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16846,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443392+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:26.443392+0000","last_peered":"2024-09-16T04:55:26.443392+0000","last_clean":"2024-09-16T04:55:26.443392+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:26.443392+0000","last_undegraded":"2024-09-16T04:55:26.443392+0000","last_fullsized":"2024-09-16T04:55:26.443392+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17192,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T04:55:25.442654+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:25.442654+0000","last_peered":"2024-09-16T04:55:25.442654+0000","last_clean":"2024-09-16T04:55:25.442654+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:25.442654+0000","last_undegraded":"2024-09-16T04:55:25.442654+0000","last_fullsized":"2024-09-16T04:55:25.442654+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17404,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444330+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:26.444330+0000","last_peered":"2024-09-16T04:55:26.444330+0000","last_clean":"2024-09-16T04:55:26.444330+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:26.444330+0000","last_undegraded":"2024-09-16T04:55:26.444330+0000","last_fullsized":"2024-09-16T04:55:26.444330+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17569,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443342+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:26.443342+0000","last_peered":"2024-09-16T04:55:26.443342+0000","last_clean":"2024-09-16T04:55:26.443342+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:26.443342+0000","last_undegraded":"2024-09-16T04:55:26.443342+0000","last_fullsized":"2024-09-16T04:55:26.443342+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:27.665374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17313,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443527+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:26.443527+0000","last_peered":"2024-09-16T04:55:26.443527+0000","last_clean":"2024-09-16T04:55:26.443527+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:26.443527+0000","last_undegraded":"2024-09-16T04:55:26.443527+0000","last_fullsized":"2024-09-16T04:55:26.443527+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.633504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":17012,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443089+0000","last_change":"2024-09-16T04:55:25.698837+0000","last_active":"2024-09-16T04:55:26.443089+0000","last_peered":"2024-09-16T04:55:26.443089+0000","last_clean":"2024-09-16T04:55:26.443089+0000","last_became_active":"2024-09-16T04:55:14.409783+0000","last_became_peered":"2024-09-16T04:55:14.409783+0000","last_unstale":"2024-09-16T04:55:26.443089+0000","last_undegraded":"2024-09-16T04:55:26.443089+0000","last_fullsized":"2024-09-16T04:55:26.443089+0000","mapping_epoch":60,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12488","last_scrub_stamp":"2024-09-16T04:55:25.698717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:25.698717+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:52.582394+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12458,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444644+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:26.444644+0000","last_peered":"2024-09-16T04:55:26.444644+0000","last_clean":"2024-09-16T04:55:26.444644+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:26.444644+0000","last_undegraded":"2024-09-16T04:55:26.444644+0000","last_fullsized":"2024-09-16T04:55:26.444644+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12672,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444025+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:26.444025+0000","last_peered":"2024-09-16T04:55:26.444025+0000","last_clean":"2024-09-16T04:55:26.444025+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:26.444025+0000","last_undegraded":"2024-09-16T04:55:26.444025+0000","last_fullsized":"2024-09-16T04:55:26.444025+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12940,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444666+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:26.444666+0000","last_peered":"2024-09-16T04:55:26.444666+0000","last_clean":"2024-09-16T04:55:26.444666+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:26.444666+0000","last_undegraded":"2024-09-16T04:55:26.444666+0000","last_fullsized":"2024-09-16T04:55:26.444666+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14978,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444208+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:26.444208+0000","last_peered":"2024-09-16T04:55:26.444208+0000","last_clean":"2024-09-16T04:55:26.444208+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:26.444208+0000","last_undegraded":"2024-09-16T04:55:26.444208+0000","last_fullsized":"2024-09-16T04:55:26.444208+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12455,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:28.998608+0000","last_change":"2024-09-16T04:55:28.998653+0000","last_active":"2024-09-16T04:55:28.998608+0000","last_peered":"2024-09-16T04:55:28.998608+0000","last_clean":"2024-09-16T04:55:28.998608+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:28.998608+0000","last_undegraded":"2024-09-16T04:55:28.998608+0000","last_fullsized":"2024-09-16T04:55:28.998608+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12305","last_scrub_stamp":"2024-09-16T04:55:28.998471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:28.998471+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:53.082759+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":129010,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151264,"num_write_kb":605114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19168,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722204,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649636,"statfs":{"total":96636764160,"available":95897227264,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731420,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640420,"statfs":{"total":96636764160,"available":95887790080,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627796,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93744044,"statfs":{"total":96636764160,"available":95993901056,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:31.819 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:55:31.819 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:32.099 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:32.099 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:32.113 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-16T04:55:30.834665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":129010,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151264,"num_write_kb":605114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19168,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803544,"kb_used_data":2048,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683816,"statfs":{"total":386547056640,"available":383676227584,"internally_reserved":0,"allocated":2097152,"data_stored":802485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-61110,"num_objects":-9,"num_object_clones":0,"num_object_copies":-36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-50195,"num_write_kb":-200817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001502"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17210,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444531+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:26.444531+0000","last_peered":"2024-09-16T04:55:26.444531+0000","last_clean":"2024-09-16T04:55:26.444531+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:26.444531+0000","last_undegraded":"2024-09-16T04:55:26.444531+0000","last_fullsized":"2024-09-16T04:55:26.444531+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12824,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:28.031628+0000","last_change":"2024-09-16T04:55:28.031672+0000","last_active":"2024-09-16T04:55:28.031628+0000","last_peered":"2024-09-16T04:55:28.031628+0000","last_clean":"2024-09-16T04:55:28.031628+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:28.031628+0000","last_undegraded":"2024-09-16T04:55:28.031628+0000","last_fullsized":"2024-09-16T04:55:28.031628+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12672","last_scrub_stamp":"2024-09-16T04:55:28.031507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:28.031507+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:06.569559+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17276,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443422+0000","last_change":"2024-09-16T04:54:50.394068+0000","last_active":"2024-09-16T04:55:26.443422+0000","last_peered":"2024-09-16T04:55:26.443422+0000","last_clean":"2024-09-16T04:55:26.443422+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:26.443422+0000","last_undegraded":"2024-09-16T04:55:26.443422+0000","last_fullsized":"2024-09-16T04:55:26.443422+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.633529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12669,"num_write_kb":50671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17272,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443911+0000","last_change":"2024-09-16T04:51:04.846432+0000","last_active":"2024-09-16T04:55:26.443911+0000","last_peered":"2024-09-16T04:55:26.443911+0000","last_clean":"2024-09-16T04:55:26.443911+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:26.443911+0000","last_undegraded":"2024-09-16T04:55:26.443911+0000","last_fullsized":"2024-09-16T04:55:26.443911+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":17448,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.684639+0000","last_change":"2024-09-16T04:55:26.684663+0000","last_active":"2024-09-16T04:55:26.684639+0000","last_peered":"2024-09-16T04:55:26.684639+0000","last_clean":"2024-09-16T04:55:26.684639+0000","last_became_active":"2024-09-16T04:55:26.446437+0000","last_became_peered":"2024-09-16T04:55:26.446437+0000","last_unstale":"2024-09-16T04:55:26.684639+0000","last_undegraded":"2024-09-16T04:55:26.684639+0000","last_fullsized":"2024-09-16T04:55:26.684639+0000","mapping_epoch":62,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12739","last_scrub_stamp":"2024-09-16T04:55:26.684554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:26.684554+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:29:17.114250+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16846,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443392+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:26.443392+0000","last_peered":"2024-09-16T04:55:26.443392+0000","last_clean":"2024-09-16T04:55:26.443392+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:26.443392+0000","last_undegraded":"2024-09-16T04:55:26.443392+0000","last_fullsized":"2024-09-16T04:55:26.443392+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17192,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T04:55:25.442654+0000","last_change":"2024-09-16T04:53:16.100110+0000","last_active":"2024-09-16T04:55:25.442654+0000","last_peered":"2024-09-16T04:55:25.442654+0000","last_clean":"2024-09-16T04:55:25.442654+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:25.442654+0000","last_undegraded":"2024-09-16T04:55:25.442654+0000","last_fullsized":"2024-09-16T04:55:25.442654+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":50388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17404,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444330+0000","last_change":"2024-09-16T04:44:12.037555+0000","last_active":"2024-09-16T04:55:26.444330+0000","last_peered":"2024-09-16T04:55:26.444330+0000","last_clean":"2024-09-16T04:55:26.444330+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:26.444330+0000","last_undegraded":"2024-09-16T04:55:26.444330+0000","last_fullsized":"2024-09-16T04:55:26.444330+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":51132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":18333696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17569,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443342+0000","last_change":"2024-09-16T04:47:52.498157+0000","last_active":"2024-09-16T04:55:26.443342+0000","last_peered":"2024-09-16T04:55:26.443342+0000","last_clean":"2024-09-16T04:55:26.443342+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:26.443342+0000","last_undegraded":"2024-09-16T04:55:26.443342+0000","last_fullsized":"2024-09-16T04:55:26.443342+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:27.665374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":51009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17313,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443527+0000","last_change":"2024-09-16T04:54:27.294697+0000","last_active":"2024-09-16T04:55:26.443527+0000","last_peered":"2024-09-16T04:55:26.443527+0000","last_clean":"2024-09-16T04:55:26.443527+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:26.443527+0000","last_undegraded":"2024-09-16T04:55:26.443527+0000","last_fullsized":"2024-09-16T04:55:26.443527+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.633504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":51022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":17012,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443089+0000","last_change":"2024-09-16T04:55:25.698837+0000","last_active":"2024-09-16T04:55:26.443089+0000","last_peered":"2024-09-16T04:55:26.443089+0000","last_clean":"2024-09-16T04:55:26.443089+0000","last_became_active":"2024-09-16T04:55:14.409783+0000","last_became_peered":"2024-09-16T04:55:14.409783+0000","last_unstale":"2024-09-16T04:55:26.443089+0000","last_undegraded":"2024-09-16T04:55:26.443089+0000","last_fullsized":"2024-09-16T04:55:26.443089+0000","mapping_epoch":60,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12488","last_scrub_stamp":"2024-09-16T04:55:25.698717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:25.698717+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:52.582394+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12458,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444644+0000","last_change":"2024-09-16T04:54:38.366367+0000","last_active":"2024-09-16T04:55:26.444644+0000","last_peered":"2024-09-16T04:55:26.444644+0000","last_clean":"2024-09-16T04:55:26.444644+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:26.444644+0000","last_undegraded":"2024-09-16T04:55:26.444644+0000","last_fullsized":"2024-09-16T04:55:26.444644+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12672,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444025+0000","last_change":"2024-09-16T04:53:52.171189+0000","last_active":"2024-09-16T04:55:26.444025+0000","last_peered":"2024-09-16T04:55:26.444025+0000","last_clean":"2024-09-16T04:55:26.444025+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:26.444025+0000","last_undegraded":"2024-09-16T04:55:26.444025+0000","last_fullsized":"2024-09-16T04:55:26.444025+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":50127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12940,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444666+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:26.444666+0000","last_peered":"2024-09-16T04:55:26.444666+0000","last_clean":"2024-09-16T04:55:26.444666+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:26.444666+0000","last_undegraded":"2024-09-16T04:55:26.444666+0000","last_fullsized":"2024-09-16T04:55:26.444666+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14978,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444208+0000","last_change":"2024-09-16T04:53:04.009719+0000","last_active":"2024-09-16T04:55:26.444208+0000","last_peered":"2024-09-16T04:55:26.444208+0000","last_clean":"2024-09-16T04:55:26.444208+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:26.444208+0000","last_undegraded":"2024-09-16T04:55:26.444208+0000","last_fullsized":"2024-09-16T04:55:26.444208+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:55:29.016158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12455,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:28.998608+0000","last_change":"2024-09-16T04:55:28.998653+0000","last_active":"2024-09-16T04:55:28.998608+0000","last_peered":"2024-09-16T04:55:28.998608+0000","last_clean":"2024-09-16T04:55:28.998608+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:28.998608+0000","last_undegraded":"2024-09-16T04:55:28.998608+0000","last_fullsized":"2024-09-16T04:55:28.998608+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12305","last_scrub_stamp":"2024-09-16T04:55:28.998471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:28.998471+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:53.082759+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":129010,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151264,"num_write_kb":605114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19168,"num_bytes_recovered":34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722204,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649636,"statfs":{"total":96636764160,"available":95897227264,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731420,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640420,"statfs":{"total":96636764160,"available":95887790080,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627796,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93744044,"statfs":{"total":96636764160,"available":95993901056,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722124,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649716,"statfs":{"total":96636764160,"available":95897309184,"internally_reserved":0,"allocated":512000,"data_stored":196035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:32.114 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:55:32.392 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:32.392 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":63,"fsid":"47829f7b-b43c-44e0-b4e2-77e6a22c542c","created":"2024-09-16T04:39:47.674616+0000","modified":"2024-09-16T04:55:26.435210+0000","last_up_change":"2024-09-16T04:41:26.169553+0000","last_in_change":"2024-09-16T04:39:49.620271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-16T04:40:10.631384+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"d8b076bc-f4ae-4100-8cdf-78560225fea9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":47,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4015439555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6801","nonce":4015439555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":4015439555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4015439555}]},"public_addr":"172.21.15.152:6800/4015439555","cluster_addr":"172.21.15.152:6801/4015439555","heartbeat_back_addr":"172.21.15.152:6803/4015439555","heartbeat_front_addr":"172.21.15.152:6802/4015439555","state":["exists","up"]},{"osd":1,"uuid":"023d4748-7485-4d8b-8594-54a238b67b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":29,"up_thru":58,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1865775950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6813","nonce":1865775950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6815","nonce":1865775950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1865775950}]},"public_addr":"172.21.15.152:6812/1865775950","cluster_addr":"172.21.15.152:6813/1865775950","heartbeat_back_addr":"172.21.15.152:6815/1865775950","heartbeat_front_addr":"172.21.15.152:6814/1865775950","state":["exists","up"]},{"osd":2,"uuid":"c7bfb282-86b4-4bdc-902d-0a0cfbadf5ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2854825795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2854825795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":2854825795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2854825795}]},"public_addr":"172.21.15.152:6804/2854825795","cluster_addr":"172.21.15.152:6805/2854825795","heartbeat_back_addr":"172.21.15.152:6807/2854825795","heartbeat_front_addr":"172.21.15.152:6806/2854825795","state":["exists","up"]},{"osd":3,"uuid":"f82045bc-6fe9-4755-8c25-ea7738b90d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":62,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1506610745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":1506610745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6811","nonce":1506610745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1506610745}]},"public_addr":"172.21.15.152:6808/1506610745","cluster_addr":"172.21.15.152:6809/1506610745","heartbeat_back_addr":"172.21.15.152:6811/1506610745","heartbeat_front_addr":"172.21.15.152:6810/1506610745","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.429729+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.175556+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.243970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:39:56.373375+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:55:33.406 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T04:55:33.407 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:55:33.544 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:55:33.544 INFO:teuthology.orchestra.run.smithi152.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:55:33.544 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:55:33.555 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T04:55:33.833 INFO:teuthology.orchestra.run.smithi152.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T04:55:33.845 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T04:55:33.846 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:55:33.972 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:55:33.972 INFO:teuthology.orchestra.run.smithi152.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-16T04:55:33.972 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:55:33.982 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T04:55:34.260 INFO:teuthology.orchestra.run.smithi152.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T04:55:34.273 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T04:55:34.273 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:55:34.395 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:55:34.395 INFO:teuthology.orchestra.run.smithi152.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:55:34.395 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:55:34.406 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T04:55:34.690 INFO:teuthology.orchestra.run.smithi152.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T04:55:34.703 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T04:55:34.703 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:55:34.824 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-16T04:55:34.824 INFO:teuthology.orchestra.run.smithi152.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:55:34.824 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-16T04:55:34.834 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T04:55:35.111 INFO:teuthology.orchestra.run.smithi152.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T04:55:35.124 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:35.401 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:35.402 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:35.414 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":528,"stamp":"2024-09-16T04:55:34.835447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37744,"num_write_kb":150988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4465,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803536,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683824,"statfs":{"total":386547056640,"available":383676235776,"internally_reserved":0,"allocated":2113536,"data_stored":808600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-169750,"num_objects":-25,"num_object_clones":0,"num_object_copies":-100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-163715,"num_write_kb":-654943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-22874,"num_bytes_recovered":-34258944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.400020"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17210,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444531+0000","last_change":"2024-09-16T04:54:15.253890+0000","last_active":"2024-09-16T04:55:26.444531+0000","last_peered":"2024-09-16T04:55:26.444531+0000","last_clean":"2024-09-16T04:55:26.444531+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:26.444531+0000","last_undegraded":"2024-09-16T04:55:26.444531+0000","last_fullsized":"2024-09-16T04:55:26.444531+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:51.081863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12825,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:28.031672+0000","last_change":"2024-09-16T04:55:28.031672+0000","last_active":"2024-09-16T04:55:28.031672+0000","last_peered":"2024-09-16T04:55:28.031672+0000","last_clean":"2024-09-16T04:55:28.031672+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:28.031672+0000","last_undegraded":"2024-09-16T04:55:28.031672+0000","last_fullsized":"2024-09-16T04:55:28.031672+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12672","last_scrub_stamp":"2024-09-16T04:55:28.031507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:28.031507+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17284,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:31.613376+0000","last_change":"2024-09-16T04:55:31.613417+0000","last_active":"2024-09-16T04:55:31.613376+0000","last_peered":"2024-09-16T04:55:31.613376+0000","last_clean":"2024-09-16T04:55:31.613376+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:31.613376+0000","last_undegraded":"2024-09-16T04:55:31.613376+0000","last_fullsized":"2024-09-16T04:55:31.613376+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12670","last_scrub_stamp":"2024-09-16T04:55:31.613266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:31.613266+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:34:20.779117+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17281,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:30.922432+0000","last_change":"2024-09-16T04:55:30.922432+0000","last_active":"2024-09-16T04:55:30.922432+0000","last_peered":"2024-09-16T04:55:30.922432+0000","last_clean":"2024-09-16T04:55:30.922432+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:30.922432+0000","last_undegraded":"2024-09-16T04:55:30.922432+0000","last_fullsized":"2024-09-16T04:55:30.922432+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12668","last_scrub_stamp":"2024-09-16T04:55:30.922253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:30.922253+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":17448,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.684639+0000","last_change":"2024-09-16T04:55:26.684663+0000","last_active":"2024-09-16T04:55:26.684639+0000","last_peered":"2024-09-16T04:55:26.684639+0000","last_clean":"2024-09-16T04:55:26.684639+0000","last_became_active":"2024-09-16T04:55:26.446437+0000","last_became_peered":"2024-09-16T04:55:26.446437+0000","last_unstale":"2024-09-16T04:55:26.684639+0000","last_undegraded":"2024-09-16T04:55:26.684639+0000","last_fullsized":"2024-09-16T04:55:26.684639+0000","mapping_epoch":62,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12739","last_scrub_stamp":"2024-09-16T04:55:26.684554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:26.684554+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:29:17.114250+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16846,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443392+0000","last_change":"2024-09-16T04:53:27.692108+0000","last_active":"2024-09-16T04:55:26.443392+0000","last_peered":"2024-09-16T04:55:26.443392+0000","last_clean":"2024-09-16T04:55:26.443392+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:26.443392+0000","last_undegraded":"2024-09-16T04:55:26.443392+0000","last_fullsized":"2024-09-16T04:55:26.443392+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:11.055225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17202,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.469752+0000","last_change":"2024-09-16T04:55:26.469809+0000","last_active":"2024-09-16T04:55:26.469752+0000","last_peered":"2024-09-16T04:55:26.469752+0000","last_clean":"2024-09-16T04:55:26.469752+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:26.469752+0000","last_undegraded":"2024-09-16T04:55:26.469752+0000","last_fullsized":"2024-09-16T04:55:26.469752+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12598","last_scrub_stamp":"2024-09-16T04:55:26.469611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:26.469611+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:13.480692+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17413,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:29.973422+0000","last_change":"2024-09-16T04:55:29.973422+0000","last_active":"2024-09-16T04:55:29.973422+0000","last_peered":"2024-09-16T04:55:29.973422+0000","last_clean":"2024-09-16T04:55:29.973422+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:29.973422+0000","last_undegraded":"2024-09-16T04:55:29.973422+0000","last_fullsized":"2024-09-16T04:55:29.973422+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12783","last_scrub_stamp":"2024-09-16T04:55:29.973282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:29.973282+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17577,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:27.698411+0000","last_change":"2024-09-16T04:55:27.698440+0000","last_active":"2024-09-16T04:55:27.698411+0000","last_peered":"2024-09-16T04:55:27.698411+0000","last_clean":"2024-09-16T04:55:27.698411+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:27.698411+0000","last_undegraded":"2024-09-16T04:55:27.698411+0000","last_fullsized":"2024-09-16T04:55:27.698411+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12752","last_scrub_stamp":"2024-09-16T04:55:27.698333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:27.698333+0000","objects_scrubbed":1,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:46:38.448237+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17321,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:30.615230+0000","last_change":"2024-09-16T04:55:30.615273+0000","last_active":"2024-09-16T04:55:30.615230+0000","last_peered":"2024-09-16T04:55:30.615230+0000","last_clean":"2024-09-16T04:55:30.615230+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:30.615230+0000","last_undegraded":"2024-09-16T04:55:30.615230+0000","last_fullsized":"2024-09-16T04:55:30.615230+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12756","last_scrub_stamp":"2024-09-16T04:55:30.615105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:30.615105+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:44:44.214177+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":17012,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.443089+0000","last_change":"2024-09-16T04:55:25.698837+0000","last_active":"2024-09-16T04:55:26.443089+0000","last_peered":"2024-09-16T04:55:26.443089+0000","last_clean":"2024-09-16T04:55:26.443089+0000","last_became_active":"2024-09-16T04:55:14.409783+0000","last_became_peered":"2024-09-16T04:55:14.409783+0000","last_unstale":"2024-09-16T04:55:26.443089+0000","last_undegraded":"2024-09-16T04:55:26.443089+0000","last_fullsized":"2024-09-16T04:55:26.443089+0000","mapping_epoch":60,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12488","last_scrub_stamp":"2024-09-16T04:55:25.698717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:25.698717+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:52.582394+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12467,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:33.923097+0000","last_change":"2024-09-16T04:55:33.923097+0000","last_active":"2024-09-16T04:55:33.923097+0000","last_peered":"2024-09-16T04:55:33.923097+0000","last_clean":"2024-09-16T04:55:33.923097+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:33.923097+0000","last_undegraded":"2024-09-16T04:55:33.923097+0000","last_fullsized":"2024-09-16T04:55:33.923097+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12330","last_scrub_stamp":"2024-09-16T04:55:33.922980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:33.922980+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12681,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:32.903548+0000","last_change":"2024-09-16T04:55:32.903548+0000","last_active":"2024-09-16T04:55:32.903548+0000","last_peered":"2024-09-16T04:55:32.903548+0000","last_clean":"2024-09-16T04:55:32.903548+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:32.903548+0000","last_undegraded":"2024-09-16T04:55:32.903548+0000","last_fullsized":"2024-09-16T04:55:32.903548+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12532","last_scrub_stamp":"2024-09-16T04:55:32.903360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:32.903360+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12941,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:26.444705+0000","last_change":"2024-09-16T04:55:02.447252+0000","last_active":"2024-09-16T04:55:26.444705+0000","last_peered":"2024-09-16T04:55:26.444705+0000","last_clean":"2024-09-16T04:55:26.444705+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:26.444705+0000","last_undegraded":"2024-09-16T04:55:26.444705+0000","last_fullsized":"2024-09-16T04:55:26.444705+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:40:11.606968+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14987,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:31.948588+0000","last_change":"2024-09-16T04:55:31.948588+0000","last_active":"2024-09-16T04:55:31.948588+0000","last_peered":"2024-09-16T04:55:31.948588+0000","last_clean":"2024-09-16T04:55:31.948588+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:31.948588+0000","last_undegraded":"2024-09-16T04:55:31.948588+0000","last_fullsized":"2024-09-16T04:55:31.948588+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12447","last_scrub_stamp":"2024-09-16T04:55:31.948417+0000","last_deep_scrub":"37'12447","last_deep_scrub_stamp":"2024-09-16T04:55:31.948417+0000","last_clean_scrub_stamp":"2024-09-16T04:55:31.948417+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12456,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:28.998653+0000","last_change":"2024-09-16T04:55:28.998653+0000","last_active":"2024-09-16T04:55:28.998653+0000","last_peered":"2024-09-16T04:55:28.998653+0000","last_clean":"2024-09-16T04:55:28.998653+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:28.998653+0000","last_undegraded":"2024-09-16T04:55:28.998653+0000","last_fullsized":"2024-09-16T04:55:28.998653+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12305","last_scrub_stamp":"2024-09-16T04:55:28.998471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:40:11.606968+0000","last_clean_scrub_stamp":"2024-09-16T04:55:28.998471+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37744,"num_write_kb":150988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4465,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722196,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649644,"statfs":{"total":96636764160,"available":95897235456,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731412,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640428,"statfs":{"total":96636764160,"available":95887798272,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627796,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93744044,"statfs":{"total":96636764160,"available":95993901056,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722132,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649708,"statfs":{"total":96636764160,"available":95897300992,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:35.418 INFO:tasks.ceph:pgid 1.f last_scrub_stamp 2024-09-16T04:40:11.606968+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=40, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:55:35.419 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2024-09-16T04:55:28.031507+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=28, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:55:35.419 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2024-09-16T04:55:31.613266+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:55:35.419 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T04:55:30.922253+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:55:35.419 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-16T04:55:26.684554+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:55:35.419 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2024-09-16T04:40:11.606968+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=40, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:55:35.419 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-16T04:55:26.469611+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:55:35.420 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-16T04:55:29.973282+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=29, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:55:35.420 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-16T04:55:27.698333+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:55:35.420 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-16T04:55:30.615105+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:55:35.420 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-16T04:55:25.698717+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:55:35.420 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2024-09-16T04:55:32.903360+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:55:35.420 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2024-09-16T04:40:11.606968+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=40, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:55:35.421 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2024-09-16T04:55:31.948417+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:55:35.421 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2024-09-16T04:55:28.998471+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=28, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=55, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:55:35.421 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T04:55:55.427 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:55:55.716 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-16T04:55:55.716 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-16T04:55:55.728 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2024-09-16T04:55:54.839598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24966,"num_write_kb":99864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4427,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2803536,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2801393,"kb_avail":374683824,"statfs":{"total":386547056640,"available":383676235776,"internally_reserved":0,"allocated":2113536,"data_stored":808600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14799,"internal_metadata":2868626993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12778,"num_write_kb":-51124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002591"},"pg_stats":[{"pgid":"1.f","version":"37'12654","reported_seq":17219,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:37.587852+0000","last_change":"2024-09-16T04:55:37.587892+0000","last_active":"2024-09-16T04:55:37.587852+0000","last_peered":"2024-09-16T04:55:37.587852+0000","last_clean":"2024-09-16T04:55:37.587852+0000","last_became_active":"2024-09-16T04:54:15.179758+0000","last_became_peered":"2024-09-16T04:54:15.179758+0000","last_unstale":"2024-09-16T04:55:37.587852+0000","last_undegraded":"2024-09-16T04:55:37.587852+0000","last_fullsized":"2024-09-16T04:55:37.587852+0000","mapping_epoch":50,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12654","last_scrub_stamp":"2024-09-16T04:55:37.587714+0000","last_deep_scrub":"37'12654","last_deep_scrub_stamp":"2024-09-16T04:55:37.587714+0000","last_clean_scrub_stamp":"2024-09-16T04:55:37.587714+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:21.294631+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":50616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12672","reported_seq":12834,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:37.880239+0000","last_change":"2024-09-16T04:55:37.880239+0000","last_active":"2024-09-16T04:55:37.880239+0000","last_peered":"2024-09-16T04:55:37.880239+0000","last_clean":"2024-09-16T04:55:37.880239+0000","last_became_active":"2024-09-16T04:53:40.061372+0000","last_became_peered":"2024-09-16T04:53:40.061372+0000","last_unstale":"2024-09-16T04:55:37.880239+0000","last_undegraded":"2024-09-16T04:55:37.880239+0000","last_fullsized":"2024-09-16T04:55:37.880239+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12672","last_scrub_stamp":"2024-09-16T04:55:37.880105+0000","last_deep_scrub":"37'12672","last_deep_scrub_stamp":"2024-09-16T04:55:37.880105+0000","last_clean_scrub_stamp":"2024-09-16T04:55:37.880105+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:53:00.649776+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12670","reported_seq":17293,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:36.707226+0000","last_change":"2024-09-16T04:55:36.707255+0000","last_active":"2024-09-16T04:55:36.707226+0000","last_peered":"2024-09-16T04:55:36.707226+0000","last_clean":"2024-09-16T04:55:36.707226+0000","last_became_active":"2024-09-16T04:54:50.321829+0000","last_became_peered":"2024-09-16T04:54:50.321829+0000","last_unstale":"2024-09-16T04:55:36.707226+0000","last_undegraded":"2024-09-16T04:55:36.707226+0000","last_fullsized":"2024-09-16T04:55:36.707226+0000","mapping_epoch":56,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12670","last_scrub_stamp":"2024-09-16T04:55:36.707118+0000","last_deep_scrub":"37'12670","last_deep_scrub_stamp":"2024-09-16T04:55:36.707118+0000","last_clean_scrub_stamp":"2024-09-16T04:55:36.707118+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:50:36.770575+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12668","reported_seq":17289,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:35.890884+0000","last_change":"2024-09-16T04:55:35.890908+0000","last_active":"2024-09-16T04:55:35.890884+0000","last_peered":"2024-09-16T04:55:35.890884+0000","last_clean":"2024-09-16T04:55:35.890884+0000","last_became_active":"2024-09-16T04:51:04.601366+0000","last_became_peered":"2024-09-16T04:51:04.601366+0000","last_unstale":"2024-09-16T04:55:35.890884+0000","last_undegraded":"2024-09-16T04:55:35.890884+0000","last_fullsized":"2024-09-16T04:55:35.890884+0000","mapping_epoch":36,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12668","last_scrub_stamp":"2024-09-16T04:55:35.890799+0000","last_deep_scrub":"37'12668","last_deep_scrub_stamp":"2024-09-16T04:55:35.890799+0000","last_clean_scrub_stamp":"2024-09-16T04:55:35.890799+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:26.259502+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12739","reported_seq":17457,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:39.608147+0000","last_change":"2024-09-16T04:55:39.608192+0000","last_active":"2024-09-16T04:55:39.608147+0000","last_peered":"2024-09-16T04:55:39.608147+0000","last_clean":"2024-09-16T04:55:39.608147+0000","last_became_active":"2024-09-16T04:55:26.446437+0000","last_became_peered":"2024-09-16T04:55:26.446437+0000","last_unstale":"2024-09-16T04:55:39.608147+0000","last_undegraded":"2024-09-16T04:55:39.608147+0000","last_fullsized":"2024-09-16T04:55:39.608147+0000","mapping_epoch":62,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12739","last_scrub_stamp":"2024-09-16T04:55:39.608027+0000","last_deep_scrub":"37'12739","last_deep_scrub_stamp":"2024-09-16T04:55:39.608027+0000","last_clean_scrub_stamp":"2024-09-16T04:55:39.608027+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:19.606998+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12312","reported_seq":16855,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:36.549820+0000","last_change":"2024-09-16T04:55:36.549869+0000","last_active":"2024-09-16T04:55:36.549820+0000","last_peered":"2024-09-16T04:55:36.549820+0000","last_clean":"2024-09-16T04:55:36.549820+0000","last_became_active":"2024-09-16T04:53:27.591710+0000","last_became_peered":"2024-09-16T04:53:27.591710+0000","last_unstale":"2024-09-16T04:55:36.549820+0000","last_undegraded":"2024-09-16T04:55:36.549820+0000","last_fullsized":"2024-09-16T04:55:36.549820+0000","mapping_epoch":42,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12312","last_scrub_stamp":"2024-09-16T04:55:36.549665+0000","last_deep_scrub":"37'12312","last_deep_scrub_stamp":"2024-09-16T04:55:36.549665+0000","last_clean_scrub_stamp":"2024-09-16T04:55:36.549665+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:31.745692+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12312,"num_write_kb":49248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12598","reported_seq":17211,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:34.627995+0000","last_change":"2024-09-16T04:55:34.628021+0000","last_active":"2024-09-16T04:55:34.627995+0000","last_peered":"2024-09-16T04:55:34.627995+0000","last_clean":"2024-09-16T04:55:34.627995+0000","last_became_active":"2024-09-16T04:53:15.984991+0000","last_became_peered":"2024-09-16T04:53:15.984991+0000","last_unstale":"2024-09-16T04:55:34.627995+0000","last_undegraded":"2024-09-16T04:55:34.627995+0000","last_fullsized":"2024-09-16T04:55:34.627995+0000","mapping_epoch":40,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12598","last_scrub_stamp":"2024-09-16T04:55:34.627890+0000","last_deep_scrub":"37'12598","last_deep_scrub_stamp":"2024-09-16T04:55:34.627890+0000","last_clean_scrub_stamp":"2024-09-16T04:55:34.627890+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:11:44.311753+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12783","reported_seq":17421,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:34.933656+0000","last_change":"2024-09-16T04:55:34.933681+0000","last_active":"2024-09-16T04:55:34.933656+0000","last_peered":"2024-09-16T04:55:34.933656+0000","last_clean":"2024-09-16T04:55:34.933656+0000","last_became_active":"2024-09-16T04:44:12.024099+0000","last_became_peered":"2024-09-16T04:44:12.024099+0000","last_unstale":"2024-09-16T04:55:34.933656+0000","last_undegraded":"2024-09-16T04:55:34.933656+0000","last_fullsized":"2024-09-16T04:55:34.933656+0000","mapping_epoch":32,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12783","last_scrub_stamp":"2024-09-16T04:55:34.933566+0000","last_deep_scrub":"37'12783","last_deep_scrub_stamp":"2024-09-16T04:55:34.933566+0000","last_clean_scrub_stamp":"2024-09-16T04:55:34.933566+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:15.314129+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12752","reported_seq":17586,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:35.561875+0000","last_change":"2024-09-16T04:55:35.561923+0000","last_active":"2024-09-16T04:55:35.561875+0000","last_peered":"2024-09-16T04:55:35.561875+0000","last_clean":"2024-09-16T04:55:35.561875+0000","last_became_active":"2024-09-16T04:47:52.411653+0000","last_became_peered":"2024-09-16T04:47:52.411653+0000","last_unstale":"2024-09-16T04:55:35.561875+0000","last_undegraded":"2024-09-16T04:55:35.561875+0000","last_fullsized":"2024-09-16T04:55:35.561875+0000","mapping_epoch":34,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12752","last_scrub_stamp":"2024-09-16T04:55:35.561749+0000","last_deep_scrub":"37'12752","last_deep_scrub_stamp":"2024-09-16T04:55:35.561749+0000","last_clean_scrub_stamp":"2024-09-16T04:55:35.561749+0000","objects_scrubbed":1,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:53:54.866848+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12756","reported_seq":17330,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:35.670764+0000","last_change":"2024-09-16T04:55:35.670788+0000","last_active":"2024-09-16T04:55:35.670764+0000","last_peered":"2024-09-16T04:55:35.670764+0000","last_clean":"2024-09-16T04:55:35.670764+0000","last_became_active":"2024-09-16T04:54:27.217363+0000","last_became_peered":"2024-09-16T04:54:27.217363+0000","last_unstale":"2024-09-16T04:55:35.670764+0000","last_undegraded":"2024-09-16T04:55:35.670764+0000","last_fullsized":"2024-09-16T04:55:35.670764+0000","mapping_epoch":52,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12756","last_scrub_stamp":"2024-09-16T04:55:35.670672+0000","last_deep_scrub":"37'12756","last_deep_scrub_stamp":"2024-09-16T04:55:35.670672+0000","last_clean_scrub_stamp":"2024-09-16T04:55:35.670672+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:20.030295+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12488","reported_seq":17022,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:38.602906+0000","last_change":"2024-09-16T04:55:38.602935+0000","last_active":"2024-09-16T04:55:38.602906+0000","last_peered":"2024-09-16T04:55:38.602906+0000","last_clean":"2024-09-16T04:55:38.602906+0000","last_became_active":"2024-09-16T04:55:14.409783+0000","last_became_peered":"2024-09-16T04:55:14.409783+0000","last_unstale":"2024-09-16T04:55:38.602906+0000","last_undegraded":"2024-09-16T04:55:38.602906+0000","last_fullsized":"2024-09-16T04:55:38.602906+0000","mapping_epoch":60,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12488","last_scrub_stamp":"2024-09-16T04:55:38.602780+0000","last_deep_scrub":"37'12488","last_deep_scrub_stamp":"2024-09-16T04:55:38.602780+0000","last_clean_scrub_stamp":"2024-09-16T04:55:38.602780+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:15:30.441163+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12475,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:40.819206+0000","last_change":"2024-09-16T04:55:40.819251+0000","last_active":"2024-09-16T04:55:40.819206+0000","last_peered":"2024-09-16T04:55:40.819206+0000","last_clean":"2024-09-16T04:55:40.819206+0000","last_became_active":"2024-09-16T04:54:38.290243+0000","last_became_peered":"2024-09-16T04:54:38.290243+0000","last_unstale":"2024-09-16T04:55:40.819206+0000","last_undegraded":"2024-09-16T04:55:40.819206+0000","last_fullsized":"2024-09-16T04:55:40.819206+0000","mapping_epoch":54,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12330","last_scrub_stamp":"2024-09-16T04:55:40.819087+0000","last_deep_scrub":"37'12330","last_deep_scrub_stamp":"2024-09-16T04:55:40.819087+0000","last_clean_scrub_stamp":"2024-09-16T04:55:40.819087+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:01.587399+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12532","reported_seq":12689,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:38.830657+0000","last_change":"2024-09-16T04:55:38.830685+0000","last_active":"2024-09-16T04:55:38.830657+0000","last_peered":"2024-09-16T04:55:38.830657+0000","last_clean":"2024-09-16T04:55:38.830657+0000","last_became_active":"2024-09-16T04:53:52.094651+0000","last_became_peered":"2024-09-16T04:53:52.094651+0000","last_unstale":"2024-09-16T04:55:38.830657+0000","last_undegraded":"2024-09-16T04:55:38.830657+0000","last_fullsized":"2024-09-16T04:55:38.830657+0000","mapping_epoch":46,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12532","last_scrub_stamp":"2024-09-16T04:55:38.830536+0000","last_deep_scrub":"37'12532","last_deep_scrub_stamp":"2024-09-16T04:55:38.830536+0000","last_clean_scrub_stamp":"2024-09-16T04:55:38.830536+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:21.493766+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12780","reported_seq":12949,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:41.790658+0000","last_change":"2024-09-16T04:55:41.790686+0000","last_active":"2024-09-16T04:55:41.790658+0000","last_peered":"2024-09-16T04:55:41.790658+0000","last_clean":"2024-09-16T04:55:41.790658+0000","last_became_active":"2024-09-16T04:55:02.365337+0000","last_became_peered":"2024-09-16T04:55:02.365337+0000","last_unstale":"2024-09-16T04:55:41.790658+0000","last_undegraded":"2024-09-16T04:55:41.790658+0000","last_fullsized":"2024-09-16T04:55:41.790658+0000","mapping_epoch":58,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12780","last_scrub_stamp":"2024-09-16T04:55:41.790548+0000","last_deep_scrub":"37'12780","last_deep_scrub_stamp":"2024-09-16T04:55:41.790548+0000","last_clean_scrub_stamp":"2024-09-16T04:55:41.790548+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:56:52.922971+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12447","reported_seq":14995,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:36.873060+0000","last_change":"2024-09-16T04:55:36.873087+0000","last_active":"2024-09-16T04:55:36.873060+0000","last_peered":"2024-09-16T04:55:36.873060+0000","last_clean":"2024-09-16T04:55:36.873060+0000","last_became_active":"2024-09-16T04:53:03.951644+0000","last_became_peered":"2024-09-16T04:53:03.951644+0000","last_unstale":"2024-09-16T04:55:36.873060+0000","last_undegraded":"2024-09-16T04:55:36.873060+0000","last_fullsized":"2024-09-16T04:55:36.873060+0000","mapping_epoch":38,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12447","last_scrub_stamp":"2024-09-16T04:55:36.872952+0000","last_deep_scrub":"37'12447","last_deep_scrub_stamp":"2024-09-16T04:55:36.872952+0000","last_clean_scrub_stamp":"2024-09-16T04:55:36.872952+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:24:25.020812+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12305","reported_seq":12464,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T04:55:39.789683+0000","last_change":"2024-09-16T04:55:39.789717+0000","last_active":"2024-09-16T04:55:39.789683+0000","last_peered":"2024-09-16T04:55:39.789683+0000","last_clean":"2024-09-16T04:55:39.789683+0000","last_became_active":"2024-09-16T04:54:03.139422+0000","last_became_peered":"2024-09-16T04:54:03.139422+0000","last_unstale":"2024-09-16T04:55:39.789683+0000","last_undegraded":"2024-09-16T04:55:39.789683+0000","last_fullsized":"2024-09-16T04:55:39.789683+0000","mapping_epoch":48,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12305","last_scrub_stamp":"2024-09-16T04:55:39.789580+0000","last_deep_scrub":"37'12305","last_deep_scrub_stamp":"2024-09-16T04:55:39.789580+0000","last_clean_scrub_stamp":"2024-09-16T04:55:39.789580+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:13:09.249609+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24966,"num_write_kb":99864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4427,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722196,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721659,"kb_avail":93649644,"statfs":{"total":96636764160,"available":95897235456,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":738979535},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731412,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93640428,"statfs":{"total":96636764160,"available":95887798272,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627796,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93744044,"statfs":{"total":96636764160,"available":95993901056,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":642313935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247995,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722132,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":721595,"kb_avail":93649708,"statfs":{"total":96636764160,"available":95897300992,"internally_reserved":0,"allocated":528384,"data_stored":202150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":738913998},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:55:55.730 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T04:55:56.027 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T04:55:56.027 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T04:55:56.027 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T04:55:56.027 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:56:01.411 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-16T04:56:02.129 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T04:56:02.129 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T04:56:02.129 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:56:02.212 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T04:56:02.212 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T04:56:02.212 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:56:06.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-16T04:56:06.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-16T04:56:08.313 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T04:56:08.314 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T04:56:08.314 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:56:12.318 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-16T04:56:12.319 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-16T04:56:14.415 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T04:56:14.415 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T04:56:14.416 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T04:56:14.416 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:56:14.447 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T04:56:14.447 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T04:56:14.447 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:56:14.447 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:56:14.501 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:56:14.502 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T04:56:14.502 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:56:14.555 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T04:56:14.556 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:56:14.556 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:56:14.609 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:56:14.609 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T04:56:14.609 DEBUG:teuthology.orchestra.run.smithi152:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2024-09-16T04:56:14.778 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T04:56:14.778 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T04:56:15.338 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T04:56:15.338 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T04:56:15.461 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T04:56:15.461 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T04:56:15.549 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T04:56:15.549 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T04:56:15.624 INFO:tasks.ceph:Archiving mon data... 2024-09-16T04:56:15.625 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906037/data/mon.a.tgz 2024-09-16T04:56:15.626 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-09-16T04:56:15.651 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.DxWvCNGPjm 2024-09-16T04:56:15.652 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.DxWvCNGPjm 2024-09-16T04:56:15.817 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.DxWvCNGPjm 2024-09-16T04:56:15.926 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.DxWvCNGPjm is 304KB 2024-09-16T04:56:16.024 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.DxWvCNGPjm 2024-09-16T04:56:16.039 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906037/data/mon.b.tgz 2024-09-16T04:56:16.039 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-09-16T04:56:16.095 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.O2yVYWWEN6 2024-09-16T04:56:16.095 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.O2yVYWWEN6 2024-09-16T04:56:16.212 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.O2yVYWWEN6 2024-09-16T04:56:16.295 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.O2yVYWWEN6 is 309KB 2024-09-16T04:56:16.391 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.O2yVYWWEN6 2024-09-16T04:56:16.405 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906037/data/mon.c.tgz 2024-09-16T04:56:16.406 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-09-16T04:56:16.460 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.M1NO98oEva 2024-09-16T04:56:16.460 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.M1NO98oEva 2024-09-16T04:56:16.578 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.M1NO98oEva 2024-09-16T04:56:16.662 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.M1NO98oEva is 309KB 2024-09-16T04:56:16.759 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.M1NO98oEva 2024-09-16T04:56:16.774 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T04:56:16.774 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:56:16.917 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:56:16.917 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T04:56:16.918 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906037/remote/smithi152/crash 2024-09-16T04:56:16.919 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:56:16.981 INFO:tasks.ceph:Compressing logs... 2024-09-16T04:56:16.981 DEBUG:teuthology.orchestra.run.smithi152:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:56:17.323 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-16T04:56:17.324 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T04:56:17.325 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65661.log 2024-09-16T04:56:17.325 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42867.log 2024-09-16T04:56:17.325 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57534.log 2024-09-16T04:56:17.325 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65661.log.gz 2024-09-16T04:56:17.325 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.42867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67173.log 2024-09-16T04:56:17.325 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42867.log.gz 2024-09-16T04:56:17.325 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54846.log 2024-09-16T04:56:17.325 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57534.log.gz 2024-09-16T04:56:17.325 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65325.log 2024-09-16T04:56:17.325 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67173.log.gz 2024-09-16T04:56:17.325 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.54846.log: /var/log/ceph/ceph-client.admin.73351.log 2024-09-16T04:56:17.325 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54846.log.gz 2024-09-16T04:56:17.325 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50016.log 2024-09-16T04:56:17.326 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65325.log.gz 2024-09-16T04:56:17.326 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56568.log 2024-09-16T04:56:17.326 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73351.log.gz 2024-09-16T04:56:17.326 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49134.log 2024-09-16T04:56:17.326 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50016.log.gz 2024-09-16T04:56:17.326 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37103.log 2024-09-16T04:56:17.326 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56568.log.gz 2024-09-16T04:56:17.326 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41467.log 2024-09-16T04:56:17.326 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49134.log.gz 2024-09-16T04:56:17.326 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43745.log/var/log/ceph/ceph-client.admin.37103.log: 2024-09-16T04:56:17.326 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37103.log.gz 2024-09-16T04:56:17.326 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.41467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68685.log 2024-09-16T04:56:17.326 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41467.log.gz 2024-09-16T04:56:17.326 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45988.log 2024-09-16T04:56:17.327 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43745.log.gz 2024-09-16T04:56:17.327 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56736.log 2024-09-16T04:56:17.327 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68685.log.gz 2024-09-16T04:56:17.327 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59554.log 2024-09-16T04:56:17.327 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45988.log.gz 2024-09-16T04:56:17.327 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56610.log 2024-09-16T04:56:17.327 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56736.log.gz 2024-09-16T04:56:17.327 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57618.log 2024-09-16T04:56:17.327 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59554.log.gz 2024-09-16T04:56:17.327 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62299.log 2024-09-16T04:56:17.327 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56610.log.gz 2024-09-16T04:56:17.327 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52788.log 2024-09-16T04:56:17.327 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57618.log.gz 2024-09-16T04:56:17.327 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67845.log 2024-09-16T04:56:17.328 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62299.log.gz 2024-09-16T04:56:17.328 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57114.log 2024-09-16T04:56:17.328 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52788.log.gz 2024-09-16T04:56:17.328 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37461.log 2024-09-16T04:56:17.328 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67845.log.gz 2024-09-16T04:56:17.328 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33107.log 2024-09-16T04:56:17.328 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57114.log.gz 2024-09-16T04:56:17.328 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60745.log 2024-09-16T04:56:17.328 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37461.log.gz 2024-09-16T04:56:17.328 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.33107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68055.log 2024-09-16T04:56:17.328 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33107.log.gz 2024-09-16T04:56:17.328 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59764.log 2024-09-16T04:56:17.328 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60745.log.gz 2024-09-16T04:56:17.328 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56232.log 2024-09-16T04:56:17.329 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68055.log.gz 2024-09-16T04:56:17.329 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51318.log 2024-09-16T04:56:17.329 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59764.log.gz 2024-09-16T04:56:17.329 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40266.log 2024-09-16T04:56:17.329 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56232.log.gz 2024-09-16T04:56:17.329 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58962.log 2024-09-16T04:56:17.329 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51318.log.gz 2024-09-16T04:56:17.329 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.40266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55812.log 2024-09-16T04:56:17.329 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40266.log.gz 2024-09-16T04:56:17.329 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37982.log 2024-09-16T04:56:17.329 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58962.log.gz 2024-09-16T04:56:17.329 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55812.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.55812.log.gz /var/log/ceph/ceph-client.admin.63645.log 2024-09-16T04:56:17.329 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T04:56:17.329 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71458.log 2024-09-16T04:56:17.330 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37982.log.gz 2024-09-16T04:56:17.330 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64149.log 2024-09-16T04:56:17.330 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63645.log.gz 2024-09-16T04:56:17.330 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69526.log 2024-09-16T04:56:17.330 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71458.log.gz 2024-09-16T04:56:17.330 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69442.log 2024-09-16T04:56:17.330 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64149.log.gz 2024-09-16T04:56:17.330 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58626.log 2024-09-16T04:56:17.330 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69526.log.gz 2024-09-16T04:56:17.330 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45618.log 2024-09-16T04:56:17.330 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69442.log.gz 2024-09-16T04:56:17.330 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58626.log.gz 2024-09-16T04:56:17.330 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63225.log 2024-09-16T04:56:17.330 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71122.log 2024-09-16T04:56:17.331 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45618.log.gz 2024-09-16T04:56:17.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45359.log 2024-09-16T04:56:17.331 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63225.log.gz 2024-09-16T04:56:17.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60913.log 2024-09-16T04:56:17.331 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71122.log.gz 2024-09-16T04:56:17.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50226.log 2024-09-16T04:56:17.331 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45359.log.gz 2024-09-16T04:56:17.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55056.log 2024-09-16T04:56:17.331 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60913.log.gz 2024-09-16T04:56:17.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57072.log 2024-09-16T04:56:17.331 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50226.log.gz 2024-09-16T04:56:17.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40155.log 2024-09-16T04:56:17.331 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55056.log.gz 2024-09-16T04:56:17.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50646.log 2024-09-16T04:56:17.332 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57072.log.gz 2024-09-16T04:56:17.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.40155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51990.log 2024-09-16T04:56:17.332 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40155.log.gz 2024-09-16T04:56:17.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57660.log 2024-09-16T04:56:17.332 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50646.log.gz 2024-09-16T04:56:17.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39822.log 2024-09-16T04:56:17.332 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51990.log.gz 2024-09-16T04:56:17.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52242.log 2024-09-16T04:56:17.332 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57660.log.gz 2024-09-16T04:56:17.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71921.log 2024-09-16T04:56:17.332 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39822.log.gz 2024-09-16T04:56:17.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50940.log 2024-09-16T04:56:17.332 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52242.log.gz 2024-09-16T04:56:17.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62468.log 2024-09-16T04:56:17.333 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71921.log.gz 2024-09-16T04:56:17.333 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54762.log 2024-09-16T04:56:17.333 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50940.log.gz 2024-09-16T04:56:17.333 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62468.log.gz 2024-09-16T04:56:17.333 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48294.log 2024-09-16T04:56:17.333 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53040.log 2024-09-16T04:56:17.333 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54762.log.gz 2024-09-16T04:56:17.333 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61669.log 2024-09-16T04:56:17.333 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48294.log.gz 2024-09-16T04:56:17.333 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58038.log 2024-09-16T04:56:17.333 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53040.log.gz 2024-09-16T04:56:17.333 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63939.log 2024-09-16T04:56:17.333 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61669.log.gz 2024-09-16T04:56:17.333 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56484.log 2024-09-16T04:56:17.334 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58038.log.gz 2024-09-16T04:56:17.334 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43368.log 2024-09-16T04:56:17.334 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63939.log.gz 2024-09-16T04:56:17.334 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59947.log 2024-09-16T04:56:17.334 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56484.log.gz 2024-09-16T04:56:17.334 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38401.log 2024-09-16T04:56:17.334 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43368.log.gz 2024-09-16T04:56:17.334 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50898.log 2024-09-16T04:56:17.334 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59947.log.gz 2024-09-16T04:56:17.334 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71038.log 2024-09-16T04:56:17.334 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38401.log.gz 2024-09-16T04:56:17.334 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64653.log 2024-09-16T04:56:17.334 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50898.log.gz 2024-09-16T04:56:17.334 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65997.log 2024-09-16T04:56:17.335 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71038.log.gz 2024-09-16T04:56:17.335 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44795.log 2024-09-16T04:56:17.335 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64653.log.gz 2024-09-16T04:56:17.335 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58500.log 2024-09-16T04:56:17.335 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65997.log.gz 2024-09-16T04:56:17.335 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53124.log 2024-09-16T04:56:17.335 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44795.log.gz 2024-09-16T04:56:17.335 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58500.log.gz 2024-09-16T04:56:17.335 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44375.log 2024-09-16T04:56:17.335 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56064.log 2024-09-16T04:56:17.335 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53124.log.gz 2024-09-16T04:56:17.335 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/ceph-client.admin.44375.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52074.log 2024-09-16T04:56:17.335 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44375.log.gz 2024-09-16T04:56:17.335 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51528.log 2024-09-16T04:56:17.336 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56064.log.gz 2024-09-16T04:56:17.336 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61879.log 2024-09-16T04:56:17.336 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52074.log.gz 2024-09-16T04:56:17.336 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44039.log 2024-09-16T04:56:17.336 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51528.log.gz 2024-09-16T04:56:17.336 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51612.log 2024-09-16T04:56:17.336 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61879.log.gz 2024-09-16T04:56:17.336 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39178.log 2024-09-16T04:56:17.336 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44039.log.gz 2024-09-16T04:56:17.336 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72173.log 2024-09-16T04:56:17.336 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51612.log.gz 2024-09-16T04:56:17.336 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55350.log 2024-09-16T04:56:17.336 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39178.log.gz 2024-09-16T04:56:17.336 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63309.log 2024-09-16T04:56:17.337 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72173.log.gz 2024-09-16T04:56:17.337 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46136.log 2024-09-16T04:56:17.337 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55350.log.gz 2024-09-16T04:56:17.337 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62257.log 2024-09-16T04:56:17.337 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63309.log.gz 2024-09-16T04:56:17.337 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.46136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39785.log 2024-09-16T04:56:17.337 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46136.log.gz 2024-09-16T04:56:17.337 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45045.log 2024-09-16T04:56:17.337 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62257.log.gz 2024-09-16T04:56:17.337 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54720.log 2024-09-16T04:56:17.337 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39785.log.gz 2024-09-16T04:56:17.337 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47328.log 2024-09-16T04:56:17.337 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45045.log.gz 2024-09-16T04:56:17.337 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52494.log 2024-09-16T04:56:17.338 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54720.log.gz 2024-09-16T04:56:17.338 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59004.log 2024-09-16T04:56:17.338 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47328.log.gz 2024-09-16T04:56:17.338 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67341.log 2024-09-16T04:56:17.338 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52494.log.gz 2024-09-16T04:56:17.338 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54006.log 2024-09-16T04:56:17.338 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59004.log.gz 2024-09-16T04:56:17.338 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49512.log 2024-09-16T04:56:17.338 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67341.log.gz 2024-09-16T04:56:17.338 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52452.log 2024-09-16T04:56:17.338 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54006.log.gz 2024-09-16T04:56:17.338 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58542.log 2024-09-16T04:56:17.338 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49512.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49512.log.gz 2024-09-16T04:56:17.338 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55434.log 2024-09-16T04:56:17.339 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52452.log.gz 2024-09-16T04:56:17.339 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43535.log 2024-09-16T04:56:17.339 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58542.log.gz 2024-09-16T04:56:17.339 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56904.log 2024-09-16T04:56:17.339 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55434.log.gz 2024-09-16T04:56:17.339 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69904.log 2024-09-16T04:56:17.339 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43535.log.gz 2024-09-16T04:56:17.339 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37800.log 2024-09-16T04:56:17.339 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56904.log.gz 2024-09-16T04:56:17.339 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64107.log 2024-09-16T04:56:17.339 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69904.log.gz 2024-09-16T04:56:17.339 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60115.log 2024-09-16T04:56:17.339 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37800.log.gz 2024-09-16T04:56:17.339 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54384.log 2024-09-16T04:56:17.340 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64107.log.gz 2024-09-16T04:56:17.340 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48000.log 2024-09-16T04:56:17.340 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60115.log.gz 2024-09-16T04:56:17.340 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43619.log 2024-09-16T04:56:17.340 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54384.log.gz 2024-09-16T04:56:17.340 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74065.log 2024-09-16T04:56:17.340 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48000.log.gz 2024-09-16T04:56:17.340 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72425.log 2024-09-16T04:56:17.340 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43619.log.gz 2024-09-16T04:56:17.340 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.74065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42741.log 2024-09-16T04:56:17.340 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74065.log.gz 2024-09-16T04:56:17.340 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72425.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T04:56:17.340 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72425.log.gz 2024-09-16T04:56:17.341 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.42741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67971.log 2024-09-16T04:56:17.341 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42741.log.gz 2024-09-16T04:56:17.341 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50352.log 2024-09-16T04:56:17.341 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47706.log 2024-09-16T04:56:17.341 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67971.log.gz 2024-09-16T04:56:17.341 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60871.log 2024-09-16T04:56:17.341 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50352.log.gz 2024-09-16T04:56:17.341 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66837.log 2024-09-16T04:56:17.341 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47706.log.gz 2024-09-16T04:56:17.341 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51444.log 2024-09-16T04:56:17.341 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60871.log.gz 2024-09-16T04:56:17.341 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37629.log 2024-09-16T04:56:17.341 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66837.log.gz 2024-09-16T04:56:17.341 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64821.log 2024-09-16T04:56:17.342 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51444.log.gz 2024-09-16T04:56:17.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61375.log 2024-09-16T04:56:17.342 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37629.log.gz 2024-09-16T04:56:17.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50772.log 2024-09-16T04:56:17.342 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64821.log.gz 2024-09-16T04:56:17.342 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/ceph-client.admin.61375.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47664.log 2024-09-16T04:56:17.342 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61375.log.gz 2024-09-16T04:56:17.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66165.log 2024-09-16T04:56:17.342 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50772.log.gz 2024-09-16T04:56:17.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47664.log.gz 2024-09-16T04:56:17.342 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50562.log 2024-09-16T04:56:17.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47496.log 2024-09-16T04:56:17.342 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66165.log.gz 2024-09-16T04:56:17.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50562.log.gz 2024-09-16T04:56:17.343 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47874.log 2024-09-16T04:56:17.343 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45951.log 2024-09-16T04:56:17.343 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47496.log.gz 2024-09-16T04:56:17.343 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47874.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.50856.log -- replaced with /var/log/ceph/ceph-client.admin.47874.log.gz 2024-09-16T04:56:17.343 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T04:56:17.343 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70828.log 2024-09-16T04:56:17.343 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45951.log.gz 2024-09-16T04:56:17.343 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59258.log 2024-09-16T04:56:17.343 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50856.log.gz 2024-09-16T04:56:17.343 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32909.log 2024-09-16T04:56:17.343 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70828.log.gz 2024-09-16T04:56:17.343 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48756.log 2024-09-16T04:56:17.343 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59258.log.gz 2024-09-16T04:56:17.343 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.32909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58374.log 2024-09-16T04:56:17.344 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32909.log.gz 2024-09-16T04:56:17.344 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68895.log 2024-09-16T04:56:17.344 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48756.log.gz 2024-09-16T04:56:17.344 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58374.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.56862.log -- replaced with /var/log/ceph/ceph-client.admin.58374.log.gz 2024-09-16T04:56:17.344 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T04:56:17.344 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49680.log 2024-09-16T04:56:17.344 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68895.log.gz 2024-09-16T04:56:17.344 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56862.log.gz 2024-09-16T04:56:17.344 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61627.log 2024-09-16T04:56:17.344 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57786.log 2024-09-16T04:56:17.344 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49680.log.gz 2024-09-16T04:56:17.344 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44459.log 2024-09-16T04:56:17.344 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61627.log.gz 2024-09-16T04:56:17.345 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57786.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57786.log.gz 2024-09-16T04:56:17.345 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49008.log 2024-09-16T04:56:17.345 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56106.log 2024-09-16T04:56:17.345 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44459.log.gz 2024-09-16T04:56:17.345 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64947.log 2024-09-16T04:56:17.345 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49008.log.gz 2024-09-16T04:56:17.345 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60073.log 2024-09-16T04:56:17.345 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56106.log.gz 2024-09-16T04:56:17.345 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38067.log 2024-09-16T04:56:17.345 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64947.log.gz 2024-09-16T04:56:17.345 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62173.log 2024-09-16T04:56:17.345 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60073.log.gz 2024-09-16T04:56:17.345 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67635.log 2024-09-16T04:56:17.346 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38067.log.gz 2024-09-16T04:56:17.346 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65493.log 2024-09-16T04:56:17.346 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62173.log.gz 2024-09-16T04:56:17.346 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44207.log 2024-09-16T04:56:17.346 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67635.log.gz 2024-09-16T04:56:17.346 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49218.log 2024-09-16T04:56:17.346 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65493.log.gz 2024-09-16T04:56:17.346 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61963.log 2024-09-16T04:56:17.346 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44207.log.gz 2024-09-16T04:56:17.346 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60577.log 2024-09-16T04:56:17.346 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49218.log.gz 2024-09-16T04:56:17.346 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72889.log 2024-09-16T04:56:17.346 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61963.log.gz 2024-09-16T04:56:17.346 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61039.log 2024-09-16T04:56:17.347 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60577.log.gz 2024-09-16T04:56:17.347 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46099.log 2024-09-16T04:56:17.347 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72889.log.gz 2024-09-16T04:56:17.347 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45168.log 2024-09-16T04:56:17.347 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61039.log.gz 2024-09-16T04:56:17.347 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.46099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43871.log 2024-09-16T04:56:17.347 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46099.log.gz 2024-09-16T04:56:17.347 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73393.log 2024-09-16T04:56:17.347 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45168.log.gz 2024-09-16T04:56:17.347 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42781.log 2024-09-16T04:56:17.347 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43871.log.gz 2024-09-16T04:56:17.347 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40044.log 2024-09-16T04:56:17.347 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73393.log.gz 2024-09-16T04:56:17.347 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.42781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65073.log 2024-09-16T04:56:17.348 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42781.log.gz 2024-09-16T04:56:17.348 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.40044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66459.log 0.0% 2024-09-16T04:56:17.348 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40044.log.gz 2024-09-16T04:56:17.348 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55140.log 2024-09-16T04:56:17.348 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65073.log.gz 2024-09-16T04:56:17.348 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60997.log 2024-09-16T04:56:17.348 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66459.log.gz 2024-09-16T04:56:17.348 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65031.log 2024-09-16T04:56:17.348 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55140.log.gz 2024-09-16T04:56:17.348 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60493.log 2024-09-16T04:56:17.348 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60997.log.gz 2024-09-16T04:56:17.348 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58206.log 2024-09-16T04:56:17.348 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65031.log.gz 2024-09-16T04:56:17.349 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69148.log 2024-09-16T04:56:17.349 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60493.log.gz 2024-09-16T04:56:17.349 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63519.log 2024-09-16T04:56:17.349 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58206.log.gz 2024-09-16T04:56:17.349 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57912.log 2024-09-16T04:56:17.349 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69148.log.gz 2024-09-16T04:56:17.349 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43242.log 2024-09-16T04:56:17.349 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63519.log.gz 2024-09-16T04:56:17.349 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63897.log 2024-09-16T04:56:17.349 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57912.log.gz 2024-09-16T04:56:17.349 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68265.log 2024-09-16T04:56:17.349 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43242.log.gz 2024-09-16T04:56:17.349 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48336.log 2024-09-16T04:56:17.349 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63897.log.gz 2024-09-16T04:56:17.350 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64695.log 2024-09-16T04:56:17.350 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68265.log.gz 2024-09-16T04:56:17.350 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36972.log 2024-09-16T04:56:17.350 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48336.log.gz 2024-09-16T04:56:17.350 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52914.log 2024-09-16T04:56:17.350 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64695.log.gz 2024-09-16T04:56:17.350 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.36972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72299.log 2024-09-16T04:56:17.350 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36972.log.gz 2024-09-16T04:56:17.350 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70240.log 2024-09-16T04:56:17.350 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52914.log.gz 2024-09-16T04:56:17.350 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33195.log 2024-09-16T04:56:17.350 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72299.log.gz 2024-09-16T04:56:17.350 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69358.log 2024-09-16T04:56:17.350 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70240.log.gz 2024-09-16T04:56:17.351 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.33195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59806.log 2024-09-16T04:56:17.351 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33195.log.gz 2024-09-16T04:56:17.351 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53586.log 2024-09-16T04:56:17.351 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69358.log.gz 2024-09-16T04:56:17.351 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45766.log 2024-09-16T04:56:17.351 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59806.log.gz 2024-09-16T04:56:17.351 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57870.log 2024-09-16T04:56:17.351 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53586.log.gz 2024-09-16T04:56:17.351 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54510.log 2024-09-16T04:56:17.351 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45766.log.gz 2024-09-16T04:56:17.351 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37271.log 2024-09-16T04:56:17.351 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57870.log.gz 2024-09-16T04:56:17.351 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52704.log 2024-09-16T04:56:17.351 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54510.log.gz 2024-09-16T04:56:17.352 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61249.log 2024-09-16T04:56:17.352 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37271.log.gz 2024-09-16T04:56:17.352 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38586.log 2024-09-16T04:56:17.352 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52704.log.gz 2024-09-16T04:56:17.352 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60955.log 2024-09-16T04:56:17.352 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61249.log.gz 2024-09-16T04:56:17.352 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65787.log 2024-09-16T04:56:17.352 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38586.log.gz 2024-09-16T04:56:17.352 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62511.log 2024-09-16T04:56:17.352 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60955.log.gz 2024-09-16T04:56:17.352 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64569.log 2024-09-16T04:56:17.352 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65787.log.gz 2024-09-16T04:56:17.352 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65199.log 2024-09-16T04:56:17.353 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62511.log.gz 2024-09-16T04:56:17.353 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68937.log 2024-09-16T04:56:17.353 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64569.log.gz 2024-09-16T04:56:17.353 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42655.log 2024-09-16T04:56:17.353 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65199.log.gz 2024-09-16T04:56:17.353 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54636.log 2024-09-16T04:56:17.353 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68937.log.gz 2024-09-16T04:56:17.353 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.42655.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42655.log.gz 2024-09-16T04:56:17.353 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47286.log 2024-09-16T04:56:17.353 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59849.log 2024-09-16T04:56:17.353 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54636.log.gz 2024-09-16T04:56:17.353 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48462.log 2024-09-16T04:56:17.353 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47286.log.gz 2024-09-16T04:56:17.353 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49764.log 2024-09-16T04:56:17.354 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59849.log.gz 2024-09-16T04:56:17.354 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50436.log 2024-09-16T04:56:17.354 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48462.log.gz 2024-09-16T04:56:17.354 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62763.log 2024-09-16T04:56:17.354 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49764.log.gz 2024-09-16T04:56:17.354 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51822.log 2024-09-16T04:56:17.354 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50436.log.gz 2024-09-16T04:56:17.354 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70408.log 2024-09-16T04:56:17.354 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62763.log.gz 2024-09-16T04:56:17.354 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51822.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51822.log.gz -5 2024-09-16T04:56:17.354 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/ceph-client.admin.59342.log 2024-09-16T04:56:17.354 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64443.log 2024-09-16T04:56:17.354 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70408.log.gz 2024-09-16T04:56:17.354 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59342.log.gz 2024-09-16T04:56:17.354 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50730.log 2024-09-16T04:56:17.355 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64443.log.gz 2024-09-16T04:56:17.355 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51738.log 2024-09-16T04:56:17.355 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50730.log.gz 2024-09-16T04:56:17.355 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32989.log 2024-09-16T04:56:17.355 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58710.log 2024-09-16T04:56:17.355 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51738.log.gz 2024-09-16T04:56:17.355 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.32989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32989.log.gz 2024-09-16T04:56:17.355 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45396.log 2024-09-16T04:56:17.355 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37503.log 2024-09-16T04:56:17.355 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58710.log.gz 2024-09-16T04:56:17.355 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45396.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.45396.log.gz -- 2024-09-16T04:56:17.355 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/ceph-client.admin.40007.log 2024-09-16T04:56:17.355 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54468.log 2024-09-16T04:56:17.355 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37503.log.gz 2024-09-16T04:56:17.356 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.40007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40007.log.gz 2024-09-16T04:56:17.356 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58794.log 2024-09-16T04:56:17.356 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49428.log 2024-09-16T04:56:17.356 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54468.log.gz 2024-09-16T04:56:17.356 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58794.log.gzgzip 2024-09-16T04:56:17.356 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48588.log 2024-09-16T04:56:17.356 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46210.log 2024-09-16T04:56:17.356 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49428.log.gz 2024-09-16T04:56:17.356 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48588.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.48588.log.gz -- 2024-09-16T04:56:17.356 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/ceph-client.admin.48966.log 2024-09-16T04:56:17.356 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.46210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73141.log 2024-09-16T04:56:17.356 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46210.log.gz 2024-09-16T04:56:17.356 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48966.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.48966.log.gz --verbose 2024-09-16T04:56:17.357 INFO:teuthology.orchestra.run.smithi152.stderr: -- /var/log/ceph/ceph-client.admin.63267.log 2024-09-16T04:56:17.357 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60157.log 2024-09-16T04:56:17.357 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73141.log.gz 2024-09-16T04:56:17.357 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52368.log 2024-09-16T04:56:17.357 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63267.log.gz 2024-09-16T04:56:17.357 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55476.log 2024-09-16T04:56:17.357 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60157.log.gz 2024-09-16T04:56:17.357 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47790.log 2024-09-16T04:56:17.357 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52368.log.gz 2024-09-16T04:56:17.357 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55476.log.gzgzip 2024-09-16T04:56:17.357 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60367.log 2024-09-16T04:56:17.357 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51696.log 2024-09-16T04:56:17.357 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47790.log.gz 2024-09-16T04:56:17.357 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38066.log 2024-09-16T04:56:17.358 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60367.log.gz 2024-09-16T04:56:17.358 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67761.log 2024-09-16T04:56:17.358 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51696.log.gz 2024-09-16T04:56:17.358 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38623.log 2024-09-16T04:56:17.358 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38066.log.gz 2024-09-16T04:56:17.358 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62679.log 2024-09-16T04:56:17.358 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67761.log.gz 2024-09-16T04:56:17.358 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43661.log 2024-09-16T04:56:17.358 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38623.log.gz 2024-09-16T04:56:17.358 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44837.log 2024-09-16T04:56:17.358 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62679.log.gz 2024-09-16T04:56:17.358 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66795.log 2024-09-16T04:56:17.358 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43661.log.gz 2024-09-16T04:56:17.358 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44837.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.56694.log 2024-09-16T04:56:17.359 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44837.log.gz 2024-09-16T04:56:17.359 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45544.log 2024-09-16T04:56:17.359 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66795.log.gz 2024-09-16T04:56:17.359 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46284.log 2024-09-16T04:56:17.359 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56694.log.gz 2024-09-16T04:56:17.359 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38734.log 2024-09-16T04:56:17.359 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45544.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45544.log.gz 2024-09-16T04:56:17.359 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45581.log 2024-09-16T04:56:17.359 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.46284.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46284.log.gz 2024-09-16T04:56:17.359 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63099.log 2024-09-16T04:56:17.359 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38734.log.gz 2024-09-16T04:56:17.359 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50478.log 2024-09-16T04:56:17.359 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45581.log.gz 2024-09-16T04:56:17.359 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56274.log 2024-09-16T04:56:17.360 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63099.log.gz 2024-09-16T04:56:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71164.log 2024-09-16T04:56:17.360 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50478.log.gz 2024-09-16T04:56:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56274.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.73225.log -- replaced with /var/log/ceph/ceph-client.admin.56274.log.gz 2024-09-16T04:56:17.360 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T04:56:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69316.log 2024-09-16T04:56:17.360 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71164.log.gz 2024-09-16T04:56:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73225.log.gz 2024-09-16T04:56:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47832.log 2024-09-16T04:56:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48924.log 2024-09-16T04:56:17.360 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69316.log.gz 2024-09-16T04:56:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47832.log.gz 2024-09-16T04:56:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68979.log 2024-09-16T04:56:17.360 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64191.log 2024-09-16T04:56:17.361 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48924.log.gz 2024-09-16T04:56:17.361 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68979.log.gz 2024-09-16T04:56:17.361 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55938.log 2024-09-16T04:56:17.361 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55644.log 2024-09-16T04:56:17.361 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64191.log.gz 2024-09-16T04:56:17.361 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55938.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.62805.log -- replaced with /var/log/ceph/ceph-client.admin.55938.log.gz 2024-09-16T04:56:17.361 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T04:56:17.361 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64233.log 2024-09-16T04:56:17.361 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55644.log.gz 2024-09-16T04:56:17.361 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62805.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62805.log.gzgzip 2024-09-16T04:56:17.361 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.69274.log 2024-09-16T04:56:17.361 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53334.log 2024-09-16T04:56:17.361 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64233.log.gz 2024-09-16T04:56:17.361 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69274.log.gz 2024-09-16T04:56:17.362 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43203.log 2024-09-16T04:56:17.362 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39289.log 2024-09-16T04:56:17.362 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53334.log.gz 2024-09-16T04:56:17.362 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43203.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43203.log.gz -5 2024-09-16T04:56:17.362 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/ceph-client.admin.65577.log 2024-09-16T04:56:17.362 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49386.log 2024-09-16T04:56:17.362 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39289.log.gz 2024-09-16T04:56:17.362 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65577.log.gz 2024-09-16T04:56:17.362 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63393.log 2024-09-16T04:56:17.362 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43077.log 2024-09-16T04:56:17.362 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49386.log.gz 2024-09-16T04:56:17.362 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63393.log.gz 2024-09-16T04:56:17.362 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58080.log 2024-09-16T04:56:17.362 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57702.log 2024-09-16T04:56:17.362 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43077.log.gz 2024-09-16T04:56:17.363 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68517.log 2024-09-16T04:56:17.363 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58080.log.gz 2024-09-16T04:56:17.363 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67467.log 2024-09-16T04:56:17.363 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57702.log.gz 2024-09-16T04:56:17.363 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57954.log 2024-09-16T04:56:17.363 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68517.log.gz 2024-09-16T04:56:17.363 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62595.log 2024-09-16T04:56:17.363 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67467.log.gz 2024-09-16T04:56:17.363 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57954.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57954.log.gz 2024-09-16T04:56:17.363 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67593.log 2024-09-16T04:56:17.363 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62595.log.gz 2024-09-16T04:56:17.363 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60661.log 2024-09-16T04:56:17.363 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72131.log 2024-09-16T04:56:17.363 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67593.log.gz 2024-09-16T04:56:17.364 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51066.log 2024-09-16T04:56:17.364 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60661.log.gz 2024-09-16T04:56:17.364 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66921.log 2024-09-16T04:56:17.364 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72131.log.gz 2024-09-16T04:56:17.364 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51066.log.gz 2024-09-16T04:56:17.364 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40229.log 2024-09-16T04:56:17.364 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44291.log 2024-09-16T04:56:17.364 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66921.log.gz 2024-09-16T04:56:17.364 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.40229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61501.log 2024-09-16T04:56:17.364 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40229.log.gz 2024-09-16T04:56:17.364 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71837.log 2024-09-16T04:56:17.364 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44291.log.gz 2024-09-16T04:56:17.364 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54342.log 2024-09-16T04:56:17.364 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61501.log.gz 2024-09-16T04:56:17.365 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71837.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.71837.log.gz -- 2024-09-16T04:56:17.365 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/ceph-client.admin.49722.log 2024-09-16T04:56:17.365 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54342.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26842.log 2024-09-16T04:56:17.365 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54342.log.gz 2024-09-16T04:56:17.365 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64023.log 2024-09-16T04:56:17.365 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49722.log.gz 2024-09-16T04:56:17.365 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph.tmp-client.admin.26842.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26842.log.gz 2024-09-16T04:56:17.365 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57030.log 2024-09-16T04:56:17.365 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37545.log 2024-09-16T04:56:17.365 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64023.log.gz 2024-09-16T04:56:17.365 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61291.log 2024-09-16T04:56:17.365 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57030.log.gz 2024-09-16T04:56:17.365 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39933.log 2024-09-16T04:56:17.366 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37545.log.gz 2024-09-16T04:56:17.366 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49554.log 2024-09-16T04:56:17.366 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61291.log.gz 2024-09-16T04:56:17.366 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40549.log 2024-09-16T04:56:17.366 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39933.log.gz 2024-09-16T04:56:17.366 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55392.log 2024-09-16T04:56:17.366 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49554.log.gz 2024-09-16T04:56:17.366 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.40549.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40549.log.gz -5 2024-09-16T04:56:17.366 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/ceph-client.admin.73771.log 2024-09-16T04:56:17.366 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62931.log 2024-09-16T04:56:17.366 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55392.log.gz 2024-09-16T04:56:17.366 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73771.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.73771.log.gz 2024-09-16T04:56:17.366 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.70870.log 2024-09-16T04:56:17.366 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66417.log 2024-09-16T04:56:17.367 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62931.log.gz 2024-09-16T04:56:17.367 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70870.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70870.log.gz -5 2024-09-16T04:56:17.367 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52578.log 2024-09-16T04:56:17.367 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61543.log 2024-09-16T04:56:17.367 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66417.log.gz 2024-09-16T04:56:17.367 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52578.log.gz 2024-09-16T04:56:17.367 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62425.log 2024-09-16T04:56:17.367 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66123.log 2024-09-16T04:56:17.367 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61543.log.gz 2024-09-16T04:56:17.367 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62425.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62425.log.gz 2024-09-16T04:56:17.367 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68223.log 2024-09-16T04:56:17.367 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70030.log 2024-09-16T04:56:17.367 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66123.log.gz 2024-09-16T04:56:17.367 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68223.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.51234.log 2024-09-16T04:56:17.368 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68223.log.gz 2024-09-16T04:56:17.368 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55014.log 2024-09-16T04:56:17.368 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70030.log.gz 2024-09-16T04:56:17.368 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51234.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.51234.log.gz --verbose 2024-09-16T04:56:17.368 INFO:teuthology.orchestra.run.smithi152.stderr: -- /var/log/ceph/ceph-client.admin.68349.log 2024-09-16T04:56:17.368 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47958.log 2024-09-16T04:56:17.368 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55014.log.gz 2024-09-16T04:56:17.368 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56820.log 2024-09-16T04:56:17.368 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68349.log.gz 2024-09-16T04:56:17.368 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61207.log 2024-09-16T04:56:17.368 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47958.log.gz 2024-09-16T04:56:17.368 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69652.log 2024-09-16T04:56:17.368 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56820.log.gz 2024-09-16T04:56:17.368 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58416.log 0.0% 2024-09-16T04:56:17.369 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61207.log.gz 2024-09-16T04:56:17.369 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73561.log 2024-09-16T04:56:17.369 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69652.log.gz 2024-09-16T04:56:17.369 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73267.log 2024-09-16T04:56:17.369 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58416.log.gz 2024-09-16T04:56:17.369 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42615.log 2024-09-16T04:56:17.369 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73561.log.gz 2024-09-16T04:56:17.369 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73267.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:56:17.369 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73267.log.gz 2024-09-16T04:56:17.369 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.42615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40414.log 2024-09-16T04:56:17.369 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42615.log.gz 2024-09-16T04:56:17.369 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42527.log 2024-09-16T04:56:17.369 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57240.log 2024-09-16T04:56:17.369 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.40414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40414.log.gz 2024-09-16T04:56:17.370 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.42527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48714.log 2024-09-16T04:56:17.370 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42527.log.gz 2024-09-16T04:56:17.370 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45131.log 2024-09-16T04:56:17.370 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57240.log.gz 2024-09-16T04:56:17.370 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49890.log 2024-09-16T04:56:17.370 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48714.log.gz 2024-09-16T04:56:17.370 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45131.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.45131.log.gz --verbose -- /var/log/ceph/ceph-client.admin.70702.log 2024-09-16T04:56:17.370 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T04:56:17.370 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49890.log.gz 2024-09-16T04:56:17.370 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70744.log 2024-09-16T04:56:17.370 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51402.log 2024-09-16T04:56:17.370 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70702.log.gz 2024-09-16T04:56:17.370 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59470.log 2024-09-16T04:56:17.371 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70744.log.gz 2024-09-16T04:56:17.371 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51402.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51402.log.gz -5 2024-09-16T04:56:17.371 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53418.log 2024-09-16T04:56:17.371 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44165.log 2024-09-16T04:56:17.371 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59470.log.gz 2024-09-16T04:56:17.371 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43997.log 2024-09-16T04:56:17.371 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53418.log.gz 2024-09-16T04:56:17.371 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69400.log 2024-09-16T04:56:17.371 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44165.log.gz 2024-09-16T04:56:17.371 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45914.log 2024-09-16T04:56:17.371 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43997.log.gz 2024-09-16T04:56:17.371 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57156.log 2024-09-16T04:56:17.371 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69400.log.gz 2024-09-16T04:56:17.371 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73981.log 2024-09-16T04:56:17.372 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45914.log.gz 2024-09-16T04:56:17.372 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73435.log 2024-09-16T04:56:17.372 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57156.log.gz 2024-09-16T04:56:17.372 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32556.log 2024-09-16T04:56:17.372 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73981.log.gz 2024-09-16T04:56:17.372 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/ceph-client.admin.73435.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.73897.log 2024-09-16T04:56:17.372 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73435.log.gz 2024-09-16T04:56:17.372 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.32556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74191.log 2024-09-16T04:56:17.372 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32556.log.gz 2024-09-16T04:56:17.372 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48126.log 2024-09-16T04:56:17.372 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73897.log.gz 2024-09-16T04:56:17.372 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/ceph-client.admin.74191.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39215.log 2024-09-16T04:56:17.372 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74191.log.gz 2024-09-16T04:56:17.372 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68769.log 2024-09-16T04:56:17.373 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48126.log.gz 2024-09-16T04:56:17.373 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44915.log 2024-09-16T04:56:17.373 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39215.log.gz 2024-09-16T04:56:17.373 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62973.log 2024-09-16T04:56:17.373 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68769.log.gz 2024-09-16T04:56:17.373 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72763.log 2024-09-16T04:56:17.373 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44915.log.gz 2024-09-16T04:56:17.373 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/ceph-client.admin.62973.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.68097.log 2024-09-16T04:56:17.373 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62973.log.gz 2024-09-16T04:56:17.373 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49092.log 2024-09-16T04:56:17.373 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72763.log.gz 2024-09-16T04:56:17.373 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68097.log.gz 2024-09-16T04:56:17.373 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59899.log 2024-09-16T04:56:17.373 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64401.log 2024-09-16T04:56:17.374 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49092.log.gz 2024-09-16T04:56:17.374 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/ceph-client.admin.59899.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.69778.log 2024-09-16T04:56:17.374 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59899.log.gz 2024-09-16T04:56:17.374 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64401.log.gz 2024-09-16T04:56:17.374 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72594.log 2024-09-16T04:56:17.374 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38512.log 2024-09-16T04:56:17.374 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69778.log.gz 2024-09-16T04:56:17.374 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72594.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.67677.log 2024-09-16T04:56:17.374 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72594.log.gz 2024-09-16T04:56:17.374 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39252.log 2024-09-16T04:56:17.374 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38512.log.gz 2024-09-16T04:56:17.374 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49344.log 2024-09-16T04:56:17.374 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67677.log.gz 2024-09-16T04:56:17.374 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43829.log 2024-09-16T04:56:17.375 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39252.log.gz 2024-09-16T04:56:17.375 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56358.log 2024-09-16T04:56:17.375 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49344.log.gz 2024-09-16T04:56:17.375 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60241.log 2024-09-16T04:56:17.375 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43829.log.gz 2024-09-16T04:56:17.375 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56358.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.56358.log.gz 2024-09-16T04:56:17.375 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/ceph-client.admin.39711.log 2024-09-16T04:56:17.375 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54552.log 2024-09-16T04:56:17.375 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60241.log.gz 2024-09-16T04:56:17.375 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61921.log 2024-09-16T04:56:17.375 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39711.log.gz 2024-09-16T04:56:17.375 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48084.log 2024-09-16T04:56:17.375 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54552.log.gz 2024-09-16T04:56:17.376 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58668.log 2024-09-16T04:56:17.376 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61921.log.gz 2024-09-16T04:56:17.376 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57408.log 2024-09-16T04:56:17.376 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48084.log.gz 2024-09-16T04:56:17.376 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58668.log.gz 2024-09-16T04:56:17.376 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64989.log 2024-09-16T04:56:17.376 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72215.log 2024-09-16T04:56:17.376 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57408.log.gz 2024-09-16T04:56:17.376 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50142.log 2024-09-16T04:56:17.376 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64989.log.gz 2024-09-16T04:56:17.376 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60409.log 2024-09-16T04:56:17.376 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72215.log.gz 2024-09-16T04:56:17.376 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43117.log 2024-09-16T04:56:17.376 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50142.log.gz 2024-09-16T04:56:17.377 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73099.log 2024-09-16T04:56:17.377 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60409.log.gz 2024-09-16T04:56:17.377 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71206.log 2024-09-16T04:56:17.377 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43117.log.gz 2024-09-16T04:56:17.377 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60703.log 2024-09-16T04:56:17.377 INFO:teuthology.orchestra.run.smithi152.stderr: 28.9% -- replaced with /var/log/ceph/ceph-client.admin.73099.log.gz 2024-09-16T04:56:17.377 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65157.log 2024-09-16T04:56:17.377 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71206.log.gz 2024-09-16T04:56:17.377 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42991.log 2024-09-16T04:56:17.377 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60703.log.gz 2024-09-16T04:56:17.377 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53838.log 2024-09-16T04:56:17.377 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65157.log.gz 2024-09-16T04:56:17.377 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.42991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56778.log 2024-09-16T04:56:17.377 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42991.log.gz 2024-09-16T04:56:17.378 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53838.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53838.log.gz 2024-09-16T04:56:17.378 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72257.log 2024-09-16T04:56:17.378 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54132.log 2024-09-16T04:56:17.378 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56778.log.gz 2024-09-16T04:56:17.378 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65955.log 2024-09-16T04:56:17.378 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72257.log.gz 2024-09-16T04:56:17.378 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54132.log.gz 2024-09-16T04:56:17.378 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39859.log 2024-09-16T04:56:17.378 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68391.log 2024-09-16T04:56:17.378 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65955.log.gz 2024-09-16T04:56:17.378 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47412.log 2024-09-16T04:56:17.378 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39859.log.gz 2024-09-16T04:56:17.378 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62089.log 2024-09-16T04:56:17.378 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68391.log.gz 2024-09-16T04:56:17.379 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53796.log 2024-09-16T04:56:17.379 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47412.log.gz 2024-09-16T04:56:17.379 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37898.log 2024-09-16T04:56:17.379 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62089.log.gz 2024-09-16T04:56:17.379 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59722.log 2024-09-16T04:56:17.379 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53796.log.gz 2024-09-16T04:56:17.379 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37898.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.37898.log.gz 2024-09-16T04:56:17.379 INFO:teuthology.orchestra.run.smithi152.stderr: -- /var/log/ceph/ceph-client.admin.72931.log 2024-09-16T04:56:17.379 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48546.log 2024-09-16T04:56:17.379 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59722.log.gz 2024-09-16T04:56:17.379 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73309.log 2024-09-16T04:56:17.379 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72931.log.gz 2024-09-16T04:56:17.379 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54174.log 2024-09-16T04:56:17.379 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48546.log.gz 2024-09-16T04:56:17.380 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36831.log 2024-09-16T04:56:17.380 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73309.log.gz 2024-09-16T04:56:17.380 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64779.log 2024-09-16T04:56:17.380 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54174.log.gz 2024-09-16T04:56:17.380 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.36831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38131.log 2024-09-16T04:56:17.380 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36831.log.gz 2024-09-16T04:56:17.380 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51570.log 2024-09-16T04:56:17.380 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64779.log.gz 2024-09-16T04:56:17.380 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53754.log 2024-09-16T04:56:17.380 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38131.log.gz 2024-09-16T04:56:17.380 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50268.log 2024-09-16T04:56:17.380 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51570.log.gz 2024-09-16T04:56:17.380 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42528.log 2024-09-16T04:56:17.381 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53754.log.gz 2024-09-16T04:56:17.381 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49302.log 2024-09-16T04:56:17.381 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50268.log.gz 2024-09-16T04:56:17.381 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52200.log 2024-09-16T04:56:17.381 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.42528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42528.log.gz 2024-09-16T04:56:17.381 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72678.log 2024-09-16T04:56:17.381 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49302.log.gz 2024-09-16T04:56:17.381 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50688.log 2024-09-16T04:56:17.381 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52200.log.gz 2024-09-16T04:56:17.381 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52326.log 2024-09-16T04:56:17.381 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72678.log.gz 2024-09-16T04:56:17.381 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55182.log 2024-09-16T04:56:17.381 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50688.log.gz 2024-09-16T04:56:17.381 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69568.log 2024-09-16T04:56:17.381 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52326.log.gz 2024-09-16T04:56:17.382 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56148.log 2024-09-16T04:56:17.382 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55182.log.gz 2024-09-16T04:56:17.382 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56316.log 2024-09-16T04:56:17.382 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69568.log.gz 2024-09-16T04:56:17.382 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39970.log 2024-09-16T04:56:17.382 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56148.log.gz 2024-09-16T04:56:17.382 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64275.log 2024-09-16T04:56:17.382 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56316.log.gz 2024-09-16T04:56:17.382 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61081.log 2024-09-16T04:56:17.382 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39970.log.gz 2024-09-16T04:56:17.382 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55560.log 2024-09-16T04:56:17.382 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64275.log.gz 2024-09-16T04:56:17.382 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73477.log 2024-09-16T04:56:17.382 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61081.log.gz 2024-09-16T04:56:17.383 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63729.log 2024-09-16T04:56:17.383 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55560.log.gz 2024-09-16T04:56:17.383 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.73477.log: /var/log/ceph/ceph-client.admin.67509.log 2024-09-16T04:56:17.383 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73477.log.gz 2024-09-16T04:56:17.383 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61165.log 2024-09-16T04:56:17.383 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63729.log.gz 2024-09-16T04:56:17.383 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67509.log.gz 2024-09-16T04:56:17.383 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66963.log 2024-09-16T04:56:17.383 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73939.log 2024-09-16T04:56:17.383 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61165.log.gz 2024-09-16T04:56:17.383 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66963.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.66963.log.gz 2024-09-16T04:56:17.383 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/ceph-client.admin.47916.log 2024-09-16T04:56:17.383 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49932.log 2024-09-16T04:56:17.383 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73939.log.gz 2024-09-16T04:56:17.384 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72341.log 2024-09-16T04:56:17.384 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47916.log.gz 2024-09-16T04:56:17.384 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47580.log 2024-09-16T04:56:17.384 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49932.log.gz 2024-09-16T04:56:17.384 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60325.log 2024-09-16T04:56:17.384 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72341.log.gz 2024-09-16T04:56:17.384 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39489.log 2024-09-16T04:56:17.384 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47580.log.gz 2024-09-16T04:56:17.384 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65535.log 2024-09-16T04:56:17.384 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60325.log.gz 2024-09-16T04:56:17.384 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55098.log 2024-09-16T04:56:17.384 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39489.log.gz 2024-09-16T04:56:17.384 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59512.log 2024-09-16T04:56:17.384 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65535.log.gz 2024-09-16T04:56:17.385 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50520.log 2024-09-16T04:56:17.385 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55098.log.gz 2024-09-16T04:56:17.385 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56946.log 2024-09-16T04:56:17.385 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59512.log.gz 2024-09-16T04:56:17.385 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69064.log 2024-09-16T04:56:17.385 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50520.log.gz 2024-09-16T04:56:17.385 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45507.log 2024-09-16T04:56:17.385 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56946.log.gz 2024-09-16T04:56:17.385 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63687.log 2024-09-16T04:56:17.385 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69064.log.gz 2024-09-16T04:56:17.385 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59216.log 2024-09-16T04:56:17.385 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45507.log.gz 2024-09-16T04:56:17.385 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71795.log 2024-09-16T04:56:17.385 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63687.log.gz 2024-09-16T04:56:17.386 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52998.log 2024-09-16T04:56:17.386 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59216.log.gz 2024-09-16T04:56:17.386 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62383.log 2024-09-16T04:56:17.386 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71795.log.gz 2024-09-16T04:56:17.386 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40303.log 2024-09-16T04:56:17.386 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52998.log.gz 2024-09-16T04:56:17.386 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39141.log 2024-09-16T04:56:17.386 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62383.log.gz 2024-09-16T04:56:17.386 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.40303.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40303.log.gz 2024-09-16T04:56:17.386 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.66879.log 2024-09-16T04:56:17.386 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53166.log 2024-09-16T04:56:17.386 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39141.log.gz 2024-09-16T04:56:17.386 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73183.log 2024-09-16T04:56:17.386 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66879.log.gz 2024-09-16T04:56:17.387 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74149.log 2024-09-16T04:56:17.387 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53166.log.gz 2024-09-16T04:56:17.387 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70114.log 2024-09-16T04:56:17.387 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73183.log.gz 2024-09-16T04:56:17.387 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.74149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72636.log 2024-09-16T04:56:17.387 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74149.log.gz 2024-09-16T04:56:17.387 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36974.log 2024-09-16T04:56:17.387 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70114.log.gz 2024-09-16T04:56:17.387 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48672.log 2024-09-16T04:56:17.387 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72636.log.gz 2024-09-16T04:56:17.387 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.36974.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T04:56:17.387 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36974.log.gz 2024-09-16T04:56:17.387 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38151.log 2024-09-16T04:56:17.387 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48672.log.gz 2024-09-16T04:56:17.387 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59680.log 2024-09-16T04:56:17.388 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51192.log 2024-09-16T04:56:17.388 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38151.log.gz 2024-09-16T04:56:17.388 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57324.log 2024-09-16T04:56:17.388 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59680.log.gz 2024-09-16T04:56:17.388 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42990.log 2024-09-16T04:56:17.388 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51192.log.gz 2024-09-16T04:56:17.388 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70996.log 2024-09-16T04:56:17.388 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57324.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57324.log.gz 2024-09-16T04:56:17.388 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.42990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64863.log 2024-09-16T04:56:17.388 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42990.log.gz 2024-09-16T04:56:17.388 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56988.log 2024-09-16T04:56:17.388 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70996.log.gz 2024-09-16T04:56:17.388 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62215.log 2024-09-16T04:56:17.389 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64863.log.gz 2024-09-16T04:56:17.389 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65913.log 2024-09-16T04:56:17.389 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56988.log.gz 2024-09-16T04:56:17.389 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60031.log 2024-09-16T04:56:17.389 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62215.log.gz 2024-09-16T04:56:17.389 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32869.log 2024-09-16T04:56:17.389 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65913.log.gz 2024-09-16T04:56:17.389 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60619.log 2024-09-16T04:56:17.389 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60031.log.gz 2024-09-16T04:56:17.389 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.32869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63057.log 2024-09-16T04:56:17.389 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55728.log 2024-09-16T04:56:17.389 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60619.log.gz 2024-09-16T04:56:17.389 INFO:teuthology.orchestra.run.smithi152.stderr: 96.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:56:17.389 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59384.log 2024-09-16T04:56:17.390 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63057.log.gz 2024-09-16T04:56:17.390 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67383.log 2024-09-16T04:56:17.390 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55728.log.gz 2024-09-16T04:56:17.390 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53502.log 2024-09-16T04:56:17.390 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59384.log.gz 2024-09-16T04:56:17.390 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/ceph-client.admin.67383.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36730.log 2024-09-16T04:56:17.390 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67383.log.gz 2024-09-16T04:56:17.390 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50100.log 2024-09-16T04:56:17.390 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53502.log.gz 2024-09-16T04:56:17.390 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.36730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63141.log 2024-09-16T04:56:17.390 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36730.log.gz 2024-09-16T04:56:17.390 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57450.log 2024-09-16T04:56:17.390 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50100.log.gz 2024-09-16T04:56:17.390 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37983.log 2024-09-16T04:56:17.391 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63141.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63141.log.gz 2024-09-16T04:56:17.391 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60535.log 2024-09-16T04:56:17.391 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57450.log.gz 2024-09-16T04:56:17.391 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72847.log 2024-09-16T04:56:17.391 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37983.log.gz 2024-09-16T04:56:17.391 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56526.log 2024-09-16T04:56:17.391 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60535.log.gz 2024-09-16T04:56:17.391 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58290.log 2024-09-16T04:56:17.391 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72847.log.gz 2024-09-16T04:56:17.391 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62341.log 2024-09-16T04:56:17.391 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56526.log.gz 2024-09-16T04:56:17.391 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48378.log 2024-09-16T04:56:17.391 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58290.log.gz 2024-09-16T04:56:17.391 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71374.log 2024-09-16T04:56:17.392 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62341.log.gz 2024-09-16T04:56:17.392 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43913.log 2024-09-16T04:56:17.392 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48378.log.gz 2024-09-16T04:56:17.392 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52536.log 2024-09-16T04:56:17.392 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71374.log.gz 2024-09-16T04:56:17.392 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68475.log 2024-09-16T04:56:17.392 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43913.log.gz 2024-09-16T04:56:17.392 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67803.log 2024-09-16T04:56:17.392 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52536.log.gz 2024-09-16T04:56:17.392 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68475.log.gz 2024-09-16T04:56:17.392 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52830.log 2024-09-16T04:56:17.392 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58584.log 2024-09-16T04:56:17.392 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67803.log.gz 2024-09-16T04:56:17.393 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51948.log 2024-09-16T04:56:17.393 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52830.log.gz 2024-09-16T04:56:17.393 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58584.log.gz 2024-09-16T04:56:17.393 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49470.log 2024-09-16T04:56:17.393 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47454.log 2024-09-16T04:56:17.393 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51948.log.gz 2024-09-16T04:56:17.393 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48630.log 2024-09-16T04:56:17.393 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49470.log.gz 2024-09-16T04:56:17.393 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43116.log 2024-09-16T04:56:17.393 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47454.log.gz 2024-09-16T04:56:17.393 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58920.log 2024-09-16T04:56:17.393 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48630.log.gz 2024-09-16T04:56:17.393 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55896.log 2024-09-16T04:56:17.394 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43116.log.gz 2024-09-16T04:56:17.394 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58920.log.gz 2024-09-16T04:56:17.394 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53670.log 2024-09-16T04:56:17.394 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67425.log 2024-09-16T04:56:17.394 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55896.log.gz 2024-09-16T04:56:17.394 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46025.log 2024-09-16T04:56:17.394 INFO:teuthology.orchestra.run.smithi152.stderr: 28.5% -- replaced with /var/log/ceph/ceph-client.admin.32869.log.gz 2024-09-16T04:56:17.394 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53670.log.gz 2024-09-16T04:56:17.394 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38845.log 2024-09-16T04:56:17.394 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67425.log.gz 2024-09-16T04:56:17.394 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.46025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73015.log 2024-09-16T04:56:17.394 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46025.log.gz 2024-09-16T04:56:17.394 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38299.log 2024-09-16T04:56:17.395 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38845.log.gz 2024-09-16T04:56:17.395 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69736.log 2024-09-16T04:56:17.395 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73015.log.gz 2024-09-16T04:56:17.395 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36973.log 2024-09-16T04:56:17.395 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38299.log.gz 2024-09-16T04:56:17.395 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59428.log 2024-09-16T04:56:17.395 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69736.log.gz 2024-09-16T04:56:17.395 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.36973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38660.log 2024-09-16T04:56:17.395 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36973.log.gz 2024-09-16T04:56:17.395 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58458.log 2024-09-16T04:56:17.395 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59428.log.gz 2024-09-16T04:56:17.395 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63015.log 2024-09-16T04:56:17.395 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38660.log.gz 2024-09-16T04:56:17.395 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70912.log 2024-09-16T04:56:17.396 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58458.log.gz 2024-09-16T04:56:17.396 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46062.log 2024-09-16T04:56:17.396 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63015.log.gz 2024-09-16T04:56:17.396 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38808.log 2024-09-16T04:56:17.396 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70912.log.gz 2024-09-16T04:56:17.396 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.46062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67005.log 2024-09-16T04:56:17.396 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46062.log.gz 2024-09-16T04:56:17.396 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68811.log 2024-09-16T04:56:17.396 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38808.log.gz 2024-09-16T04:56:17.396 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39326.log 2024-09-16T04:56:17.396 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67005.log.gz 2024-09-16T04:56:17.396 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49974.log 2024-09-16T04:56:17.396 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68811.log.gz 2024-09-16T04:56:17.397 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39896.log 2024-09-16T04:56:17.397 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39326.log.gz 2024-09-16T04:56:17.397 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71585.log 2024-09-16T04:56:17.397 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49974.log.gz 2024-09-16T04:56:17.397 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65115.log 2024-09-16T04:56:17.397 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39896.log.gz 2024-09-16T04:56:17.397 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58332.log 2024-09-16T04:56:17.397 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71585.log.gz 2024-09-16T04:56:17.397 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63981.log 2024-09-16T04:56:17.397 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65115.log.gz 2024-09-16T04:56:17.397 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55224.log 2024-09-16T04:56:17.397 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58332.log.gz 2024-09-16T04:56:17.397 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47538.log 2024-09-16T04:56:17.397 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63981.log.gz 2024-09-16T04:56:17.398 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45692.log 2024-09-16T04:56:17.398 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55224.log.gz 2024-09-16T04:56:17.398 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64905.log 2024-09-16T04:56:17.398 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47538.log.gz 2024-09-16T04:56:17.398 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53460.log 2024-09-16T04:56:17.398 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45692.log.gz 2024-09-16T04:56:17.398 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66669.log 2024-09-16T04:56:17.398 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64905.log.gz 2024-09-16T04:56:17.398 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53460.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T04:56:17.398 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53460.log.gz 2024-09-16T04:56:17.398 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70492.log 2024-09-16T04:56:17.398 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66669.log.gz 2024-09-16T04:56:17.398 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40192.log 2024-09-16T04:56:17.398 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62847.log 2024-09-16T04:56:17.399 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70492.log.gz 2024-09-16T04:56:17.399 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.40192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44417.log 2024-09-16T04:56:17.399 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40192.log.gz 2024-09-16T04:56:17.399 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48504.log 2024-09-16T04:56:17.399 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62847.log.gz 2024-09-16T04:56:17.399 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68559.log 2024-09-16T04:56:17.399 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44417.log.gz 2024-09-16T04:56:17.399 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45470.log 2024-09-16T04:56:17.399 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48504.log.gz 2024-09-16T04:56:17.399 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60199.log 2024-09-16T04:56:17.399 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68559.log.gz 2024-09-16T04:56:17.399 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39600.log 2024-09-16T04:56:17.399 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45470.log.gz 2024-09-16T04:56:17.400 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51150.log 2024-09-16T04:56:17.400 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60199.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60199.log.gz 2024-09-16T04:56:17.400 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51906.log 2024-09-16T04:56:17.400 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39600.log.gz 2024-09-16T04:56:17.400 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55686.log 2024-09-16T04:56:17.400 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51150.log.gz 2024-09-16T04:56:17.400 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69988.log 2024-09-16T04:56:17.400 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51906.log.gz 2024-09-16T04:56:17.400 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52410.log 2024-09-16T04:56:17.400 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55686.log.gz 2024-09-16T04:56:17.400 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51654.log 2024-09-16T04:56:17.400 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69988.log.gz 2024-09-16T04:56:17.400 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68181.log 2024-09-16T04:56:17.400 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52410.log.gz 2024-09-16T04:56:17.401 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40377.log 2024-09-16T04:56:17.401 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51654.log.gz 2024-09-16T04:56:17.401 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51780.log 2024-09-16T04:56:17.401 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68181.log.gz 2024-09-16T04:56:17.401 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.40377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45005.log 2024-09-16T04:56:17.401 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40377.log.gz 2024-09-16T04:56:17.401 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39748.log 2024-09-16T04:56:17.401 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51780.log.gz 2024-09-16T04:56:17.401 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64485.log 2024-09-16T04:56:17.401 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45005.log.gz 2024-09-16T04:56:17.401 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71080.log 2024-09-16T04:56:17.401 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39748.log.gz 2024-09-16T04:56:17.401 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73603.log 2024-09-16T04:56:17.401 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64485.log.gz 2024-09-16T04:56:17.402 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42529.log 2024-09-16T04:56:17.402 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71080.log.gz 2024-09-16T04:56:17.402 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67215.log 2024-09-16T04:56:17.402 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73603.log.gz 2024-09-16T04:56:17.402 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.42529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57492.log 2024-09-16T04:56:17.402 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42529.log.gz 2024-09-16T04:56:17.402 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71416.log 2024-09-16T04:56:17.402 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67215.log.gz 2024-09-16T04:56:17.402 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50604.log 2024-09-16T04:56:17.402 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57492.log.gz 2024-09-16T04:56:17.402 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70660.log 2024-09-16T04:56:17.402 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71416.log.gz 2024-09-16T04:56:17.402 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49848.log 2024-09-16T04:56:17.402 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50604.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50604.log.gz 2024-09-16T04:56:17.403 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36645.log 2024-09-16T04:56:17.403 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70660.log.gz 2024-09-16T04:56:17.403 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48882.log 2024-09-16T04:56:17.403 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49848.log.gz 2024-09-16T04:56:17.403 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.36645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56652.log 2024-09-16T04:56:17.403 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36645.log.gz 2024-09-16T04:56:17.403 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71753.log 2024-09-16T04:56:17.403 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48882.log.gz 2024-09-16T04:56:17.403 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53376.log 2024-09-16T04:56:17.403 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56652.log.gz 2024-09-16T04:56:17.403 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38771.log 2024-09-16T04:56:17.403 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71753.log.gz 2024-09-16T04:56:17.403 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67719.log 2024-09-16T04:56:17.403 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53376.log.gz 2024-09-16T04:56:17.404 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67131.log 2024-09-16T04:56:17.404 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38771.log.gz 2024-09-16T04:56:17.404 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62005.log 2024-09-16T04:56:17.404 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67719.log.gz 2024-09-16T04:56:17.404 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71500.log 2024-09-16T04:56:17.404 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67131.log.gz 2024-09-16T04:56:17.404 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38697.log 2024-09-16T04:56:17.404 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62005.log.gz 2024-09-16T04:56:17.404 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44333.log 2024-09-16T04:56:17.404 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71500.log.gz 2024-09-16T04:56:17.404 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71669.log 2024-09-16T04:56:17.404 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38697.log.gz 2024-09-16T04:56:17.404 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45299.log 2024-09-16T04:56:17.404 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44333.log.gz 2024-09-16T04:56:17.405 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67929.log 2024-09-16T04:56:17.405 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71669.log.gz 2024-09-16T04:56:17.405 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70450.log 2024-09-16T04:56:17.405 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45299.log.gz 2024-09-16T04:56:17.405 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42989.log 2024-09-16T04:56:17.405 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67929.log.gz 2024-09-16T04:56:17.405 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70450.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.70450.log.gz --verbose 2024-09-16T04:56:17.405 INFO:teuthology.orchestra.run.smithi152.stderr: -- /var/log/ceph/ceph-client.admin.48798.log 2024-09-16T04:56:17.405 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.42989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42989.log.gzgzip 2024-09-16T04:56:17.405 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53880.log 2024-09-16T04:56:17.405 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44081.log 2024-09-16T04:56:17.405 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48798.log.gz 2024-09-16T04:56:17.405 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72552.log 2024-09-16T04:56:17.406 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53880.log.gz 2024-09-16T04:56:17.406 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44123.log 2024-09-16T04:56:17.406 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44081.log.gz 2024-09-16T04:56:17.406 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65367.log 2024-09-16T04:56:17.406 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72552.log.gz 2024-09-16T04:56:17.406 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62721.log 2024-09-16T04:56:17.406 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44123.log.gz 2024-09-16T04:56:17.406 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58752.log 2024-09-16T04:56:17.406 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65367.log.gz 2024-09-16T04:56:17.406 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57366.log 2024-09-16T04:56:17.406 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62721.log.gz 2024-09-16T04:56:17.406 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70282.log 2024-09-16T04:56:17.406 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58752.log.gz 2024-09-16T04:56:17.407 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43243.log 2024-09-16T04:56:17.407 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57366.log.gz 2024-09-16T04:56:17.407 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70282.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T04:56:17.407 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70282.log.gz 2024-09-16T04:56:17.407 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74233.log 2024-09-16T04:56:17.407 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43243.log.gz 2024-09-16T04:56:17.407 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59596.log 2024-09-16T04:56:17.407 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.74233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69021.log 2024-09-16T04:56:17.407 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74233.log.gz 2024-09-16T04:56:17.407 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38549.log 2024-09-16T04:56:17.407 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59596.log.gz 2024-09-16T04:56:17.407 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73519.log 2024-09-16T04:56:17.407 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69021.log.gz 2024-09-16T04:56:17.407 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40081.log 2024-09-16T04:56:17.408 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38549.log.gz 2024-09-16T04:56:17.408 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45433.log 2024-09-16T04:56:17.408 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73519.log.gz 2024-09-16T04:56:17.408 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.40081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40118.log 2024-09-16T04:56:17.408 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40081.log.gz 2024-09-16T04:56:17.408 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66249.log 2024-09-16T04:56:17.408 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45433.log.gz 2024-09-16T04:56:17.408 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.40118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57198.log 2024-09-16T04:56:17.408 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40118.log.gz 2024-09-16T04:56:17.408 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71543.log 2024-09-16T04:56:17.408 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66249.log.gz 2024-09-16T04:56:17.408 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53544.log 2024-09-16T04:56:17.408 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57198.log.gz 2024-09-16T04:56:17.408 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47748.log 2024-09-16T04:56:17.409 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71543.log.gz 2024-09-16T04:56:17.409 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52662.log 2024-09-16T04:56:17.409 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53544.log.gz 2024-09-16T04:56:17.409 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61459.log 2024-09-16T04:56:17.409 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47748.log.gz 2024-09-16T04:56:17.409 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66543.log 2024-09-16T04:56:17.409 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52662.log.gz 2024-09-16T04:56:17.409 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70072.log 2024-09-16T04:56:17.409 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61459.log.gz 2024-09-16T04:56:17.409 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67257.log 2024-09-16T04:56:17.409 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66543.log.gz 2024-09-16T04:56:17.409 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45171.log 2024-09-16T04:56:17.409 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70072.log.gz 2024-09-16T04:56:17.409 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53208.log 2024-09-16T04:56:17.409 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67257.log.gz 2024-09-16T04:56:17.410 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70954.log 2024-09-16T04:56:17.410 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45171.log.gz 2024-09-16T04:56:17.410 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53712.log 2024-09-16T04:56:17.410 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53208.log.gz 2024-09-16T04:56:17.410 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51276.log 2024-09-16T04:56:17.410 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70954.log.gz 2024-09-16T04:56:17.410 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45655.log 2024-09-16T04:56:17.410 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53712.log.gz 2024-09-16T04:56:17.410 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39104.log 2024-09-16T04:56:17.410 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51276.log.gz 2024-09-16T04:56:17.410 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43369.log 2024-09-16T04:56:17.410 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45655.log.gz 2024-09-16T04:56:17.410 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37823.log 2024-09-16T04:56:17.411 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39104.log.gz 2024-09-16T04:56:17.411 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68013.log 2024-09-16T04:56:17.411 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43369.log.gz 2024-09-16T04:56:17.411 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49806.log 2024-09-16T04:56:17.411 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37823.log.gz 2024-09-16T04:56:17.411 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45877.log 2024-09-16T04:56:17.411 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68013.log.gz 2024-09-16T04:56:17.411 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73057.log 2024-09-16T04:56:17.411 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49806.log.gz 2024-09-16T04:56:17.411 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32731.log 2024-09-16T04:56:17.411 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45877.log.gz 2024-09-16T04:56:17.411 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40507.log 2024-09-16T04:56:17.411 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73057.log.gz 2024-09-16T04:56:17.411 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.32731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45322.log 2024-09-16T04:56:17.412 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32731.log.gz 2024-09-16T04:56:17.412 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.40507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61123.log 2024-09-16T04:56:17.412 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40507.log.gz 2024-09-16T04:56:17.412 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63771.log 2024-09-16T04:56:17.412 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45322.log.gz 2024-09-16T04:56:17.412 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66711.log 2024-09-16T04:56:17.412 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61123.log.gz 2024-09-16T04:56:17.412 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63771.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.63771.log.gz -5 2024-09-16T04:56:17.412 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/ceph-client.admin.65703.log 2024-09-16T04:56:17.412 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72047.log 2024-09-16T04:56:17.412 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66711.log.gz 2024-09-16T04:56:17.412 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65703.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.65703.log.gz 2024-09-16T04:56:17.412 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53628.log 2024-09-16T04:56:17.412 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38993.log 2024-09-16T04:56:17.413 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72047.log.gz 2024-09-16T04:56:17.413 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43577.log 2024-09-16T04:56:17.413 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53628.log.gz 2024-09-16T04:56:17.413 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38993.log.gz 2024-09-16T04:56:17.413 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51864.log 2024-09-16T04:56:17.413 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32949.log 2024-09-16T04:56:17.413 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43577.log.gz 2024-09-16T04:56:17.413 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51864.log.gz 2024-09-16T04:56:17.413 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68433.log 2024-09-16T04:56:17.413 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66753.log 2024-09-16T04:56:17.413 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.32949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32949.log.gz 2024-09-16T04:56:17.413 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68433.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:56:17.413 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68433.log.gz 2024-09-16T04:56:17.414 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66039.log 2024-09-16T04:56:17.414 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66753.log.gz 2024-09-16T04:56:17.414 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72805.log 2024-09-16T04:56:17.414 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66039.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66039.log.gz 2024-09-16T04:56:17.414 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50394.log 2024-09-16T04:56:17.414 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72805.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72805.log.gz 2024-09-16T04:56:17.414 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38364.log 2024-09-16T04:56:17.414 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57576.log 2024-09-16T04:56:17.414 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50394.log.gz 2024-09-16T04:56:17.414 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60829.log 2024-09-16T04:56:17.414 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38364.log.gz 2024-09-16T04:56:17.414 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48210.log 2024-09-16T04:56:17.414 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57576.log.gz 2024-09-16T04:56:17.414 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60829.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.63435.log 2024-09-16T04:56:17.415 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60829.log.gz 2024-09-16T04:56:17.415 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59174.log 2024-09-16T04:56:17.415 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48210.log.gz 2024-09-16T04:56:17.415 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72383.log 2024-09-16T04:56:17.415 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63435.log.gz 2024-09-16T04:56:17.415 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66585.log 2024-09-16T04:56:17.415 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59174.log.gz 2024-09-16T04:56:17.415 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61837.log 2024-09-16T04:56:17.415 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72383.log.gz 2024-09-16T04:56:17.415 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64065.log 2024-09-16T04:56:17.415 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66585.log.gz 2024-09-16T04:56:17.415 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59088.log 2024-09-16T04:56:17.415 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61837.log.gz 2024-09-16T04:56:17.415 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39424.log 2024-09-16T04:56:17.416 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64065.log.gz 2024-09-16T04:56:17.416 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59088.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59088.log.gz 2024-09-16T04:56:17.416 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61333.log 2024-09-16T04:56:17.416 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50058.log 2024-09-16T04:56:17.416 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39424.log.gz 2024-09-16T04:56:17.416 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61333.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.61333.log.gz 2024-09-16T04:56:17.416 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/ceph-client.admin.61753.log 2024-09-16T04:56:17.416 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55308.log 2024-09-16T04:56:17.416 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50058.log.gz 2024-09-16T04:56:17.416 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61753.log.gz 2024-09-16T04:56:17.416 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66081.log 2024-09-16T04:56:17.416 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46247.log 2024-09-16T04:56:17.416 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55308.log.gz 2024-09-16T04:56:17.416 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62131.log 2024-09-16T04:56:17.417 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66081.log.gz 2024-09-16T04:56:17.417 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.46247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59130.log 2024-09-16T04:56:17.417 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46247.log.gz 2024-09-16T04:56:17.417 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:56:17.417 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49260.log 2024-09-16T04:56:17.417 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62131.log.gz 2024-09-16T04:56:17.417 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72089.log 2024-09-16T04:56:17.417 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59130.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59130.log.gz 2024-09-16T04:56:17.417 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64527.log 2024-09-16T04:56:17.417 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49260.log.gz 2024-09-16T04:56:17.417 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68727.log 2024-09-16T04:56:17.417 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72089.log.gz 2024-09-16T04:56:17.417 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72973.log 2024-09-16T04:56:17.417 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64527.log.gz 2024-09-16T04:56:17.418 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61711.log 2024-09-16T04:56:17.418 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68727.log.gz 2024-09-16T04:56:17.418 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72973.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.40340.log 2024-09-16T04:56:17.418 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72973.log.gz 2024-09-16T04:56:17.418 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64737.log 2024-09-16T04:56:17.418 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61711.log.gz 2024-09-16T04:56:17.418 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.40340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67089.log 2024-09-16T04:56:17.418 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40340.log.gz 2024-09-16T04:56:17.418 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59989.log 2024-09-16T04:56:17.418 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64737.log.gz 2024-09-16T04:56:17.418 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67089.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.67089.log.gz -5 2024-09-16T04:56:17.418 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T04:56:17.418 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54258.log 2024-09-16T04:56:17.418 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59989.log.gz 2024-09-16T04:56:17.419 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54804.log 2024-09-16T04:56:17.419 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63183.log 2024-09-16T04:56:17.419 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54258.log.gz 2024-09-16T04:56:17.419 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48420.log 2024-09-16T04:56:17.419 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54804.log.gz 2024-09-16T04:56:17.419 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37587.log 2024-09-16T04:56:17.419 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63183.log.gz 2024-09-16T04:56:17.419 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64317.log 2024-09-16T04:56:17.419 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48420.log.gz 2024-09-16T04:56:17.419 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56190.log 2024-09-16T04:56:17.419 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37587.log.gz 2024-09-16T04:56:17.419 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73687.log 2024-09-16T04:56:17.419 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64317.log.gz 2024-09-16T04:56:17.420 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56190.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.56190.log.gz 2024-09-16T04:56:17.420 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.72510.log 2024-09-16T04:56:17.420 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71879.log 2024-09-16T04:56:17.420 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73687.log.gz 2024-09-16T04:56:17.420 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65619.log 2024-09-16T04:56:17.420 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72510.log.gz 2024-09-16T04:56:17.420 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38882.log 2024-09-16T04:56:17.420 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71879.log.gz 2024-09-16T04:56:17.420 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73813.log 2024-09-16T04:56:17.420 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65619.log.gz 2024-09-16T04:56:17.420 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38257.log 2024-09-16T04:56:17.420 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38882.log.gz 2024-09-16T04:56:17.420 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43478.log 2024-09-16T04:56:17.421 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73813.log.gz 2024-09-16T04:56:17.421 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45729.log 2024-09-16T04:56:17.421 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38257.log.gz 2024-09-16T04:56:17.421 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61585.log 2024-09-16T04:56:17.421 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49050.log 2024-09-16T04:56:17.421 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45729.log.gz 2024-09-16T04:56:17.421 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69820.log 2024-09-16T04:56:17.421 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61585.log.gz 2024-09-16T04:56:17.421 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47370.log 2024-09-16T04:56:17.421 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49050.log.gz 2024-09-16T04:56:17.421 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44753.log 2024-09-16T04:56:17.421 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69820.log.gz 2024-09-16T04:56:17.421 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44249.log 2024-09-16T04:56:17.421 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47370.log.gz 2024-09-16T04:56:17.422 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37757.log 2024-09-16T04:56:17.422 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44753.log.gz 2024-09-16T04:56:17.422 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51024.log 2024-09-16T04:56:17.422 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44249.log.gz 2024-09-16T04:56:17.422 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38475.log 2024-09-16T04:56:17.422 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37757.log.gz 2024-09-16T04:56:17.422 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52956.log 2024-09-16T04:56:17.422 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51024.log.gz 2024-09-16T04:56:17.422 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52032.log 2024-09-16T04:56:17.422 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38475.log.gz 2024-09-16T04:56:17.422 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37671.log 2024-09-16T04:56:17.422 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52956.log.gz 2024-09-16T04:56:17.422 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37713.log 2024-09-16T04:56:17.422 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52032.log.gz 2024-09-16T04:56:17.423 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59300.log 2024-09-16T04:56:17.423 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37671.log.gz 2024-09-16T04:56:17.423 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42447.log 2024-09-16T04:56:17.423 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37713.log.gz 2024-09-16T04:56:17.423 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53922.log 2024-09-16T04:56:17.423 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59300.log.gz 2024-09-16T04:56:17.423 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.42447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69610.log 2024-09-16T04:56:17.423 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42447.log.gz 2024-09-16T04:56:17.423 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44501.log 2024-09-16T04:56:17.423 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53922.log.gz 2024-09-16T04:56:17.423 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56442.log 2024-09-16T04:56:17.423 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69610.log.gz 2024-09-16T04:56:17.423 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.44501.log: -- /var/log/ceph/ceph-client.admin.68307.log 2024-09-16T04:56:17.423 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44501.log.gz 2024-09-16T04:56:17.424 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40451.log 2024-09-16T04:56:17.424 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56442.log.gz 2024-09-16T04:56:17.424 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68643.log 2024-09-16T04:56:17.424 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68307.log.gz 2024-09-16T04:56:17.424 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.40451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65745.log 2024-09-16T04:56:17.424 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40451.log.gz 2024-09-16T04:56:17.424 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52284.log 0.0% 2024-09-16T04:56:17.424 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68643.log.gz 2024-09-16T04:56:17.424 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73855.log 2024-09-16T04:56:17.424 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65745.log.gz 2024-09-16T04:56:17.424 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55602.log 2024-09-16T04:56:17.424 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52284.log.gz 2024-09-16T04:56:17.424 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55266.log 2024-09-16T04:56:17.424 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73855.log.gz 2024-09-16T04:56:17.425 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69190.log 2024-09-16T04:56:17.425 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55602.log.gz 2024-09-16T04:56:17.425 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63603.log 2024-09-16T04:56:17.425 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55266.log.gz 2024-09-16T04:56:17.425 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51108.log 2024-09-16T04:56:17.425 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69190.log.gz 2024-09-16T04:56:17.425 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67887.log 2024-09-16T04:56:17.425 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63603.log.gz 2024-09-16T04:56:17.425 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54300.log 2024-09-16T04:56:17.425 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51108.log.gz 2024-09-16T04:56:17.425 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71248.log 2024-09-16T04:56:17.425 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67887.log.gz 2024-09-16T04:56:17.425 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37102.log 2024-09-16T04:56:17.426 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54300.log.gz 2024-09-16T04:56:17.426 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68601.log 2024-09-16T04:56:17.426 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71248.log.gz 2024-09-16T04:56:17.426 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42654.log 2024-09-16T04:56:17.426 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37102.log.gz 2024-09-16T04:56:17.426 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68853.log 2024-09-16T04:56:17.426 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68601.log.gz 2024-09-16T04:56:17.426 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.42654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70366.log 2024-09-16T04:56:17.426 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42654.log.gz 2024-09-16T04:56:17.426 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44916.log 2024-09-16T04:56:17.426 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68853.log.gz 2024-09-16T04:56:17.426 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62637.log 2024-09-16T04:56:17.426 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70366.log.gz 2024-09-16T04:56:17.426 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41509.log 2024-09-16T04:56:17.427 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44916.log.gz 2024-09-16T04:56:17.427 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60787.log 2024-09-16T04:56:17.427 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62637.log.gz 2024-09-16T04:56:17.427 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.41509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50184.log 2024-09-16T04:56:17.427 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41509.log.gz 2024-09-16T04:56:17.427 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58122.log 2024-09-16T04:56:17.427 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60787.log.gz 2024-09-16T04:56:17.427 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50982.log 2024-09-16T04:56:17.427 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50184.log.gz 2024-09-16T04:56:17.427 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37321.log 2024-09-16T04:56:17.427 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58122.log.gz 2024-09-16T04:56:17.427 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50310.log 2024-09-16T04:56:17.427 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50982.log.gz 2024-09-16T04:56:17.427 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36873.log 2024-09-16T04:56:17.428 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37321.log.gz 2024-09-16T04:56:17.428 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71290.log 2024-09-16T04:56:17.428 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50310.log.gz 2024-09-16T04:56:17.428 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.36873.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T04:56:17.428 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36873.log.gz 2024-09-16T04:56:17.428 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65241.log 2024-09-16T04:56:17.428 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71290.log.gz 2024-09-16T04:56:17.428 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56400.log 2024-09-16T04:56:17.428 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70324.log 2024-09-16T04:56:17.428 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65241.log.gz 2024-09-16T04:56:17.428 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44711.log 2024-09-16T04:56:17.428 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56400.log.gz 2024-09-16T04:56:17.428 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43787.log 2024-09-16T04:56:17.429 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70324.log.gz 2024-09-16T04:56:17.429 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38956.log 2024-09-16T04:56:17.429 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44711.log.gz 2024-09-16T04:56:17.429 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55854.log 2024-09-16T04:56:17.429 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43787.log.gz 2024-09-16T04:56:17.429 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70786.log 2024-09-16T04:56:17.429 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38956.log.gz 2024-09-16T04:56:17.429 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72467.log 2024-09-16T04:56:17.429 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55854.log.gz 2024-09-16T04:56:17.429 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67299.log 2024-09-16T04:56:17.429 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70786.log.gz 2024-09-16T04:56:17.429 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54090.log 2024-09-16T04:56:17.429 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72467.log.gz 2024-09-16T04:56:17.430 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67299.log.gz 2024-09-16T04:56:17.430 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63561.log 2024-09-16T04:56:17.430 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54888.log 2024-09-16T04:56:17.430 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54090.log.gz 2024-09-16T04:56:17.430 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63477.log 2024-09-16T04:56:17.430 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63561.log.gz 2024-09-16T04:56:17.430 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59046.log 2024-09-16T04:56:17.430 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54888.log.gz 2024-09-16T04:56:17.430 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49638.log 2024-09-16T04:56:17.430 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63477.log.gz 2024-09-16T04:56:17.430 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48840.log 2024-09-16T04:56:17.430 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59046.log.gz 2024-09-16T04:56:17.430 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43455.log 2024-09-16T04:56:17.430 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49638.log.gz 2024-09-16T04:56:17.431 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66501.log 2024-09-16T04:56:17.431 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48840.log.gz 2024-09-16T04:56:17.431 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44543.log 2024-09-16T04:56:17.431 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43455.log.gz 2024-09-16T04:56:17.431 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45257.log/var/log/ceph/ceph-client.admin.66501.log: 2024-09-16T04:56:17.431 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66501.log.gz 2024-09-16T04:56:17.431 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52620.log 2024-09-16T04:56:17.431 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44543.log.gz 2024-09-16T04:56:17.431 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45257.log.gz 2024-09-16T04:56:17.431 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38438.log 2024-09-16T04:56:17.431 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70618.log 2024-09-16T04:56:17.431 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52620.log.gz 2024-09-16T04:56:17.431 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70576.log 2024-09-16T04:56:17.431 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38438.log.gz 2024-09-16T04:56:17.431 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64359.log 2024-09-16T04:56:17.432 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70618.log.gz 2024-09-16T04:56:17.432 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65871.log 2024-09-16T04:56:17.432 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70576.log.gz 2024-09-16T04:56:17.432 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44627.log 2024-09-16T04:56:17.432 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64359.log.gz 2024-09-16T04:56:17.432 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51486.log 2024-09-16T04:56:17.432 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65871.log.gz 2024-09-16T04:56:17.432 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49596.log 2024-09-16T04:56:17.432 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44627.log.gz 2024-09-16T04:56:17.432 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39030.log 2024-09-16T04:56:17.432 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51486.log.gz 2024-09-16T04:56:17.432 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61417.log 2024-09-16T04:56:17.432 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49596.log.gz 2024-09-16T04:56:17.432 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63813.log 2024-09-16T04:56:17.433 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39030.log.gz 2024-09-16T04:56:17.433 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62047.log 2024-09-16T04:56:17.433 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61417.log.gz 2024-09-16T04:56:17.433 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54972.log 2024-09-16T04:56:17.433 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63813.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63813.log.gz 2024-09-16T04:56:17.433 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48168.log 2024-09-16T04:56:17.433 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62047.log.gz 2024-09-16T04:56:17.433 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66375.log 2024-09-16T04:56:17.433 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54972.log.gz 2024-09-16T04:56:17.433 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57744.log 2024-09-16T04:56:17.433 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48168.log.gz 2024-09-16T04:56:17.433 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37143.log 2024-09-16T04:56:17.433 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66375.log.gz 2024-09-16T04:56:17.433 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38919.log 0.0% 2024-09-16T04:56:17.434 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57744.log.gz 2024-09-16T04:56:17.434 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52746.log 2024-09-16T04:56:17.434 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37143.log.gz 2024-09-16T04:56:17.434 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72721.log 2024-09-16T04:56:17.434 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38919.log.gz 2024-09-16T04:56:17.434 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74107.log 2024-09-16T04:56:17.434 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52746.log.gz 2024-09-16T04:56:17.434 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58878.log 2024-09-16T04:56:17.434 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72721.log.gz 2024-09-16T04:56:17.434 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.74107.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.74107.log.gz 2024-09-16T04:56:17.434 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/ceph-client.admin.65283.log 2024-09-16T04:56:17.434 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39382.log 2024-09-16T04:56:17.434 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58878.log.gz 2024-09-16T04:56:17.434 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57282.log 2024-09-16T04:56:17.435 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65283.log.gz 2024-09-16T04:56:17.435 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.39382.log: /var/log/ceph/ceph-client.admin.47244.log 2024-09-16T04:56:17.435 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39382.log.gz 2024-09-16T04:56:17.435 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54594.log 2024-09-16T04:56:17.435 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57282.log.gz 2024-09-16T04:56:17.435 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42780.log 2024-09-16T04:56:17.435 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47244.log.gz 2024-09-16T04:56:17.435 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66291.log 2024-09-16T04:56:17.435 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54594.log.gz 2024-09-16T04:56:17.435 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.42780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70198.log 2024-09-16T04:56:17.435 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42780.log.gz 2024-09-16T04:56:17.435 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66291.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.66291.log.gz 2024-09-16T04:56:17.435 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/ceph-client.admin.42909.log 2024-09-16T04:56:17.435 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53082.log 2024-09-16T04:56:17.436 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70198.log.gz 2024-09-16T04:56:17.436 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.42909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42909.log.gz 2024-09-16T04:56:17.436 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60283.log 2024-09-16T04:56:17.436 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54930.log 2024-09-16T04:56:17.436 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53082.log.gz 2024-09-16T04:56:17.436 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52116.log 2024-09-16T04:56:17.436 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60283.log.gz 2024-09-16T04:56:17.436 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51360.log 2024-09-16T04:56:17.436 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54930.log.gz 2024-09-16T04:56:17.436 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54426.log 2024-09-16T04:56:17.436 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52116.log.gz 2024-09-16T04:56:17.436 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.51360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71711.log 2024-09-16T04:56:17.436 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51360.log.gz 2024-09-16T04:56:17.436 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52158.log 2024-09-16T04:56:17.436 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54426.log.gz 2024-09-16T04:56:17.437 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45044.log 2024-09-16T04:56:17.437 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71711.log.gz 2024-09-16T04:56:17.437 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74023.log 2024-09-16T04:56:17.437 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52158.log.gz 2024-09-16T04:56:17.437 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61795.log 2024-09-16T04:56:17.437 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45044.log.gz 2024-09-16T04:56:17.437 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.74023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53964.log 2024-09-16T04:56:17.437 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74023.log.gz 2024-09-16T04:56:17.437 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67551.log 2024-09-16T04:56:17.437 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.61795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61795.log.gz 2024-09-16T04:56:17.437 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48252.log 2024-09-16T04:56:17.437 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53964.log.gz 2024-09-16T04:56:17.437 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52872.log 2024-09-16T04:56:17.437 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67551.log.gz 2024-09-16T04:56:17.438 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46173.log 2024-09-16T04:56:17.438 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48252.log.gz 2024-09-16T04:56:17.438 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.52872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74273.log 2024-09-16T04:56:17.438 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52872.log.gz 2024-09-16T04:56:17.438 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53292.log 2024-09-16T04:56:17.438 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.46173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46173.log.gz 2024-09-16T04:56:17.438 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.74273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50814.log 2024-09-16T04:56:17.438 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74273.log.gz 2024-09-16T04:56:17.438 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45803.log 2024-09-16T04:56:17.438 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53292.log.gz 2024-09-16T04:56:17.438 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.50814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64611.log 2024-09-16T04:56:17.438 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50814.log.gz 2024-09-16T04:56:17.438 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55770.log 2024-09-16T04:56:17.438 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45803.log.gz 2024-09-16T04:56:17.439 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.64611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55518.log 2024-09-16T04:56:17.439 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64611.log.gz 2024-09-16T04:56:17.439 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69484.log 2024-09-16T04:56:17.439 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55770.log.gz 2024-09-16T04:56:17.439 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37899.log 2024-09-16T04:56:17.439 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55518.log.gz 2024-09-16T04:56:17.439 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69106.log 2024-09-16T04:56:17.439 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69484.log.gz 2024-09-16T04:56:17.439 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73729.log 2024-09-16T04:56:17.439 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37899.log.gz 2024-09-16T04:56:17.439 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69106.log.gz 2024-09-16T04:56:17.439 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69946.log 2024-09-16T04:56:17.439 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71627.log 2024-09-16T04:56:17.439 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73729.log.gz 2024-09-16T04:56:17.439 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69862.log 2024-09-16T04:56:17.440 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69946.log.gz 2024-09-16T04:56:17.440 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44585.log 2024-09-16T04:56:17.440 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71627.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71627.log.gz 2024-09-16T04:56:17.440 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49176.log 2024-09-16T04:56:17.440 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69862.log.gz 2024-09-16T04:56:17.440 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37251.log 2024-09-16T04:56:17.440 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44585.log.gz 2024-09-16T04:56:17.440 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.49176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70156.log 2024-09-16T04:56:17.440 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49176.log.gz 2024-09-16T04:56:17.440 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43703.log 2024-09-16T04:56:17.440 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37251.log.gz 2024-09-16T04:56:17.440 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70156.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.70156.log.gz 2024-09-16T04:56:17.440 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.73645.log 2024-09-16T04:56:17.441 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71963.log 2024-09-16T04:56:17.441 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43703.log.gz 2024-09-16T04:56:17.441 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.73645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62553.log 2024-09-16T04:56:17.441 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73645.log.gz 2024-09-16T04:56:17.441 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39466.log 2024-09-16T04:56:17.441 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71963.log.gz 2024-09-16T04:56:17.441 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38341.log 2024-09-16T04:56:17.441 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62553.log.gz 2024-09-16T04:56:17.441 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56022.log 2024-09-16T04:56:17.441 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39466.log.gz 2024-09-16T04:56:17.441 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66207.log 2024-09-16T04:56:17.441 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38341.log.gz 2024-09-16T04:56:17.441 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.56022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37419.log 2024-09-16T04:56:17.441 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56022.log.gz 2024-09-16T04:56:17.442 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47622.log 2024-09-16T04:56:17.442 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66207.log.gz 2024-09-16T04:56:17.442 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39637.log 2024-09-16T04:56:17.442 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37419.log.gz 2024-09-16T04:56:17.442 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.47622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45840.log 2024-09-16T04:56:17.442 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47622.log.gz 2024-09-16T04:56:17.442 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68139.log 2024-09-16T04:56:17.442 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39637.log.gz 2024-09-16T04:56:17.442 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.45840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66333.log 2024-09-16T04:56:17.442 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45840.log.gz 2024-09-16T04:56:17.442 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.68139.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T04:56:17.442 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68139.log.gz 2024-09-16T04:56:17.442 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36975.log 2024-09-16T04:56:17.442 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66333.log.gz 2024-09-16T04:56:17.443 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59638.log 2024-09-16T04:56:17.443 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.36975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63351.log 2024-09-16T04:56:17.443 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36975.log.gz 2024-09-16T04:56:17.443 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.59638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57996.log 2024-09-16T04:56:17.443 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59638.log.gz 2024-09-16T04:56:17.443 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55980.log 2024-09-16T04:56:17.443 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63351.log.gz 2024-09-16T04:56:17.443 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57996.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57996.log.gz 2024-09-16T04:56:17.443 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58164.log 2024-09-16T04:56:17.443 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.55980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39674.log 2024-09-16T04:56:17.443 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55980.log.gz 2024-09-16T04:56:17.443 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58164.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58164.log.gz --verbose 2024-09-16T04:56:17.443 INFO:teuthology.orchestra.run.smithi152.stderr: -- /var/log/ceph/ceph-client.admin.65829.log 2024-09-16T04:56:17.443 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65409.log 2024-09-16T04:56:17.443 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39674.log.gz 2024-09-16T04:56:17.444 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54216.log 2024-09-16T04:56:17.444 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65829.log.gz 2024-09-16T04:56:17.444 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58836.log 2024-09-16T04:56:17.444 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65409.log.gz 2024-09-16T04:56:17.444 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69694.log 2024-09-16T04:56:17.444 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54216.log.gz 2024-09-16T04:56:17.444 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65451.log 2024-09-16T04:56:17.444 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58836.log.gz 2024-09-16T04:56:17.444 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.69694.log: -- /var/log/ceph/ceph-client.admin.62889.log 2024-09-16T04:56:17.444 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69694.log.gz 2024-09-16T04:56:17.444 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.65451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70534.log 2024-09-16T04:56:17.444 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65451.log.gz 2024-09-16T04:56:17.444 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.62889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39563.log 2024-09-16T04:56:17.444 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62889.log.gz 2024-09-16T04:56:17.445 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.70534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72005.log 2024-09-16T04:56:17.445 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70534.log.gz 2024-09-16T04:56:17.445 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69232.log 2024-09-16T04:56:17.445 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39563.log.gz 2024-09-16T04:56:17.445 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.72005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63855.log 2024-09-16T04:56:17.445 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72005.log.gz 2024-09-16T04:56:17.445 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.69232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39526.log 2024-09-16T04:56:17.445 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69232.log.gz 2024-09-16T04:56:17.445 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.63855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57828.log 2024-09-16T04:56:17.445 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63855.log.gz 2024-09-16T04:56:17.445 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38215.log 2024-09-16T04:56:17.445 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39526.log.gz 2024-09-16T04:56:17.445 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.57828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60451.log 2024-09-16T04:56:17.445 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57828.log.gz 2024-09-16T04:56:17.446 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.38215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66627.log 2024-09-16T04:56:17.446 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38215.log.gz 2024-09-16T04:56:17.446 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.60451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43329.log 2024-09-16T04:56:17.446 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60451.log.gz 2024-09-16T04:56:17.446 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.66627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39067.log 2024-09-16T04:56:17.446 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66627.log.gz 2024-09-16T04:56:17.446 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44669.log 2024-09-16T04:56:17.446 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43329.log.gz 2024-09-16T04:56:17.446 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.39067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37320.log 2024-09-16T04:56:17.446 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39067.log.gz 2024-09-16T04:56:17.446 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58248.log 2024-09-16T04:56:17.446 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44669.log.gz 2024-09-16T04:56:17.446 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54678.log 2024-09-16T04:56:17.446 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37320.log.gz 2024-09-16T04:56:17.447 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.58248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54048.log 2024-09-16T04:56:17.447 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58248.log.gz 2024-09-16T04:56:17.447 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36789.log 2024-09-16T04:56:17.447 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54678.log.gz 2024-09-16T04:56:17.447 INFO:teuthology.orchestra.run.smithi152.stderr: 95.1% -- replaced with /var/log/ceph/ceph-client.admin.43478.log.gz 2024-09-16T04:56:17.447 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.54048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43955.log 2024-09-16T04:56:17.447 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54048.log.gz 2024-09-16T04:56:17.447 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.36789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37142.log 2024-09-16T04:56:17.447 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36789.log.gz 2024-09-16T04:56:17.447 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.43955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71332.log 2024-09-16T04:56:17.447 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43955.log.gz 2024-09-16T04:56:17.447 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.37142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48042.log 2024-09-16T04:56:17.447 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37142.log.gz 2024-09-16T04:56:17.447 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.71332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44919.log 2024-09-16T04:56:17.447 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71332.log.gz 2024-09-16T04:56:17.448 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.48042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67047.log 2024-09-16T04:56:17.448 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48042.log.gz 2024-09-16T04:56:17.448 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.44919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53250.log 2024-09-16T04:56:17.448 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44919.log.gz 2024-09-16T04:56:17.448 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.67047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67047.log.gz 2024-09-16T04:56:17.448 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.53250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53250.log.gz 2024-09-16T04:56:19.090 INFO:teuthology.orchestra.run.smithi152.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T04:56:19.133 INFO:teuthology.orchestra.run.smithi152.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T04:56:21.103 INFO:teuthology.orchestra.run.smithi152.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T04:56:22.463 INFO:teuthology.orchestra.run.smithi152.stderr: 89.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T05:08:52.795 INFO:teuthology.orchestra.run.smithi152.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T05:11:27.165 INFO:teuthology.orchestra.run.smithi152.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T05:11:39.987 INFO:teuthology.orchestra.run.smithi152.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T05:11:48.620 INFO:teuthology.orchestra.run.smithi152.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T05:11:48.623 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-16T05:11:48.623 INFO:teuthology.orchestra.run.smithi152.stderr:real 15m31.587s 2024-09-16T05:11:48.623 INFO:teuthology.orchestra.run.smithi152.stderr:user 6m12.739s 2024-09-16T05:11:48.623 INFO:teuthology.orchestra.run.smithi152.stderr:sys 0m17.544s 2024-09-16T05:11:48.623 INFO:tasks.ceph:Archiving logs... 2024-09-16T05:11:48.625 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906037/remote/smithi152/log 2024-09-16T05:11:48.626 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T05:12:51.900 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T05:12:51.911 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T05:12:51.911 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T05:12:51.994 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T05:12:52.002 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T05:12:52.002 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T05:12:52.018 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-09-16T05:12:52.041 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T05:12:52.041 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-09-16T05:12:52.041 INFO:teuthology.orchestra.run.smithi152.stdout:^* hv01.front.sepia.ceph.com 3 6 377 63 +373us[ +376us] +/- 48ms 2024-09-16T05:12:52.042 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 +882us[ +882us] +/- 66ms 2024-09-16T05:12:52.042 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 -1630us[-1630us] +/- 75ms 2024-09-16T05:12:52.042 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T05:12:52.042 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T05:12:52.050 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T05:12:52.050 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T05:12:52.058 DEBUG:teuthology.orchestra.run.smithi152:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T05:12:52.223 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T05:12:52.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:52.252 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T05:12:52.261 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T05:12:52.269 INFO:teuthology.task.internal:Duration was 2363.675720 seconds 2024-09-16T05:12:52.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T05:12:52.278 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T05:12:52.278 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T05:12:52.357 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T05:12:52.534 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T05:12:52.534 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2024-09-16T05:12:52.534 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T05:12:52.567 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T05:12:52.567 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T05:12:52.634 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T05:12:52.634 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T05:12:52.820 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T05:12:52.838 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T05:12:52.838 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T05:12:52.881 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T05:12:52.892 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:12:52.961 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2024-09-16T05:12:52.977 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:12:53.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:53.032 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T05:12:53.044 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T05:12:53.045 DEBUG:teuthology.misc:Transferring archived files from smithi152:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906037/remote/smithi152 2024-09-16T05:12:53.046 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T05:12:53.108 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T05:12:53.109 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T05:12:53.158 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T05:12:53.169 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T05:12:53.169 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T05:12:53.181 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T05:12:53.182 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T05:12:53.213 INFO:teuthology.orchestra.run.smithi152.stdout: 265708 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 05:12 /home/ubuntu/cephtest 2024-09-16T05:12:53.214 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T05:12:53.224 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T05:12:53.249 INFO:teuthology.run:Summary data: description: rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} duration: 2363.6757202148438 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T05:12:53.250 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T05:12:53.334 INFO:teuthology.run:pass